Cost Per Request

AI Cost Management

Cost per request is the fully loaded cost of serving a single call to an AI-powered feature, covering model inference, retrieval, tool calls, and the supporting infrastructure the call consumes. Cost per request connects an AI feature to unit economics by converting a variable per-token rate into a per-call figure that can be compared against the value one call produces. The term also describes per-invocation pricing for API gateways and serverless functions, but cost per request for an AI feature varies from call to call with token consumption. Any cost per request figure depends on how a request is defined, since one user-facing action can trigger several model calls in a multi-step feature.

Cost Component	What It Covers	How It Is Typically Billed
Model inference	Input and output tokens processed by the large language model (LLM) for the call	Per token, at separate input and output rates, so cost moves with prompt and response length
Retrieval and embeddings	Vector search queries and any embedding generated to serve the call	Per query or per provisioned capacity, depending on the vector database and embedding service
Tool and function calls	External APIs, code execution, or data lookups the model triggers during the call	Billed by each downstream service on its own terms, often outside the LLM API invoice entirely
Gateway and orchestration	The API gateway, serverless functions, or LLM proxy that route and manage the call	Per invocation and per compute second, which sets a floor under every call regardless of how many tokens the call consumes
Observability and logging	Traces, prompt and response records, and evaluation data retained for the call	Per volume ingested and per retention period, which scales with request count rather than with token count
Retries and failed attempts	Additional model calls spent producing one usable answer	Billed as ordinary usage in most cases, so a retried request costs a multiple of a clean one. Whether a failed call is billed depends on the provider and the failure mode

Understanding Cost Per Request in AI Features

A per-token rate prices the raw input to an AI feature. Cost per request prices the unit a user actually triggers. Teams reach for cost per request because neither a total monthly bill nor a per-token rate shows what one use of a feature costs. A total bill aggregates every feature together, and a per-token rate says nothing about how many tokens a typical call consumes.

Cost per request is only meaningful once the request itself is defined. Two conventions are common: one cost per request per user-facing action, and one cost per request per model call. A single user-facing action in a multi-step or agentic feature can issue many model calls, so the two conventions produce different numbers for the same feature. Mixing them across teams makes the resulting figures incomparable, which is why the denominator belongs alongside the number whenever it is reported.

Cost per request varies from call to call rather than holding steady. Prompt length, response length, the volume of retrieved context, and the number of retries all move the figure, and none of them are constant across requests. This variability is why an average cost per request describes a distribution rather than a fixed price.

Cost per request means something different in traditional cloud billing. An API gateway charges a published price per invocation, and a serverless function charges per invocation plus compute duration, so cost per request there is predictable from configuration and traffic. For an AI feature, cost per request is a measured average with a tail, because each call consumes a different amount of model capacity.

The mechanics are visible in how retries land on a bill. A feature that retries a failed generation bills two model calls to produce one usable answer, which raises cost per request without any change in the published per-token rate. A feature that grows its retrieved context bills more input tokens per call, which raises cost per request even when request volume is flat.

Cost Components of a Single AI Request

A model-only cost per request counts input and output tokens and stops there. A fully loaded cost per request adds every other cost the call sets in motion. The gap between the two is where feature-level cost estimates most often go wrong.

Components go missing from a cost per request figure for predictable reasons, and each omission distorts the number in a different direction:

Retrieval and embeddings are billed by a different vendor than the model, so they land on an invoice nobody opens when the question is about AI spend.
Tool and function calls are triggered by the model rather than by the application, which makes the most expensive calls the least visible in the feature's own code path.
Gateway and orchestration costs scale with request count rather than token count, so they distort cost per request most for high-volume features with short prompts.
Observability and logging costs are set by retention settings and log verbosity as much as by traffic, so a request can carry more logging cost than its token count suggests.
Retries are usually tracked as a reliability metric rather than a cost one, even though a retried request bills its model inference more than once.

Leaving these components out biases cost per request low, and the bias is largest for high-volume features with heavy retrieval or high retry rates, which are the features a manager most needs to watch. Because the components are billed by different vendors on different mechanisms, the cost of a single request rarely appears as one line on any invoice. Assembling cost per request is a deliberate calculation rather than a number a billing console reports.

How Cost Per Request Supports Feature-Level Cost Accountability

Cost per request is the metric that makes an AI feature's unit economics legible. Total spend answers whether a bill is affordable this month, while cost per request answers whether the feature still works at ten times the volume. A feature with an acceptable monthly bill and a high cost per request becomes unaffordable as it succeeds, and only the per-request figure shows that before the volume arrives.

Allocation of AI spend depends on capturing cost per request at the level the organization intends to report on. A shared API key or account serving several features produces an invoice nobody can decompose after the fact. Attributing cost per request to a feature requires usage tagged at the call level, and without that tagging a rising AI bill cannot be traced to the team that caused it.

Cost per request per feature converts an aggregate invoice into something a manager can review alongside delivery work. Reported per feature, cost per request sits in the same conversation as scope and velocity rather than arriving as a finance escalation a quarter later.

An average cost per request hides the failure this metric is most often used to catch. Cost per request tends to follow a long-tailed distribution, where a minority of calls consume much more model capacity than the median call. A mean that looks acceptable can sit on top of a small set of expensive requests that drive most of the spend. Reporting a high percentile alongside the average cost per request makes that tail visible.

Cost per request ties to three FinOps dimensions at once. Cost per request serves unit economics as the per-call figure that determines whether a feature is sustainable, and allocation as the figure that must be attributed to a feature or team. Cost visibility is the measurement practice that makes both of those possible.

Measuring Cost Per Request in Production

Measuring cost per request accurately depends on a few decisions about scope:

Define and document the denominator first, so figures stay comparable across features and teams.
Scope the calculation to production traffic, since development and evaluation volume flatter the result.
Report a high percentile alongside the average, so the expensive tail stays visible.
Calculate per feature rather than per account, because an account-level figure mostly measures feature mix.

A manager who owns the budget rather than the instrumentation can ask a team for three things: cost per request per feature, the denominator behind that figure, and the share of requests that are retries.

The levers that reduce cost per request sit with engineering, though knowing which ones exist makes it possible to ask for them:

Routing simpler calls to smaller model tiers rather than defaulting to a flagship model.
Capping maximum output length so no single request generates an unbounded response.
Structuring prompts so stable content sits at the front and qualifies for a cached rate where a provider offers one.
Reducing retry rates by fixing the underlying failure causes rather than raising retry limits.

A rising cost per request with no vendor price change is a diagnostic signal rather than a billing error. The usual causes are longer prompts or responses, more retrieved context per call, and a growing retry rate.

Cost per request for a self-hosted model is derived rather than billed. Cost per request for a self-hosted model equals the infrastructure cost of serving the model over a period divided by the requests served in that period, so idle GPU capacity raises cost per request while the feature itself is unchanged.

Infracost estimates the cost of cloud infrastructure defined in Terraform and other infrastructure as code before it is deployed, and posts that estimate on the pull request introducing the change. Those estimates cover the provisioned side of a self-hosted request cost, including the GPU instances a team would run to serve a model itself. Infracost does not meter third-party LLM API token usage and does not measure runtime cost per request.

Related Concepts

Cost Per Token: The unit rate charged for processing one token through a large language model, which supplies the model inference component of a cost per request figure.

Token Economics: The wider cost structure created by per-token pricing, which explains why cost per request varies from one call to the next rather than holding steady.

LLM Observability: The practice that supplies the per-call and per-feature usage data a cost per request calculation depends on.

AI Cost Governance: The policy and accountability layer that sets model tier and output length limits, both of which cap how expensive a single request can become.

AI FinOps: The practice of applying FinOps discipline to AI spend, within which cost per request serves as a primary unit economics metric.

Frequently Asked Questions (FAQs)

What is cost per request?

Cost per request is the fully loaded cost of serving a single call to an AI-powered feature. Cost per request covers model inference, retrieval, tool calls, and the supporting infrastructure the call consumes, not the model tokens alone. Because token consumption differs from call to call, cost per request is a measured average rather than a fixed published price.

How do you calculate cost per request for an AI feature?

Cost per request is calculated by dividing the total cost a feature incurred over a period by the number of requests it served in that period. The numerator should include retrieval, tool calls, gateway and orchestration compute, logging, and retried attempts alongside model inference. Calculating cost per request per feature rather than per account keeps the figure from reflecting feature mix instead of the feature itself.

What is the difference between cost per request and cost per token?

Cost per token is the unit rate charged for processing one token through a large language model (LLM), while cost per request is the total cost of one call to a feature. Cost per request is built partly from cost per token, applied to the input and output tokens a typical call consumes, then added to every non-model cost the call triggers. Cost per token is the metric for comparing models, and cost per request is the metric for comparing features.

What counts as one request when a single user action triggers several model calls?

Cost per request has to state its denominator explicitly, because no single convention exists. Counting one request per user-facing action measures what a feature costs to use, while counting one request per model call measures what the underlying inference costs. A multi-step or agentic feature produces different cost per request figures under the two definitions, so the definition should be reported alongside the number.

Does cost per request include retrieval and tool call costs?

A fully loaded cost per request includes retrieval and tool call costs, along with gateway compute, logging, and retried attempts. A model-only cost per request counts input and output tokens and excludes all of those, which understates what the call actually costs. Because retrieval and tool calls are billed by services other than the model provider, cost per request has to be assembled from several invoices rather than read from one.

Why can an average cost per request hide a cost problem?

Cost per request tends to follow a long-tailed distribution, where a minority of calls consume much more model capacity than the median call. An average cost per request can therefore look acceptable while a small share of expensive requests drives most of the spend. Reporting a high percentile alongside the average cost per request makes that tail visible.