Cost Per Token

Cost per token is the unit price charged for processing a single token of text through a large language model (LLM), typically quoted per million tokens and priced separately for input and output. Cost per token is the base rate behind every downstream AI cost metric, including cost per request, cost per user, and cost per feature. The published list price for a model and the effective rate an organization actually pays are different numbers, and a forecast built on the wrong one will understate AI spend. Cost per token applies to third-party LLM API billing, where the rate is charged directly, and to self-hosted models, where the rate is derived from infrastructure cost rather than billed by a vendor.

Understanding Cost Per Token in LLM API Pricing

A token is a sub-word unit of text that a large language model (LLM) processes and bills on, rather than a word or a character. Cost per token attaches a price to that billing unit, which is why token counts and not word counts determine what a request costs. Token counts vary by language and by the specific tokenizer a provider uses, and the Token Economics entry covers those mechanics in more depth.

Most providers quote cost per token per million tokens, while some older documentation still quotes per thousand tokens. The difference matters for a forecast: reading a per-thousand figure as though it were a per-million figure produces an estimate off by three orders of magnitude. Confirming the unit before building any model of AI spend removes a common source of error in the calculation.

A single model does not carry one cost per token. Most models carry several published rates at once:

• Standard input rate: the price for prompt and context tokens sent to the model.

• Output rate: the price for tokens the model generates, set above the input rate on every major provider.

• Cached input rate: a discounted rate some providers apply to repeated prompt prefixes, available only on cache hits.

• Batch rate: a discounted rate for requests processed asynchronously rather than in real time, where a provider offers it.

These rates interact inside a single request. A request that generates a long response raises the effective cost per token for that request, because output tokens bill at a higher rate than input tokens. A request that reuses a long, stable system prompt lowers the effective cost per token, because the cached input rate applies to the repeated prefix instead of the standard input rate.

List Price Versus Effective Cost Per Token

The blended effective cost per token is total spend on a model over a period divided by the total tokens that model processed in the same period. The blended effective cost per token is calculated per model rather than across an entire fleet, and it represents the rate an organization actually pays rather than the rate a vendor publishes. This figure is the correct input to an AI spend forecast, because it already accounts for how the organization uses the model.

The blended effective cost per token almost never matches any single published price. Four factors move the blended rate: the ratio of output tokens to input tokens, the share of input tokens served from cache, retried requests that bill twice for one usable answer, and reasoning tokens, which providers bill at the output rate on models that generate them. Reasoning tokens are not always returned in the response, so a short visible answer can carry a much larger billed output count.

A blended rate calculated across every model at once mostly measures model mix rather than efficiency. Shifting traffic from a flagship model to a smaller tier lowers a fleet-wide blended rate even if every individual model became more expensive to run. Calculating the rate per model keeps those two effects separate.

Cost per token for a self-hosted or fine-tuned model is derived rather than billed. Cost per token for a self-hosted model equals the infrastructure cost of serving the model over a period divided by the tokens it produced in that period. GPU utilization therefore moves the rate directly, because higher utilization spreads the same infrastructure cost across more tokens. Idle GPU capacity raises cost per token without any change in the model, the prompts, or the request volume.

How Cost Per Token Drives AI Unit Economics

Cost per token is the base rate that every AI unit economics metric is derived from. Cost per request is the cost per token rate applied to the input and output token counts of a typical request. Cost per user and cost per feature extend the same arithmetic across request volume, so an error in the underlying cost per token figure propagates into every metric built above it.

Allocation of AI spend depends on token counts captured at the level the organization intends to report on. A single API key or account often serves several features and teams at once, so a rising bill cannot be traced to a source without per-feature or per-team token attribution. Cost visibility at the account level alone is not enough to hold any one team accountable for its share of AI spend.

A forecast built on a published list price will understate actual spend. A list price is a single rate, usually the input rate, while a real bill blends input, output, cached, and batch rates along with retried requests. Building the forecast on the blended effective cost per token closes that gap, and tracking the blended rate over time shows whether AI unit economics are improving as usage grows.

Cost per token ties to three FinOps dimensions at once. Cost per token serves unit economics as the base rate under every per-unit metric, and allocation as the figure that must be attributed to a team or feature. Cost visibility is the measurement practice that makes both of those possible.

Measuring and Reducing Effective Cost Per Token

Measuring cost per token accurately depends on a few decisions about scope:

• Calculate the blended effective rate per model rather than across all models, so the number reflects efficiency instead of model mix.

• Scope the calculation to production traffic, since development and testing volume can flatter the result.

• Use a trailing window long enough to be stable, so ordinary week to week variation is not read as a real trend.

• Track the input to output token ratio alongside the rate, because that ratio explains most of the movement in the blended figure.

The levers that reduce effective cost per token sit mostly with engineering teams, though a FinOps lead needs to know which ones exist in order to ask for them:

• Routing simpler tasks to lower cost model tiers instead of defaulting to a flagship model.

• Capping maximum output length, and capping reasoning effort where a provider exposes that control.

• Structuring prompts so stable content sits at the front, which raises the share of input tokens eligible for a cached rate.

• Moving work that tolerates delay to batch processing where a provider offers it.

A rising blended cost per token with no vendor price change is a diagnostic signal rather than a billing error. The usual causes are a shift in mix toward output tokens, a drop in cache hit rate, or growth in retried requests.

Infracost estimates the cost of cloud infrastructure defined in Terraform before it is deployed, including the GPU instances a team would provision to serve a model itself. Those estimates cover the numerator of a self-hosted cost per token calculation. They do not cover third-party LLM API token spend, which is metered and billed by the provider rather than estimated from infrastructure code.

Related Concepts

Token Economics: The wider cost structure and spending behavior created by per-token pricing in LLM APIs, of which cost per token is the underlying unit rate.

LLM Observability: The practice that supplies the per-model and per-feature token counts a blended cost per token calculation depends on.

AI Cost Management: The discipline of controlling AI spend, within which cost per token is a primary tracked metric.

AI Cost Governance: The policy and accountability layer that sets limits on model choice and output length, both of which move cost per token.

FinOps Tools: The tooling category used to collect, allocate, and report the spend and usage data behind a cost per token figure.

Frequently Asked Questions (FAQs)

What is cost per token?

Cost per token is the unit price charged for processing a single token of text through a large language model (LLM). Cost per token is typically quoted per million tokens and priced separately for input tokens and output tokens. A single model usually carries several published cost per token rates rather than one price.

How is cost per token calculated?

Published cost per token is set by the provider and listed on its pricing page, usually per million tokens for each rate type. The effective cost per token an organization pays is calculated by dividing total spend on a model over a period by the total tokens that model processed in the same period. The effective cost per token is the figure to use for forecasting, because it reflects the actual mix of input, output, and cached tokens.

Why do output tokens cost more per token than input tokens?

Output tokens carry a higher cost per token than input tokens on every major LLM provider, commonly between two and eight times higher depending on the model. Output tokens are generated one at a time, while input tokens can be processed together in a single parallel pass, so generation consumes more compute per token. This gap means a request that produces a long response raises the effective cost per token for that request.

What is the difference between cost per token and Token Economics?

Cost per token is the unit rate itself, a single published price or calculated figure for one model. Token Economics is the wider cost structure and spending behavior that per-token pricing creates, including the budgeting and forecasting difficulty caused by usage-driven variability. Cost per token is the number, and Token Economics is the system that number sits inside.

Why is the effective cost per token higher than the price on a vendor's pricing page?

The effective cost per token blends every rate a model charges, while a pricing page lists each rate separately and is often read as the input rate alone. Output tokens bill above input tokens, retried requests bill twice for one usable answer, and reasoning tokens are billed at the output rate on models that generate them. A low cache hit rate also raises the effective cost per token, because fewer input tokens qualify for a discounted cached rate.

How is cost per token calculated for a self-hosted model?

Cost per token for a self-hosted model is calculated by dividing the total infrastructure cost of serving the model over a period by the number of tokens it produced in that period. Because the numerator is infrastructure cost rather than a billed rate, GPU utilization moves cost per token directly, and idle capacity raises the figure without any change in model or usage. This makes cost per token for a self-hosted model a derived metric rather than a published price.