Energy per token is the amount of electricity consumed to process a single token through a large language model (LLM), typically expressed in joules or watt-hours per token. Energy per token is the physical cost basis underneath per-token pricing, which makes energy per token the constraint on how far the cost of serving a token can fall over the long run. As used in cloud cost management, energy per token refers to inference, the recurring energy of serving a deployed model, rather than the one-time energy of training it. Energy per token is also not a fixed property of a model, varying with hardware generation, batch size, and whether a token is an input token processed during prefill or an output token generated during decode.
Variable | Direction of Effect on Energy Per Token | Why |
|---|---|---|
Active parameters per token | A larger active parameter count raises energy per token | More arithmetic and more weight movement per token generated. For a mixture of experts model, the driver is the parameters activated per token, not the total parameter count |
Accelerator generation | Newer accelerators generally lower energy per token | Performance per watt improves across hardware generations, so the same work draws less energy on newer silicon |
Batch size and utilization | Larger batches lower energy per token, up to a practical limit | Fixed per-pass overhead and weight loading are spread across more tokens. Latency targets and accelerator memory capacity cap how far batching goes |
Prefill versus decode | Output tokens generally consume more energy per token than input tokens | Prefill evaluates the prompt in a largely parallel pass, while decode generates output one token at a time |
Numeric precision | Lower precision generally lowers energy per token | Quantized weights and activations move fewer bytes and use cheaper arithmetic, with a possible quality tradeoff |
Measurement boundary | A facility-level figure is higher than an accelerator-level figure for identical work | Facility figures include cooling, power conversion, and networking, an overhead captured by power usage effectiveness (PUE) |
Understanding Energy Per Token in LLM Inference
Energy per token measures the electricity drawn by the hardware serving a large language model (LLM), divided by the number of tokens that hardware processes. That energy goes to three places: arithmetic on the accelerator, movement of model weights and cached attention state through memory, and the facility overhead of cooling and power conversion around both. During output generation, memory bandwidth rather than raw arithmetic is typically the limiting factor.
Energy per token describes inference, the recurring energy of serving a deployed model to users. Training energy is a separate, one-time figure that is spread across all subsequent inference. Inference energy and training energy should not be summed or compared directly, because inference energy scales with usage while training energy does not.
LLM inference runs in two phases with different energy profiles. Prefill processes the input prompt, and the model can evaluate the prompt's tokens together in a largely parallel pass. Decode produces the output one token at a time, and each new token requires its own pass over the model weights.
Because decode repeats that pass for every token generated, output tokens generally consume more energy per token than input tokens. This asymmetry has a direct consequence for cost: generating a longer response raises the energy consumed by a request more than sending a longer prompt does, because each additional output token triggers another sequential decode pass.
Energy per token figures are only comparable when they state a measurement boundary. A reading taken at the graphics processing unit (GPU) reports the accelerator's draw alone. A facility-level figure also includes cooling, power conversion, and networking, an overhead captured by power usage effectiveness (PUE), the ratio of total facility energy to the energy delivered to computing equipment. Two honest measurements of identical work will differ substantially depending on which boundary was used.
What Determines Energy Per Token: Model, Hardware, and Batch Size
Four variables account for most of the variation in energy per token across deployments:
Active parameters per token: Energy per token tracks the parameters actually activated to produce a token. For a mixture of experts model, that figure is a fraction of the total parameter count, so total parameters overstate the energy cost.
Hardware generation: Performance per watt improves across accelerator generations. The same model served on newer hardware generally consumes less energy per token, which is why energy per token is a property of a deployment rather than of a model.
Batch size and utilization: Batching spreads the fixed cost of loading model weights across more tokens in a single pass, so energy per token falls as batch size rises. Latency requirements and accelerator memory capacity cap the effect. Idle or underused hardware pushes the average sharply the other way, because unused capacity still draws power.
Serving optimizations: Quantization reduces the bytes moved per token by storing weights and activations at lower numeric precision. Distillation replaces a large model with a smaller one trained to reproduce its behavior. Speculative decoding uses a small draft model to propose tokens that the larger model verifies in batches. Each lowers energy per token through a different mechanism, and each carries its own quality or complexity tradeoff.
Energy Per Token as the Physical Floor Under Token Pricing
Energy per token measures physical electricity consumption, while cost per token measures a billed price. Energy per token and cost per token are related but not proportional. A published per-token price carries hardware capital recovery, amortized training cost, operating expense, and margin on top of the electricity consumed, so the price always reflects commercial decisions that energy per token alone does not explain.
Energy per token therefore sets a floor on what serving a token can cost a provider over the long run, without setting the price on any given day. A provider can price below its fully loaded cost of serving to win market position. Energy per token constrains where prices can eventually settle, not where they currently sit.
That distinction matters for any budget that extends beyond a single contract term. Where per-token prices decline, the causes divide into physical gains such as hardware efficiency and serving optimization, and commercial choices such as margin and market positioning. Energy per token is the variable that indicates how much of that headroom is physical rather than commercial. A budget built on the assumption of continued price decline is implicitly assuming continued movement in this number.
Energy per token should also be kept separate from carbon per token. Carbon per token is energy per token multiplied by the carbon intensity of the grid serving the request, which varies by region and by time of day. A single energy per token figure maps to many possible emissions figures, so energy per token and carbon per token answer different questions and should not be used interchangeably.
How Energy Per Token Affects AI Unit Economics and Cost Allocation
Energy per token connects to unit economics differently depending on who operates the hardware. For a self-hosted or fine-tuned model, energy per token feeds directly into cost per token, because the electricity bill and the hardware drawing it are part of the infrastructure cost divided by tokens produced. For a third-party LLM API, energy per token is the provider's input rather than the customer's, and reaches the customer only through the published price. Recognizing which of these two situations applies is the first step in deciding whether energy per token is a metric a team can act on or only a metric the team should understand.
Cost allocation becomes harder for teams running their own inference. A single cluster typically serves several features and teams at once, so attributing energy and hardware cost requires token counts tagged at that level of granularity. Without per-feature or per-team token tracking, a rising infrastructure bill cannot be traced to its source, and no team can be held accountable for its share.
Energy per token also gives governance a lever with a physical basis. Limits on model tier, caps on maximum output length, and minimum batch efficiency targets are all policy decisions an organization can set, and each moves energy per token in a predictable direction.
Cost visibility is the constraint that makes this difficult in practice. A cloud bill reports instance hours, not watt-hours, so energy per token rarely appears on any invoice. Reconstructing the figure requires token counts from the serving layer combined with power and utilization data from the hardware, which is why energy per token is usually a derived metric rather than a reported one.
Measuring and Reducing Energy Per Token
Measuring energy per token means recording the power drawn by the serving hardware over a window, then dividing the resulting energy by the tokens processed in that same window. Accelerator telemetry supplies the power side and the serving layer supplies the token counts. Any figure produced should be published alongside its measurement boundary, since an accelerator-level reading and a facility-level reading are not comparable.
Teams should also decide whether they need an average or a marginal figure. An average divides total energy by total tokens over a period, so the average includes idle capacity and suits capacity planning. A marginal figure describes the additional energy of serving one more token on hardware that is already running, which is the number that matters for evaluating incremental traffic.
The levers available for reducing energy per token are the same variables that determine the figure, applied deliberately:
Raise effective batch size to the limit the latency budget allows
Route simpler requests to smaller models rather than serving all traffic from the largest one
Cap maximum output length, since output tokens carry the higher energy cost
Apply quantization where the quality tradeoff is acceptable for the workload
For teams self-hosting a model, the hardware footprint that sets a deployment's energy profile is chosen when the infrastructure is defined, not when it runs. Tools like Infracost estimate the cost of cloud infrastructure defined in Terraform before it is deployed, including the GPU instances used to serve a model, which makes the financial side of that decision visible in a pull request. Infracost does not measure electricity or power draw, and does not meter token usage billed by a third-party LLM API.
Related Concepts
Token Economics: The cost structure and spending behavior created by per-token pricing in LLM APIs, for which energy per token is the underlying physical input on the provider's side.
AI Cost Management: The wider discipline of controlling AI spend, within which energy per token acts as a driver of long-run cost rather than a line item on any bill.
LLM Observability: The practice of instrumenting model usage, which supplies the per-model token counts and serving-layer telemetry that an energy per token calculation depends on.
AI Cost Governance: The policy and accountability layer that sets limits on model tier and output length, both of which move energy per token directly.
FinOps Tools: The tooling category used to collect, allocate, and report the infrastructure spend and usage data from which a self-hosted energy per token figure is derived.
Frequently Asked Questions (FAQs)
What is energy per token?
Energy per token is the amount of electricity consumed to process a single token through a large language model (LLM), typically expressed in joules or watt-hours per token. Energy per token covers inference, the recurring energy of serving a deployed model, rather than the one-time energy of training it. Energy per token is a property of a specific deployment rather than of a model, because hardware and serving configuration both change the figure.
How much energy does a single token use?
No single figure describes energy per token, because model architecture, accelerator generation, batch size, and the boundary of the measurement all change the result. A reading taken at the accelerator excludes cooling and power conversion, so it will be lower than a facility-level figure for identical work. Any energy per token figure should be treated as specific to the deployment and the measurement method that produced it.
Why do output tokens use more energy per token than input tokens?
Output tokens use more energy per token than input tokens because of how the two phases of inference run. Prefill evaluates the input prompt in a largely parallel pass, while decode generates output one token at a time, with each token requiring its own pass over the model weights. That repeated pass is why energy per token is generally higher on the output side.
How does batch size affect energy per token?
Larger batch sizes generally lower energy per token, because the fixed cost of loading model weights is spread across more tokens in the same pass. The effect has a practical limit, since latency targets and accelerator memory capacity both cap how large a batch can be. Underused hardware pushes energy per token in the opposite direction, because idle capacity still draws power.
What is the difference between energy per token and cost per token?
Energy per token measures physical electricity consumption in joules or watt-hours, while cost per token measures a billed price in currency. Energy per token and cost per token are related but not proportional, because a per-token price also carries hardware capital recovery, amortized training cost, operating expense, and margin. A provider can set a price above or below the level implied by energy per token alone.
How is energy per token measured?
Energy per token is measured by recording the power drawn by the serving hardware over a period, then dividing the resulting energy by the number of tokens processed in that same period. The measurement boundary must be stated, since an accelerator-level reading excludes the cooling and power conversion that a facility-level figure includes. Energy per token measured under benchmark conditions will not match a production figure, because batch composition and utilization differ.
Prevent Cloud Budget
Overruns Earlier
Download the whitepaper to see how teams shift FinOps left and add cost guardrails in pull requests.