OpenAI Service – Consider TPM Limits

The OpenAI Service TPM limits policy checks that every Azure OpenAI model deployment uses a Tokens-Per-Minute (TPM) limit at or below the value your organization has set for that model and SKU. Oversized TPM limits raise the maximum rate at which a deployment can consume billable tokens and lock up regional quota that other deployments could use. Apply this policy to every Azure OpenAI deployment defined in Terraform, with the strictest limits in development and test environments.

Why This Policy Matters

How It Helps Reduce Cloud Costs

An Azure OpenAI TPM limit is a rate limit, not a billed resource. Standard and Global Standard deployments bill for the tokens actually processed, so the TPM value itself carries no direct charge.

The TPM limit still controls cost exposure. It sets the highest rate at which a deployment can burn through billable tokens when a retry loop, a runaway batch job, or a misbehaving agent sends traffic nonstop.

A deployment with capacity = 120 allows up to 120,000 tokens per minute, or roughly 172.8 million tokens per day. A deployment with capacity = 10 caps that at 14.4 million tokens per day.

A TPM limit is the cheapest spending guardrail available for Azure OpenAI. It costs nothing to set and bounds the damage of a bug before the invoice arrives.

Azure OpenAI quota is also finite. Quota is assigned per subscription, per region, per model, and per deployment type, and every deployment consumes TPM from that shared pool.

One oversized development deployment can exhaust the regional quota for a model. Teams then request quota increases or deploy in another region, which adds cross-region latency and operational overhead.

Potential Savings

Azure OpenAI TPM limit savings come from avoided spend, not from a lower unit price. Right-sizing TPM does not change what each token costs.

The value of this policy shows up in incidents that never escalate. A development deployment capped at 10K TPM cannot generate the same token bill as one left at 120K TPM, regardless of what the calling code does.

Right-sizing TPM also returns quota to the regional pool. Teams can reuse that quota for new production deployments instead of waiting on a quota increase request.

This policy delivers the most value in non-production subscriptions, where traffic is low and a high TPM ceiling has no legitimate purpose.

Implementation Guide

Infrastructure-as-Code Example (Terraform)

The following examples show an Azure OpenAI deployment with an oversized TPM limit, and the corrected configuration.

Non-compliant configuration: TPM limit far above the preferred value

resource "azurerm_cognitive_account" "openai" {
  name                = "dev-openai"
  location            = azurerm_resource_group.ai.location
  resource_group_name = azurerm_resource_group.ai.name
  kind                = "OpenAI"
  sku_name            = "S0"
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 120
  }
}

resource "azurerm_cognitive_account" "openai" {
  name                = "dev-openai"
  location            = azurerm_resource_group.ai.location
  resource_group_name = azurerm_resource_group.ai.name
  kind                = "OpenAI"
  sku_name            = "S0"
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 120
  }
}

resource "azurerm_cognitive_account" "openai" {
  name                = "dev-openai"
  location            = azurerm_resource_group.ai.location
  resource_group_name = azurerm_resource_group.ai.name
  kind                = "OpenAI"
  sku_name            = "S0"
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 120
  }
}

This deployment allows 120,000 tokens per minute in a development account. This configuration increases maximum token spend and consumes regional quota without improving anything a development workload needs.

Compliant configuration: TPM limit at the preferred value

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 10
  }
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 10
  }
}

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat-gpt-4"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = var.model_name    # for example "gpt-4"
    version = var.model_version
  }

  sku {
    name     = "Standard"
    capacity = 10
  }
}

Setting capacity = 10 limits the deployment to 10,000 TPM, which matches a gpt-4:Standard:10 preferred limit. Changing capacity updates the deployment in place, while changing the sku name or the model format or name forces Terraform to recreate the deployment.

Set var.model_name and var.model_version to a model and version that are currently available in your region. Azure retires model versions on a rolling schedule, and az cognitiveservices account list-models lists what an Azure OpenAI account can deploy today.

Step-by-Step Fix Instructions

1. Find every azurerm_cognitive_deployment resource in your Terraform code and record its model name, skuname, and capacity.

2. Define a preferred TPM limit for each model and SKU combination, per environment, in the model:SKU:limitformat.

3. Check actual peak token usage for each deployment in Azure Monitor metrics before lowering a limit on a production deployment.

4. Set capacity in the sku block to the preferred limit, remembering that one unit equals 1,000 TPM.

5. Run terraform plan and confirm the change is an in-place update of capacity, not a destroy and recreate.

6. Apply the change and confirm the new TPM value on the deployment in the Azure portal or the Foundry portal.

7. Add Infracost to your CI/CD pipeline so new or modified deployments that exceed the preferred limit get flagged in the pull request.

Best Practices

• Set TPM limits per environment. Development and test deployments rarely need more than a small fraction of production capacity.

• Size production TPM from observed peak usage plus headroom, not from the maximum quota available in the region.

• Handle HTTP 429 responses with exponential backoff in client code. A lower TPM limit is safe only when callers retry correctly.

• Keep preferred limits in one shared policy definition instead of hardcoding different values across Terraform modules.

• This policy is not a substitute for Azure budgets or cost alerts. TPM limits cap the rate of spend, while budgets cap the total.

Tools and Scripts

The OpenAI Service TPM limits policy is available in Infracost, including in the free trial. Infracost checks Azure OpenAI deployments in your Terraform code against the preferred limits you define in the model:SKU:limit format, for example gpt-4:Standard:10.

When Infracost runs in CI/CD, it flags a pull request that adds or raises a TPM limit above the preferred value, before the change merges. Platform engineers see the finding in the pull request instead of discovering an oversized deployment after a usage spike.

Infracost also surfaces existing deployments that already violate the policy. FinOps and platform teams use that list to burn down TPM violations over time and measure progress toward full compliance.

To list deployments and their current capacity with the Azure CLI:

az cognitiveservices account deployment list \
  --name <account-name> \
  --resource-group <resource-group> \
  --query "[].{name:name, model:properties.model.name, sku:sku.name, capacity:sku.capacity}" \
  --output

az cognitiveservices account deployment list \
  --name <account-name> \
  --resource-group <resource-group> \
  --query "[].{name:name, model:properties.model.name, sku:sku.name, capacity:sku.capacity}" \
  --output

az cognitiveservices account deployment list \
  --name <account-name> \
  --resource-group <resource-group> \
  --query "[].{name:name, model:properties.model.name, sku:sku.name, capacity:sku.capacity}" \
  --output

Examples of Impact

Illustrative example: a retry loop in development. A developer ships a test script that retries failed completions without backoff. With a 120K TPM limit, the script can consume up to 172.8 million tokens per day before anyone notices. With a 10K TPM limit, the same bug is capped at 14.4 million tokens per day and surfaces as HTTP 429 errors instead of a large invoice line.

Illustrative example: quota blocked by idle deployments. A platform team cannot create a new production deployment because development deployments hold most of the regional quota for the model. Lowering the development TPM limits returns that quota to the pool without a quota increase request.

(These are illustrative, composite scenarios, not specific customer accounts.)

Considerations and Caveats

• Rate limit, not a bill: A lower TPM limit on a Standard deployment caps maximum spend. It does not reduce the cost of tokens a workload actually needs.

• Throttling risk: A TPM limit set below real peak demand causes HTTP 429 errors. Validate production limits against usage data first.

• Requests per minute follow TPM: Azure sets the requests-per-minute (RPM) limit in proportion to TPM, with a ratio that varies by model. Lowering TPM also lowers RPM.

• Provisioned deployments differ: Provisioned throughput SKUs bill by deployed capacity, so the cost logic for those deployments is different from Standard deployments.

• Model versions retire: Azure retires model versions on a schedule, so confirm the model version in Terraform is still available in your region.

Related Policies and Concepts

• OpenAI Service: Consider using latest models: a companion AI-group policy that flags older Azure OpenAI models where newer models offer better cost efficiency.

• OpenAI Service: Consider using preferred SKUs: a companion AI-group policy that restricts which deployment SKUs teams can use for each model.

• Token Economics (Glossary): explains how token volume and per-token pricing drive AI spend, which TPM limits bound at the deployment level.

• Cost Per Token (Glossary): defines the unit rate that, multiplied by the TPM ceiling, sets the worst-case spend for an Azure OpenAI deployment.

• AI Cost Optimization (Glossary): covers the broader set of practices that this policy belongs to.

Frequently Asked Questions (FAQs)

Is the OpenAI Service TPM limits policy supported in Infracost?

Yes. The OpenAI Service TPM limits policy is available in Infracost, including in the free trial. Infracost flags Azure OpenAI deployments in pull requests when their TPM limit exceeds the preferred value for that model and SKU.

Can the OpenAI Service TPM limits policy be customized?

Yes. Teams define preferred TPM limits in the model:SKU:limit format, for example gpt-4:Standard:10, which means gpt-4 deployments on the Standard SKU should stay at or below 10K TPM.

Does Infracost automatically fix TPM limit violations?

No. Infracost detects and reports Azure OpenAI deployments that exceed the preferred TPM limit. Engineers lower the capacity value in Terraform and apply the change themselves.

Does a lower TPM limit reduce my Azure OpenAI bill?

A lower TPM limit does not change the price of each token on Standard deployments. A lower TPM limit caps the maximum rate of token spend, which prevents runaway usage from turning into a large Azure OpenAI bill.

Is the OpenAI Service TPM limits policy cloud-agnostic?

No. The OpenAI Service TPM limits policy targets Azure OpenAI deployments managed with the azurerm Terraform provider. Other AI platforms use different resources and quota models.

What does the capacity value mean in an Azure OpenAI deployment?

In an azurerm_cognitive_deployment resource, capacity in the sku block sets the TPM limit in units of 1,000 tokens per minute. A capacity of 10 equals 10,000 tokens per minute.

How often should I review Azure OpenAI TPM limits?

Infracost checks Azure OpenAI TPM limits on every pull request that adds or changes a deployment. For existing deployments, review TPM limits against actual usage at least quarterly, and after any major traffic change.