Distributed Tracing

Distributed tracing is a method of following a single request as it moves across multiple services, recording the sequence, duration, and metadata of each step so engineers can see exactly where time and resources were spent. In a microservices or serverless architecture, a request that touches a dozen services produces no single log or metric that shows its full path, and distributed tracing is what reconstructs it. The same span-level data that already shows which service made a request slow can also show which service made it expensive, once spans carry cost-relevant metadata correlated against actual infrastructure spend. Distributed tracing is an observability technique on its own; it becomes a cost visibility mechanism only when that correlation step is added deliberately.

How Distributed Tracing Follows a Request Across Services

A trace represents the complete path of one request as it moves through a distributed system. Within that trace, a span represents a single unit of work, such as one service handling one step of the request. Spans nest into a parent-child hierarchy: a span for an API gateway call can contain child spans for every downstream service that call triggers, which is what lets distributed tracing reconstruct the full shape of a request rather than a flat list of events.

Distributed tracing depends on context propagation to hold this structure together. When a request crosses a service boundary, the calling service passes a trace ID and span ID to the next service, typically through HTTP headers following a standard such as W3C Trace Context. Every service the request touches attaches its own span to that same trace ID, which is how a tracing backend can later reassemble the full path from spans generated by many independent services.

OpenTelemetry has become the common standard for generating this data. It defines how applications create and export traces, metrics, and logs, so a team can instrument a service once and send the resulting spans to whichever tracing backend it uses.

Distributed tracing is one of three commonly cited observability signals, alongside logs and metrics. Logs and metrics describe what happened inside one service. Distributed tracing is the signal that shows how a request moved between services, which is why it is the natural place to attach cross-service cost data.

Traces, Spans, and Sampling: The Building Blocks

Two mechanisms determine what a trace can actually show: span attributes and sampling.

• Span attributes (tags): Key-value metadata attached to a span, such as service name, environment, region, or instance type. Span attributes are what make a span usable for anything beyond timing, including cost attribution, but they only carry cost meaning once correlated against real infrastructure spend.

• Head-based sampling: A decision made at the start of a trace about whether to keep it, applied before the outcome of the request is known. Head-based sampling is cheap to run but cannot preferentially retain expensive or failed traces.

• Tail-based sampling: A decision made after a full trace completes, which can preferentially keep traces that were slow, failed, or unusually costly, at the price of buffering every span until that decision is made.

Span attributes are the mechanism, not the cost figure itself. A span tagged with a service name and instance type still needs to be joined against billing or infrastructure cost data before it says anything about spend.

Attaching Cost to Spans: The Cost Visibility Case for Distributed Tracing

Distributed tracing already makes an expensive request as visible as a slow one, provided spans carry cost-relevant metadata and that metadata is correlated against real infrastructure spend. A trace that shows a request was slow is only half the picture if nothing shows that the same request also triggered a fan-out of downstream calls to an oversized instance.

Aggregate cost metrics hide exactly the pattern distributed tracing can surface. A service can look inexpensive on average while a specific trace, driven by a fan-out to many downstream calls or a retry storm, never appears in a monthly cost report. Distributed tracing operates at the level a monthly bill cannot: one request, with every service it touched along the way.

This is a direct cost visibility connection: distributed tracing turns cost from a property of an account or service into a property of an individual request. It also supports cost allocation, since span attributes that name a service, team, or environment are what let a specific request's cost be attributed rather than absorbed into shared infrastructure spend. Reaching this level of detail depends on the same span attributes and unit economics discipline, such as tracking cost per request, that a fully loaded cost figure requires.

Implementing Cost-Aware Distributed Tracing

Cost-aware distributed tracing starts with instrumentation choices, not analysis after the fact. Spans should carry cost-relevant attributes, such as service name, environment, and instance type, at creation time. Attributes added later cannot be applied retroactively to traces that have already completed.

Trace data becomes cost data once it is correlated with billing or cost allocation exports. A span's service name and time window are what let an expensive trace be matched back to the infrastructure spend it generated, rather than treated as a timing artifact.

Sampling is the main trade-off in a cost-aware tracing setup. Capturing every trace gives the most complete cost visibility picture, but increases the storage and processing cost of the tracing infrastructure itself. Tail-based sampling, which can preferentially retain expensive traces, addresses this more directly than head-based sampling, at the cost of buffering every span until a decision is made. Tools such as OpenTelemetry Collector, Jaeger, Datadog APM, and Honeycomb implement these sampling strategies, though the specific configuration options vary by tool.

Distributed tracing shows cost after a request has already executed. Tools like Infracost estimate the cost of infrastructure defined in Terraform before it is provisioned, surfacing that estimate on the pull request introducing the change. That is a different, earlier layer of cost visibility than distributed tracing provides: one prices a change before deployment, the other measures what a request actually cost once the system is running.

Related Concepts

Observability: The broader practice distributed tracing is one signal within, alongside logs and metrics, and the property that determines whether a system's telemetry can answer cost as well as reliability questions.

Cost Per Request: The fully loaded cost of serving one request, which span-level cost attribution from distributed tracing directly feeds by attributing cost to the individual services a request touched.

Cloud Billing Data: The line item usage and cost records a cloud provider emits, which distributed tracing data must be correlated against before a span's cost attribution means anything.

Token Economics: The cost structure created by per-token LLM API pricing, relevant when a traced request includes a call to a large language model and that call's token cost needs to be attributed to the same trace.

Frequently Asked Questions (FAQs)

What is distributed tracing?

Distributed tracing is a method of following a single request as it moves across multiple services, recording the sequence, duration, and metadata of each step. Distributed tracing reconstructs the full path of a request from the individual spans each service generates. It is the observability signal best suited to showing how a request moved between services, rather than what happened inside just one.

How does distributed tracing work across microservices?

Distributed tracing works across microservices through context propagation: a calling service passes a trace ID and span ID to the next service, typically in HTTP headers following a standard such as W3C Trace Context. Each service the request touches attaches its own span to that trace ID. A tracing backend then reassembles the full trace from spans generated independently by every service in the path.

What is the difference between distributed tracing and logging?

Distributed tracing and logging capture different things. Logging records timestamped events from within a single service, while distributed tracing records the path and timing of one request across every service it touches. Distributed tracing is the signal that shows how a request moved between services, which no single service's logs can show on their own.

What is the difference between a trace and a span?

A trace is the complete path of one request across a distributed system, while a span is a single unit of work within that trace, such as one service handling one step. Spans nest into a parent-child hierarchy, so a trace is assembled from many related spans rather than being a single record. Distributed tracing depends on this trace-and-span structure to reconstruct a request's full path.

How do you attach cost data to a distributed trace?

Cost data gets attached to a distributed trace by tagging spans with cost-relevant attributes, such as service name, environment, and instance type, at instrumentation time. That span data then needs to be correlated with billing or cost allocation records, since a span attribute alone identifies a cost driver without stating a cost figure. Distributed tracing supplies the request-level structure; billing data supplies the dollar amount.

Can distributed tracing help reduce cloud costs?

Distributed tracing can help reduce cloud costs by surfacing expensive requests that aggregate cost metrics hide, such as a fan-out to many downstream calls or a retry storm. Distributed tracing does not reduce cost directly; it makes an expensive request visible in the same way it already makes a slow one visible, which is what lets a team act on it. Whether that visibility leads to a fix still depends on the team investigating what the trace shows.

How does sampling affect the cost of running distributed tracing itself?

Sampling determines how many traces a distributed tracing system actually captures and stores, which directly affects the tracing infrastructure's own storage and processing cost. Head-based sampling decides whether to keep a trace before the request completes, which is cheap but cannot preferentially retain expensive traces. Tail-based sampling can preferentially retain expensive or failed traces, at the cost of buffering every span from a trace until that decision is made.