Observability

Observability is the ability to understand a system's internal state from the data it emits externally, including metrics, logs, traces, and profiles. The same telemetry also records capacity consumed and requests served, so observability can attribute cloud spend to a specific service or code path, which a monthly bill cannot. Observability is broader than monitoring, which watches a fixed set of signals for conditions defined ahead of time. Observability describes systems that are already running, which is what separates observability from pre-deployment cost estimation.

Understanding Observability: Inferring System State from Emitted Data

Observability is a property of a system rather than a category of tool. The term originates in control theory, where it describes how well the internal states of a system can be determined from its external outputs. Applied to distributed software, observability measures whether the telemetry a system emits is rich enough to answer questions about its behavior.

Observability and monitoring solve different problems. Monitoring collects a predefined set of metrics and fires alerts on thresholds decided in advance, which works well when the failure modes are already known. Observability supports questions that nobody anticipated when the instrumentation was written. Most teams operate both, since a monitoring dashboard is only useful when the underlying system is observable.

Distributed architectures are what made observability necessary. A single request may cross dozens of services, so no individual host's metrics explain why that request was slow. Correlating metrics, logs, and traces across service boundaries became the requirement, rather than reading any one telemetry type in isolation.

OpenTelemetry, a graduated Cloud Native Computing Foundation (CNCF) project, standardizes how applications collect, process, and export this telemetry independently of any single vendor. Its adoption means that instrumentation written once can be sent to different observability backends through configuration rather than code changes.

Resource consumption and request volume are already present in this telemetry. Utilization, request rate, and instance counts are collected as a matter of course for reliability purposes. Cost is therefore derivable from data most teams already emit, rather than a separate data collection problem.

The Telemetry Types Behind Observability: Metrics, Logs, Traces, and Profiles

Observability rests on several distinct telemetry types, each answering a different question and carrying a different collection cost. The phrase "three pillars of observability" is common shorthand for the first three below, not a formal standard.

• Metrics: Numeric measurements sampled over time, such as CPU utilization, request rate, and error rate. Metrics aggregate well and are cheap to store, but they lose per-request detail.

• Logs: Timestamped event records emitted by applications and infrastructure. Logs carry the highest detail per event and typically generate the largest raw volume of any telemetry type.

• Traces: The path of a single request across services, assembled from spans that each represent one unit of work. Traces answer where time was spent in a request, which no single service's metrics can show.

• Profiles: CPU and memory consumption attributed to specific functions and code paths. Continuous profiling is the most recent addition to the set, and its tooling support is less mature than that for metrics, logs, and traces.

Cardinality is the number of distinct values a telemetry label can take, and it is the concept that connects diagnostic power to cost. High-cardinality labels, such as user identifiers or request identifiers, make it possible to isolate one customer's experience from millions of others. Those same labels multiply the number of stored time series, which increases both storage and query cost.

Teams rarely adopt all four telemetry types at once. Metrics and logs are typically instrumented first, with distributed tracing and continuous profiling added later as systems grow more distributed and root cause analysis gets harder.

How Does Observability Affect Cloud Cost Visibility?

Observability affects cloud cost visibility because observability telemetry already contains the utilization and demand data needed to reason about cost. Most observability practices treat cost as a finance reporting concern rather than an operational signal, which leaves that data unused.

The mechanism is direct. An instance that consistently runs far below its provisioned capacity generates cost without producing proportional value, and utilization metrics compared against provisioned capacity expose that gap. Request rate combined with published instance pricing produces a cost-per-request figure. Trace data attributes compute time to a specific service in a request path, which identifies the dependency driving the most cost per call.

Cloud billing data answers a different question. The AWS Cost and Usage Report delivers cost and usage data to Amazon S3 at least once a day, and can be configured for hourly, resource-level granularity. Even at that granularity, billing data is organized by resource, account, and service. Billing data records what a resource cost, not which request path or which code change caused the consumption.

Cost allocation depends on labels applied consistently at instrumentation time. Attributing spend by service, team, or environment requires that telemetry and cloud resources carry the same identifiers. Inconsistent labeling breaks cost allocation and distributed tracing at the same time, because both practices depend on the same tagging discipline.

Treating Cost as a Signal Alongside Latency and Errors

The Google Site Reliability Engineering book defines four golden signals for monitoring a user-facing system: latency, traffic, errors, and saturation. Cost is not among them. Teams that treat cost as an observability signal emit it alongside those four rather than routing it to a separate finance tool.

• Derive a cost-rate metric from data already collected. Cost per thousand requests, calculated from utilization and published instance pricing, needs no telemetry beyond what most teams already collect. What matters is placement: the metric belongs on the dashboard engineers already watch during a deploy.

• Alert on cost rate rather than cost total. A threshold on spend per service per hour surfaces a regression within hours. A monthly budget alert surfaces the same regression after the money is spent.

• Apply cost allocation tags to telemetry at instrumentation time. Using the same tags in telemetry and in cloud resource definitions lets the runtime view and the billing view reconcile instead of disagreeing.

• Track cost per unit of output alongside total spend. Cost per request or cost per tenant shows whether a service is becoming more or less efficient as traffic grows. A rising total alone cannot separate an efficiency regression from rising demand.

Observability has a firm boundary worth naming. Observability describes what a running system costs, and cannot describe what a proposed change will cost before it is merged. A change that doubles an instance size becomes visible in telemetry only after it reaches production, which means the rework happens after the fact.

Pre-deploy cost estimation covers that gap. Tools like Infracost estimate the cost of infrastructure defined in Terraform and post the cost difference as a comment on the pull request, so cost impact is discussed during code review. That estimate is derived from infrastructure code rather than runtime telemetry, which makes pre-deploy estimation complementary to an observability practice rather than a replacement for one.

The Cost of Observability Itself: Telemetry Volume and Retention

Observability is itself a cost center, distinct from the cloud spend that observability data helps explain. Observability platforms commonly bill on some combination of data ingested, hosts monitored, retention period, and user seats, though the specific model varies by vendor.

Telemetry volume scales with both traffic and cardinality. Adding a high-cardinality label to a widely used metric multiplies the stored time series without any change in traffic at all. An observability bill can therefore grow faster than the infrastructure it observes.

Four levers control that growth:

• Sampling: Head-based sampling decides whether to keep a trace when the trace begins, which is cheap but cannot preferentially retain errors. Tail-based sampling decides after the full trace is collected, which keeps the interesting traces but requires buffering every span until the decision.

• Retention tiering: Keeping recent telemetry in fast, queryable storage and moving older telemetry to cheaper storage.

• Cardinality limits: Capping or dropping labels that multiply time series without changing any decision anyone makes.

• Dropping unqueried telemetry: Removing metrics and logs that no dashboard, alert, or investigation has referenced.

Telemetry retention is a governance question as much as a tooling one. Someone has to decide what gets collected and for how long, and that decision is frequently unowned. Without an owner, collection defaults to keeping everything, which is the most expensive option available.

Related Concepts

LLM Observability: The application of observability practice to large language model systems, adding LLM-specific signals such as token counts and output quality that general observability does not capture.

FinOps Tools: The category of tooling built to surface and manage cloud spend, which overlaps with observability platforms in what it reports but differs in granularity and purpose.

Harness engineering: The practices behind building and operating delivery pipelines, where observability signals inform deployment gates and rollback decisions.

Opportunity Cost: The value of the next-best alternative given up by a decision, which applies directly to telemetry retention choices where more stored data means better diagnostics and higher spend.

Frequently Asked Questions (FAQs)

What is observability?

Observability is the ability to understand a system's internal state from the data it emits externally, including metrics, logs, traces, and profiles. Observability is a property of a system rather than a category of tool, and it depends on whether the emitted telemetry is rich enough to answer unanticipated questions. Observability applies to cost as well as reliability, since the same telemetry records resource consumption and request volume.

What is the difference between observability and monitoring?

Observability differs from monitoring in the kind of question each one answers. Monitoring collects a predefined set of metrics and fires alerts on thresholds decided in advance, which works when failure modes are already known. Observability supports questions that nobody anticipated when the instrumentation was written, and most teams operate both together.

What are the three pillars of observability?

The three pillars of observability are conventionally listed as metrics, logs, and traces, though this framing is common shorthand rather than a formal standard. Continuous profiling is often treated as a fourth signal, attributing CPU and memory consumption to specific code paths. The pillars of observability describe telemetry types, not a required order of implementation.

How does observability help with cloud cost visibility?

Observability helps with cloud cost visibility because observability telemetry already records utilization, request volume, and provisioned capacity. Comparing utilization against provisioned capacity exposes over-provisioning, and combining request rate with instance pricing produces a cost-per-request figure. Observability data can therefore identify which service or code path drives spend, which aggregated billing data cannot.

Why does an observability bill grow faster than expected?

An observability bill grows faster than expected because telemetry volume scales with both traffic and cardinality. Adding a high-cardinality label, such as a user or request identifier, multiplies the number of stored time series without any change in traffic. Observability platforms commonly bill on data ingested and retention period, so volume growth translates directly into spend.

Can observability show what a change will cost before it is deployed?

Observability cannot show what a change will cost before it is deployed, because observability describes systems that are already running. A change that increases an instance size becomes visible in telemetry only after it reaches production. Pre-deploy cost estimation tools that read infrastructure code, rather than runtime telemetry, are what cover this gap.