GPU Utilization

GPU utilization is the percentage of time over a sampling period during which one or more kernels were executing on a graphics processing unit (GPU), as reported by NVIDIA's nvidia-smi command line tool and by NVIDIA Data Center GPU Manager (DCGM). Low GPU utilization is one of the most expensive forms of cloud waste, because a GPU instance bills at its full hourly rate whether or not the GPU is doing work. The metric measures time spent executing, not how much of the GPU's compute capacity that execution used. A workload can therefore report high GPU utilization while using a small share of the compute available on the device.

Understanding GPU Utilization and What nvidia-smi Actually Reports

GPU utilization is a time-based sample rather than a capacity measurement. NVIDIA defines the figure as the percent of time over the past sample period during which one or more kernels were executing on the GPU, with the sample period falling between one second and one sixth of a second depending on the product. The number answers one question: was the device busy at all?

Because GPU utilization counts time rather than capacity, a small kernel occupying a single streaming multiprocessor (SM) reports the same figure as a job saturating every SM on the device. A workload that runs a tiny kernel continuously will report GPU utilization near 100 percent while using a fraction of the compute available. The consequence for cost is direct: a dashboard showing high GPU utilization is not on its own evidence that a GPU instance is sized correctly.

This is where the distinction becomes operational rather than academic. Teams that read the time-based figure as a capacity figure conclude the GPU is fully used, close the investigation, and stop looking for savings that are still available. The same device can report 100 percent GPU utilization while its Tensor Cores sit almost entirely idle.

In practice, GPU utilization reaches a monitoring system through a short chain of tools. The nvidia-smi command line tool gives a point-in-time read on a single host and is the fastest way to check a running job. NVIDIA Data Center GPU Manager (DCGM) collects the same measurement across a fleet as the field DCGM_FI_DEV_GPU_UTIL, and DCGM Exporter publishes DCGM fields in a format Prometheus can scrape.

Grafana then stores and displays the resulting GPU utilization time series. Cloud provider monitoring agents offer a managed alternative to running that collection stack directly.

GPU Utilization Metrics: Kernel Time, Streaming Multiprocessor Activity, Memory, and Allocation

No single metric describes how well a GPU is being used. Several families of measurement answer different questions, and reading only kernel time produces false confidence in a device's efficiency.

• Kernel time: GPU utilization, reported by nvidia-smi as GPU-Util and by DCGM as DCGM_FI_DEV_GPU_UTIL, records whether any kernel was running. GPU utilization works as a liveness signal rather than an efficiency signal.

• Streaming multiprocessor activity and occupancy: DCGM_FI_PROF_SM_ACTIVE reports the ratio of cycles during which a streaming multiprocessor (SM) had at least one warp assigned. DCGM_FI_PROF_SM_OCCUPANCY reports how many warps were resident against the maximum an SM supports. Read together, the two describe both the breadth and the depth of use.

• Tensor Core activity: DCGM_FI_PROF_PIPE_TENSOR_ACTIVE reports the ratio of cycles the Tensor Core pipe was active. For matrix-heavy training and inference, this metric sits closest to work that produces model output.

• Memory: DCGM_FI_DEV_FB_USED reports how much framebuffer memory a workload holds. That is a different measurement from memory utilization as NVML, the NVIDIA Management Library, defines it, which is the percent of time device memory was being read or written.

• Allocation: the NVIDIA device plugin for Kubernetes treats GPUs as indivisible by default, with its sharing option set to none. A pod requesting nvidia.com/gpu receives a whole physical device, so a cluster can report every GPU allocated while those devices run mostly idle.

Training and inference produce different utilization signatures, and a recommendation that fits one will not fit the other. Training runs tend toward sustained high GPU utilization, where the common gap is time lost to data loading and checkpointing rather than to the model itself. Inference serving tends toward bursty, low average GPU utilization, driven by latency-bound requests that arrive at small batch sizes and leave the device idle between them.

How Low GPU Utilization Drives Cloud Cost Waste

Low GPU utilization wastes money without changing the bill. A GPU instance bills at its full hourly rate for every hour it runs, regardless of what the GPU is doing, so GPU utilization determines what share of that fixed spend produced output. A training cluster running at low GPU utilization costs the same as the identical cluster running at high GPU utilization. The difference is what the organization received in return.

That makes GPU utilization an optimization metric first. Cost reporting tools show what was spent; GPU utilization shows whether the spend bought anything. A GPU instance that is expensive and busy raises a budgeting question. A GPU instance that is expensive and idle is a waste finding with a specific owner and a specific fix.

Unit economics is where GPU utilization becomes legible to finance teams. A percentage does not translate into a budget conversation on its own, but cost per training run and cost per thousand inference requests do. Tracking those units over time shows whether efficiency work is producing savings, or whether a rising GPU bill reflects real growth in workload volume.

Allocation is the third connection and the one most often missed. Because whole GPUs are the default allocation unit in Kubernetes, an underutilized GPU is capacity no other team can claim. That waste does not appear as an idle resource line in a cost report. It appears as a request for more GPU quota, which reads like a growth signal rather than an efficiency problem.

A few patterns account for most low GPU utilization in practice:

• Development and notebook instances left running outside working hours, holding GPUs that no job is using.

• Instance types chosen by default or by precedent rather than by measurement, leaving jobs on larger GPUs than the work requires.

• Training jobs bottlenecked on data loading, where the GPU waits on input rather than on compute.

• Low-traffic inference endpoints each given a dedicated GPU, when several could share one device.

Improving GPU Utilization Across Training and Inference Workloads

Raising GPU utilization generally means one of three things: sharing a device between workloads, changing the workload so it keeps the device busy, or shutting the device off when nothing needs it.

Sharing a physical GPU. Multi-Instance GPU (MIG) partitions a supported GPU into several predefined instances, each presenting as a smaller GPU with memory and fault isolation at the hardware layer. Time-slicing works differently: the NVIDIA device plugin advertises a configurable number of replicas for one physical GPU, and workloads scheduled onto those replicas interleave with one another. Time-slicing is simpler to configure and works on GPUs without MIG support, but it provides no memory or fault isolation between replicas. MIG availability depends on the GPU generation, so confirm support for a specific model before planning around it.

Changing the workload. Increasing batch size raises GPU utilization where the latency budget allows. Fixing data pipeline bottlenecks, through faster storage, more loader workers, or prefetching, removes the stalls that leave a GPU waiting on input during training. Consolidating low-traffic inference endpoints onto shared serving infrastructure raises average GPU utilization across a fleet rather than within a single job.

Managing the lifecycle. Automatic shutdown for idle development and notebook GPU instances removes the spend from GPUs that are billing without running any job. Running interruptible training on spot or preemptible capacity lowers the hourly rate for jobs that can checkpoint and resume.

Measurement then feeds the next provisioning decision. GPU utilization data from a completed run is the evidence for what instance type and count the next run actually needs, and that choice gets made before deployment, in infrastructure code. Tools like Infracost estimate the cost of GPU instances and other resources defined in Terraform and surface that estimate in the pull request, so an instance type or count change can be priced before it is provisioned. Infracost does not measure runtime GPU utilization, which comes from DCGM-based monitoring instead. The two operate at different points in the lifecycle.

Related Concepts

GPU Cost Optimization: The practice of managing and reducing GPU spend in cloud environments, for which GPU utilization is the primary diagnostic measurement.

AI Workload Cost Management: The discipline of tracking and optimizing machine learning infrastructure spend across the full lifecycle, within which GPU utilization is one input signal among several.

AI Training Costs: The cost of model training runs specifically, where GPU utilization determines how much of a run's compute spend produced model progress.

AI Cost Optimization: The set of technical levers that reduce AI spend once it has been authorized, several of which target GPU utilization directly.

LLM Observability: The monitoring layer for large language model behavior and performance in production, adjacent to GPU utilization monitoring but focused on model outputs rather than device efficiency.

Frequently Asked Questions (FAQs)

What is GPU utilization?

GPU utilization is the percentage of time during a sampling period in which one or more kernels were executing on a graphics processing unit. GPU utilization is reported by NVIDIA's nvidia-smi command line tool and by NVIDIA Data Center GPU Manager (DCGM). GPU utilization indicates whether a device was busy, not how much of its compute capacity the work consumed.

What is a good GPU utilization percentage?

There is no single target GPU utilization percentage that applies across workloads. Sustained training jobs and latency-bound inference services have different expected GPU utilization profiles, so a figure that signals health for one can signal a problem for the other. Because GPU utilization measures time rather than capacity, the figure should be read alongside streaming multiprocessor activity and Tensor Core activity before judging efficiency.

How is GPU utilization measured?

GPU utilization is measured by the NVIDIA driver and exposed through NVML, the NVIDIA Management Library. The nvidia-smi command line tool reads GPU utilization for a single host, while NVIDIA Data Center GPU Manager (DCGM) collects it across a fleet as the field DCGM_FI_DEV_GPU_UTIL. DCGM Exporter publishes those fields for Prometheus to scrape, which is the common path for storing GPU utilization history and alerting on it.

What is the difference between GPU utilization and GPU memory utilization?

GPU utilization measures the percent of time during which one or more kernels were executing on the device. GPU memory utilization, as NVML defines it, measures the percent of time device memory was being read or written, which is a bandwidth measurement rather than a capacity one. Neither figure is the same as GPU memory used, which reports how much framebuffer memory a workload currently holds.

Why is GPU utilization low during model training?

GPU utilization during model training is most often low because the GPU is waiting rather than computing. Data loading bottlenecks, checkpoint writes, and synchronization overhead across distributed training workers all leave the device idle between kernels. Small batch sizes also reduce GPU utilization by giving the device less work to do per step.

Does low GPU utilization increase cloud costs?

Low GPU utilization does not increase cloud costs directly, because a GPU instance bills at its hourly rate whether or not the GPU is executing work. Low GPU utilization reduces what that spend produces, which makes it a waste problem rather than an overspend problem. The cost impact of low GPU utilization appears as a higher cost per training run or per inference request, not as a larger invoice line.