GPU Utilization Monitoring with nvidia-smi and DCGM

NVIDIA's nvidia-smi reports device status and utilization summaries, while DCGM provides monitoring, health, and profiling functions suited to managed systems and clusters. The GPU utilization percentage in nvidia-smi is the share of the recent sample period during which one or more kernels were executing. It is not the percentage of theoretical FLOPs achieved. Memory utilization is a separate activity measure and is not the same as the amount of VRAM allocated. A device can be active nearly all the time yet fail to deliver expected throughput. Kernels may be memory-bound, use few compute units, wait on dependencies, or perform operations that do not use specialized tensor hardware. Conversely, low utilization between steps can point to slow input decoding, small batches, CPU preprocessing, synchronization, or an undersized workload. Compare GPU metrics with tokens or samples per second, step time, latency, and host CPU activity. Memory capacity and memory traffic also answer different questions. nvidia-smi reports used memory in units such as MiB; DCGM profiling can report whether device-memory traffic is active. A high memory-activity ratio can signal bandwidth pressure, while high allocated capacity may simply reflect caching. Watch memory growth over time for leaks and include reserved memory from the framework. Power, clock frequency, and temperature help explain throttling or power limits. ECC and health diagnostics can reveal hardware or communication issues. DCGM's profiling metrics are interval averages; they do not identify an individual slow kernel or source line. Use a profiler when aggregate telemetry suggests a bottleneck but cannot locate it. Sample cadence matters. A short burst can disappear in a long reporting interval, and container or virtualization boundaries may affect which processes are visible. Monitor representative steady-state and burst workloads, and correlate metrics with logs and job phases. Use the tools to form a hypothesis, then verify it with performance profiling.