Packages, modules and tooling · 3 MIN READ

Profile CPU, allocation and waiting as different bottlenecks

Use pprof and traces to investigate latency under load instead of increasing pools blindly.

A high-latency request does not necessarily consume much CPU. It may be waiting on a mutex, scheduling, a network response or a connection pool. Capture evidence for the same workload and time window, and choose a profile that can explain the suspected delay.

Packages and production tooling · A MAP OF THE IDEA

  1. Latency symptom
  2. CPU / allocation / wait
  3. Profile and trace
  4. One measured change

Useful when

  • Use pprof and traces to investigate latency under load instead of increasing pools blindly.

Watch out

  • A blocked request can spend most of its time waiting while consuming little CPU. Inspect queue, pool and synchronization delays as well as execution samples.

Implementation & operation cost: Profiles consume sampling overhead and need matching binaries, source revisions and a representative workload.

01

Capture a bounded benchmark profile

CPU profiles show sampled execution time. Heap and allocation profiles answer different memory questions, so choose the sample type deliberately. Keep the test binary with its profiles and source revision; otherwise symbol names and comparisons can become misleading.

bash
go test ./internal/codec -run='^$' -bench=BenchmarkEncode -cpuprofile=cpu.out -memprofile=mem.out
go tool pprof -top cpu.out
go tool pprof -sample_index=alloc_space -top mem.out
Read the code

The commands create local profile artifacts and show where time or allocated bytes accumulate in the measured benchmark. They do not show every blocked request or replace pool wait metrics from a real service.

02

Trace scheduling and explicit contention

Execution traces provide a timeline of goroutines and runtime activity. Block and mutex profiles need suitable runtime sampling configuration; collecting zero samples is not evidence that a service never waits. Profiling itself can affect the workload, so compare with the uninstrumented baseline.

bash
go test ./internal/service -run TestBurst -trace=trace.out
go tool trace trace.out
go test ./internal/service -run TestBurst -blockprofile=block.out -mutexprofile=mutex.out
Read the code

Replace TestBurst with a bounded representative test. Inspect overlap among waiting, CPU work and goroutine growth before changing concurrency or blaming the garbage collector.

03

Turn evidence into one falsifiable change

If pool checkout dominates, inspect request arrival rate and slow operations before raising pool limits. If allocation dominates, find retained objects and hot allocation sites. Expose production profiling only through an appropriately restricted diagnostic path because profiles can reveal application data and implementation details.

TAKE THIS WITH YOU

Match the profile to the suspected bottleneck and compare one controlled change. CPU time, allocated bytes and waiting time are related but different measurements.

Check your understanding

Why can a slow service have a mostly idle CPU?

Go to the source

Progress stays in this browser. No account needed.