Go

How do you find a memory leak in production with pprof?

Question 363MediumGo 1.22 to 1.25
import _ "net/http/pprof" // registers /debug/pprof/* on DefaultServeMux

func main() {
    go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()
    // ...
}

Steps:

  1. Take two heap snapshots a few minutes apart:
    curl -o a.pb.gz localhost:6060/debug/pprof/heap, wait, then take b.pb.gz.
  2. Diff them: go tool pprof -sample_index=inuse_space -base a.pb.gz b.pb.gz, then use top, list Func, or the -http=:8080 flame graph.
  3. Also check /debug/pprof/goroutine, because leaked goroutines hold heap memory.

The heap profile has four views:

  • inuse_space / inuse_objects: memory still live as of the last GC, shown where it was allocated. Use these to find leaks.
  • alloc_space / alloc_objects: everything allocated since the program started. Use these to find allocation churn and GC pressure.

Gotchas: the profile shows where memory was allocated, not who is still holding it. You have to work out the retaining path yourself; the viewcore tool can help with core dumps. The profile is sampled, by default one sample per 512 KB allocated (runtime.MemProfileRate). It does not show memory allocated by C, goroutine stacks, or overhead from the runtime or fragmentation. Those are why RSS can exceed what the profile shows, so compare with runtime/metrics. Continuous profiling tools such as Pyroscope or Parca make diffs over time easy.

More on Memory, GC & Runtime Internals

All 38 Memory, GC & Runtime Internals questions