How do you instrument a Go service with Prometheus metrics (counters, gauges, histograms)? What are the label cardinality pitfalls?

Use the github.com/prometheus/client_golang library:

• Counters only go up (requests, errors). Query them with rate().
• Gauges go up and down (in-flight requests, queue depth).
• Histograms put observations into buckets (latency, sizes). You compute quantiles on the server with histogram_quantile, and you can aggregate across instances, which summaries cannot do.

var (
	reqs = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "http_requests_total", Help: "HTTP requests.",
	}, []string{"route", "method", "code"})
	inflight = promauto.NewGauge(prometheus.GaugeOpts{
		Name: "http_in_flight_requests", Help: "In-flight requests.",
	})
	latency = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "http_request_duration_seconds",
		Help:    "Request latency.",
		Buckets: prometheus.DefBuckets,
	}, []string{"route", "method"})
)

func instrument(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		inflight.Inc()
		defer inflight.Dec()
		rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
		start := time.Now()
		next.ServeHTTP(rec, r)
		route := r.Pattern // Go 1.23+: "GET /users/{id}", not "/users/42"
		latency.WithLabelValues(route, r.Method).Observe(time.Since(start).Seconds())
		reqs.WithLabelValues(route, r.Method, strconv.Itoa(rec.code)).Inc()
	})
}

type statusRecorder struct {
	http.ResponseWriter
	code int
}

func (s *statusRecorder) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) }

// mux.Handle("GET /metrics", promhttp.Handler())

Cardinality pitfalls: Prometheus creates a separate time series for every unique combination of label values. Labels such as user ID, email, raw URL path, query string, request ID or error message create unbounded series. The result is memory exhaustion in Prometheus and in your own process, because the client keeps every series in memory forever.

• Use the route template, not the path.
• Bucket status codes (2xx/5xx) if needed.
• Keep labels to small, fixed sets.
• Use exemplars or traces for per-request detail.

Other gotchas:

• Name metrics with base units (_seconds, _bytes) and give counters a _total suffix.
• Choose histogram buckets around your SLO.
• Registering the same metric twice panics.