How do you add distributed tracing with OpenTelemetry in Go? How does the trace context flow through context.Context and HTTP/gRPC?

Setup takes three steps:

1. Create a TracerProvider with an exporter (usually OTLP), a resource (service name) and a sampler.
2. Register it globally along with a propagator.
3. Wrap your transports.

Within a process, the active span lives in context.Context. tracer.Start(ctx, ...) returns a child context that holds the new span. Across processes, the propagator writes the W3C traceparent/tracestate headers into HTTP headers or gRPC metadata, and the receiving side extracts them.

func initTracing(ctx context.Context) (func(context.Context) error, error) {
	exp, err := otlptracegrpc.New(ctx) // honours OTEL_EXPORTER_OTLP_ENDPOINT
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithResource(resource.NewSchemaless(semconv.ServiceName("orders"))),
		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
	)
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))
	return tp.Shutdown, nil // call on exit to flush spans
}

var tracer = otel.Tracer("orders/service")

func (s *Service) Place(ctx context.Context, o Order) error {
	ctx, span := tracer.Start(ctx, "Place", trace.WithAttributes(attribute.Int("items", len(o.Items))))
	defer span.End()
	if err := s.repo.Save(ctx, o); err != nil { // pass ctx down or the trace breaks
		span.RecordError(err)
		span.SetStatus(codes.Error, err.Error())
		return err
	}
	return nil
}

// HTTP server:  otelhttp.NewHandler(mux, "server")
// HTTP client:  &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
// gRPC server:  grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))
// gRPC client:  grpc.NewClient(addr, grpc.WithStatsHandler(otelgrpc.NewClientHandler()), ...)

Gotchas:

• Using context.Background() in the middle of a call chain, or starting a goroutine without passing ctx, silently breaks the trace.
• For background work that must outlive the request but stay linked to it, use context.WithoutCancel(ctx) (Go 1.21).
• Forgetting span.End() leaks the span.
• Not calling Shutdown on exit loses the last batch of spans.
• Keep attributes low-cardinality where your backend indexes them.
• Use ParentBased sampling so every service keeps or drops the same trace.
• The OTel gRPC interceptors are deprecated. Use the stats handlers shown above.