Fan out N HTTP calls with bounded concurrency, fail fast on first error, and collect results in order.
Question 472HardGo 1.22 to 1.25
golang.org/x/sync/errgroup is the idiomatic tool: WithContext cancels the shared context on the first error, SetLimit bounds concurrency, and Wait returns the first error. Writing to distinct indexes of a pre-sized slice needs no mutex.
func fetchAll(ctx context.Context, c *http.Client, urls []string) ([][]byte, error) {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8) // at most 8 in flight
results := make([][]byte, len(urls))
for i, u := range urls { // Go 1.22: i, u are per-iteration
g.Go(func() error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return err
}
resp, err := c.Do(req)
if err != nil {
return fmt.Errorf("get %s: %w", u, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("get %s: %s", u, resp.Status)
}
b, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20))
if err != nil {
return err
}
results[i] = b
return nil
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
Points to raise: g.Go blocks when the limit is reached (that's the backpressure); the transport's MaxConnsPerHost/MaxIdleConnsPerHost must match your concurrency or connections churn; before Go 1.22 you had to copy i, u inside the loop. For "collect all errors" semantics, use a sync.WaitGroup plus errors.Join instead.
More on Standard Library, HTTP & Systems Design in Go
- Q470What is a cache stampede and how does singleflight solve it?
- Q471Design an in-process pub/sub broker. How do you handle slow subscribers?
- Q473How do you unit-test HTTP handlers and HTTP clients with net/http/httptest?
- Q474Implement Server-Sent Events (SSE) in net/http. How do http.ResponseController, flushing and WriteTimeout interact?
- Q475Write an HTTP client retry helper with exponential backoff and jitter. What makes a request safe to retry?
- Q476Build a reverse proxy with httputil.ReverseProxy. Why is Rewrite preferred over Director?