Go

Fan out N HTTP calls with bounded concurrency, fail fast on first error, and collect results in order.

Question 472HardGo 1.22 to 1.25

golang.org/x/sync/errgroup is the idiomatic tool: WithContext cancels the shared context on the first error, SetLimit bounds concurrency, and Wait returns the first error. Writing to distinct indexes of a pre-sized slice needs no mutex.

func fetchAll(ctx context.Context, c *http.Client, urls []string) ([][]byte, error) {
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(8) // at most 8 in flight
	results := make([][]byte, len(urls))

	for i, u := range urls { // Go 1.22: i, u are per-iteration
		g.Go(func() error {
			req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
			if err != nil {
				return err
			}
			resp, err := c.Do(req)
			if err != nil {
				return fmt.Errorf("get %s: %w", u, err)
			}
			defer resp.Body.Close()
			if resp.StatusCode != http.StatusOK {
				return fmt.Errorf("get %s: %s", u, resp.Status)
			}
			b, err := io.ReadAll(io.LimitReader(resp.Body, 10<<20))
			if err != nil {
				return err
			}
			results[i] = b
			return nil
		})
	}
	if err := g.Wait(); err != nil {
		return nil, err
	}
	return results, nil
}

Points to raise: g.Go blocks when the limit is reached (that's the backpressure); the transport's MaxConnsPerHost/MaxIdleConnsPerHost must match your concurrency or connections churn; before Go 1.22 you had to copy i, u inside the loop. For "collect all errors" semantics, use a sync.WaitGroup plus errors.Join instead.

More on Standard Library, HTTP & Systems Design in Go

All 35 Standard Library, HTTP & Systems Design in Go questions