How do you implement a circuit breaker and bulkhead in Go? What state machine and concurrency concerns are involved?

A circuit breaker is a state machine. Closed: calls pass and failures are counted. Open: calls fail fast until a cooldown ends. Half-Open: a limited number of probe calls decide whether it goes back to Closed or to Open. A bulkhead caps concurrent calls per dependency, so one slow backend can't use up every goroutine and connection.

type State int

const (
	Closed State = iota
	Open
	HalfOpen
)

var ErrOpen = errors.New("circuit open")

type Breaker struct {
	mu        sync.Mutex
	state     State
	failures  int
	threshold int
	cooldown  time.Duration
	openedAt  time.Time
}

func (b *Breaker) Do(ctx context.Context, fn func(context.Context) error) error {
	if err := b.allow(); err != nil {
		return err
	}
	err := fn(ctx) // never hold the lock during the call
	b.record(err)
	return err
}

func (b *Breaker) allow() error {
	b.mu.Lock()
	defer b.mu.Unlock()
	switch b.state {
	case Open:
		if time.Since(b.openedAt) < b.cooldown {
			return ErrOpen
		}
		b.state = HalfOpen // this caller becomes the single probe
	case HalfOpen:
		return ErrOpen // a probe is already running
	}
	return nil
}

func (b *Breaker) record(err error) {
	b.mu.Lock()
	defer b.mu.Unlock()
	if err == nil {
		b.state, b.failures = Closed, 0
		return
	}
	b.failures++
	if b.state == HalfOpen || b.failures >= b.threshold {
		b.state, b.openedAt, b.failures = Open, time.Now(), 0
	}
}

type Bulkhead struct{ sem chan struct{} }

func NewBulkhead(n int) *Bulkhead { return &Bulkhead{sem: make(chan struct{}, n)} }

func (b *Bulkhead) Do(ctx context.Context, fn func(context.Context) error) error {
	select {
	case b.sem <- struct{}{}:
		defer func() { <-b.sem }()
		return fn(ctx)
	case <-ctx.Done():
		return ctx.Err() // or fail immediately with a default: case
	}
}

Concerns: never hold the mutex during the remote call. Results that arrive late from a previous state must not flip the current state; production breakers such as sony/gobreaker use a generation counter for this. Decide whether context.Canceled or 4xx responses count as failures (usually not). Prefer a rolling window over a raw counter. Allow only a bounded number of Half-Open probes. Report state changes as metrics. Order matters: timeout inside the bulkhead, breaker outside the retries.