Concurrently crawl URLs (the Tour of Go web crawler) with depth limits, deduplication and bounded parallelism.

There are three concerns:

• Dedup: a mutex-guarded seen map. You must check and mark it in one critical section, before fetching. Otherwise two goroutines both see "not seen" and fetch the same URL.
• Bounded parallelism: a buffered-channel semaphore held only around Fetch.
• Completion: a WaitGroup where Add is called before each go.

type Fetcher interface {
	Fetch(ctx context.Context, url string) (body string, urls []string, err error)
}

func Crawl(ctx context.Context, root string, depth, parallel int, f Fetcher) map[string]error {
	var (
		mu   sync.Mutex
		seen = make(map[string]error)
		wg   sync.WaitGroup
		sem  = make(chan struct{}, parallel)
	)
	var visit func(url string, depth int)
	visit = func(url string, depth int) {
		defer wg.Done()
		if depth <= 0 {
			return
		}
		mu.Lock()
		if _, ok := seen[url]; ok {
			mu.Unlock()
			return
		}
		seen[url] = nil // claim before fetching
		mu.Unlock()

		select {
		case sem <- struct{}{}:
		case <-ctx.Done():
			mu.Lock()
			seen[url] = ctx.Err()
			mu.Unlock()
			return
		}
		_, urls, err := f.Fetch(ctx, url)
		<-sem // release BEFORE spawning children

		mu.Lock()
		seen[url] = err
		mu.Unlock()
		if err != nil {
			return
		}
		for _, u := range urls {
			wg.Add(1)
			go visit(u, depth-1)
		}
	}
	wg.Add(1)
	go visit(root, depth)
	wg.Wait()
	return seen
}

Gotchas:

• If a goroutine held the semaphore while waiting on its children, the crawl would deadlock once the depth exceeds parallel.
• wg.Add inside the child goroutine is a race, because Wait might see zero first.
• The semaphore bounds fetches, not goroutines. A page with 10k links creates 10k parked goroutines, which is cheap (a few KB each) but unbounded.

For strict bounds, use a fixed worker pool that reads from a frontier queue and counts outstanding work, so it can close the queue when the count reaches zero. Another option is errgroup.SetLimit, but its Go blocks when the limit is full. Calling it from inside a worker can deadlock, so use TryGo or a separate dispatcher.