Go

Compose io primitives: upload a file to HTTP while computing its SHA-256 and limiting size, without buffering it in memory.

Question 459HardGo 1.22 to 1.25

The power of io.Reader/io.Writer is composition: io.TeeReader (read and copy to a writer), io.MultiWriter (fan-out writes), io.LimitReader, io.Pipe (connect a writer-producing API to a reader-consuming one, synchronously, no buffer).

func upload(ctx context.Context, c *http.Client, url, path string) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer f.Close()

	h := sha256.New()
	body := io.TeeReader(io.LimitReader(f, 100<<20), h) // hash while streaming

	req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, body)
	if err != nil {
		return "", err
	}
	resp, err := c.Do(req)
	if err != nil {
		return "", err
	}
	defer resp.Body.Close()
	if resp.StatusCode/100 != 2 {
		return "", fmt.Errorf("upload failed: %s", resp.Status)
	}
	return hex.EncodeToString(h.Sum(nil)), nil
}

// io.Pipe: gzip on the fly into a request body
func gzipBody(src io.Reader) io.Reader {
	pr, pw := io.Pipe()
	go func() {
		zw := gzip.NewWriter(pw)
		_, err := io.Copy(zw, src)
		if cerr := zw.Close(); err == nil {
			err = cerr
		}
		pw.CloseWithError(err) // nil err => reader sees io.EOF
	}()
	return pr
}

Gotchas: io.LimitReader silently truncates at the limit (no error) — to reject oversized files, check f.Stat() first or read limit+1 bytes and fail if you got more; with io.Pipe always CloseWithError or the reader blocks forever; io.Copy uses WriterTo/ReaderFrom fast paths (e.g. sendfile for *os.File to TCP) when available.

More on Standard Library, HTTP & Systems Design in Go

All 35 Standard Library, HTTP & Systems Design in Go questions