Compose io primitives: upload a file to HTTP while computing its SHA-256 and limiting size, without buffering it in memory.
Question 459HardGo 1.22 to 1.25
The power of io.Reader/io.Writer is composition: io.TeeReader (read and copy to a writer), io.MultiWriter (fan-out writes), io.LimitReader, io.Pipe (connect a writer-producing API to a reader-consuming one, synchronously, no buffer).
func upload(ctx context.Context, c *http.Client, url, path string) (string, error) {
f, err := os.Open(path)
if err != nil {
return "", err
}
defer f.Close()
h := sha256.New()
body := io.TeeReader(io.LimitReader(f, 100<<20), h) // hash while streaming
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, body)
if err != nil {
return "", err
}
resp, err := c.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return "", fmt.Errorf("upload failed: %s", resp.Status)
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// io.Pipe: gzip on the fly into a request body
func gzipBody(src io.Reader) io.Reader {
pr, pw := io.Pipe()
go func() {
zw := gzip.NewWriter(pw)
_, err := io.Copy(zw, src)
if cerr := zw.Close(); err == nil {
err = cerr
}
pw.CloseWithError(err) // nil err => reader sees io.EOF
}()
return pr
}
Gotchas: io.LimitReader silently truncates at the limit (no error) — to reject oversized files, check f.Stat() first or read limit+1 bytes and fail if you got more; with io.Pipe always CloseWithError or the reader blocks forever; io.Copy uses WriterTo/ReaderFrom fast paths (e.g. sendfile for *os.File to TCP) when available.
More on Standard Library, HTTP & Systems Design in Go
- Q457How do you handle transactions, NULLs and "no rows" correctly with database/sql?
- Q458What is the exact io.Reader contract? What's wrong with this read loop?
- Q460What are the pitfalls of bufio.Scanner and bufio.Writer?
- Q461Implement graceful shutdown for an HTTP server with background workers.
- Q462How does log/slog work? Discuss handlers, attributes, LogValuer, and performance.
- Q463What does this print? Explain the monotonic clock and time.Time comparison.