go / s3

updated September 02, 2026

The AWS SDK for Go is a large dependency tree. I call three S3 operations: list objects, get an object, and presign a PUT. The SDK's main work is request signing, and AWS Signature Version 4 is a few hundred lines, so I wrote a client.

Client

s3.Client holds two HTTP clients: one with a 10-second timeout for small calls, and one without a timeout for streaming downloads:

type Client struct {
	region     string
	signer     *SigV4
	httpClient *http.Client
	// streamClient is separate from httpClient because GetObject returns
	// a streaming body that callers may read for much longer than the
	// 10s timeout we use for small request/response calls.
	streamClient *http.Client
	// endpointFn is a hook so tests can route requests to httptest.
	endpointFn func(bucket string) string
}

func NewClient(region, accessKeyID, secretAccessKey string) (*Client, error) {
	signer, err := NewSigV4("s3", region, accessKeyID, secretAccessKey)
	if err != nil {
		return nil, err
	}
	return &Client{
		region:       region,
		signer:       signer,
		httpClient:   &http.Client{Timeout: 10 * time.Second},
		streamClient: &http.Client{},
		endpointFn: func(bucket string) string {
			return fmt.Sprintf("https://%s.s3.%s.amazonaws.com", bucket, region)
		},
	}, nil
}

SigV4

The signer supports header signing for GET requests and query-string presigning for PUT URLs. It skips session tokens, asymmetric signatures, event streams, and path normalization.

NewSigV4 validates every field, so a misconfigured worker fails at startup:

func NewSigV4(service, region, accessKeyID, secretAccessKey string) (*SigV4, error) {
	// ... trim each field
	if service == "" || region == "" || accessKeyID == "" || secretAccessKey == "" {
		return nil, errors.New("s3: missing service, region, access_key_id, or secret_access_key")
	}
	// ...
}

Signing builds the canonical request, derives the string to sign, and computes the signature. The canonical request joins six elements with newlines:

func (s *SigV4) canonicalRequest(method string, u *url.URL, headers map[string]string, contentSHA string) string {
	return strings.Join([]string{
		strings.ToUpper(method),
		canonicalPath(u),
		canonicalQuery(u.RawQuery),
		canonicalHeaders(headers) + "\n",
		signedHeaders(headers),
		contentSHA,
	}, "\n")
}

The signing key derives from a chain of HMAC calculations over the date, region, service, and a fixed string:

func (s *SigV4) signature(date, stringToSign string) string {
	kDate := hmacSHA256([]byte("AWS4"+s.secretAccessKey), date)
	kRegion := hmacSHA256(kDate, s.region)
	kService := hmacSHA256(kRegion, s.service)
	kSigning := hmacSHA256(kService, "aws4_request")
	return hex.EncodeToString(hmacSHA256(kSigning, stringToSign))
}

Escaping

SigV4 requires RFC 3986 URL escaping. url.QueryEscape differs in two ways: a space encodes as +, and a tilde encodes as %7E. A wrong escape produces a signature mismatch, which S3 reports as a 403:

func escape(s string) string {
	escaped := url.QueryEscape(s)
	escaped = strings.ReplaceAll(escaped, "+", "%20")
	escaped = strings.ReplaceAll(escaped, "%7E", "~")
	return escaped
}

Presigned PUT

PresignedPutURL makes no HTTP call. It signs a URL the uploader PUTs to. The uploader must send each signed header, such as Content-Type, with the same value:

func (c *Client) PresignedPutURL(bucket, key, contentType string, expiresIn time.Duration) (string, error) {
	return c.signer.PresignURL(http.MethodPut, c.objectURL(bucket, key), expiresIn, map[string]string{
		"Content-Type": contentType,
	}, time.Now())
}

Presigning puts the auth in query params such as X-Amz-Signature. It sets the payload hash to UNSIGNED-PAYLOAD because the signer never sees the bytes.

Two retry paths

ListObjectsV2 buffers a small XML body, so it uses the backoff helper. The build closure re-signs each attempt because the signature binds to x-amz-date:

func (c *Client) getBytes(ctx context.Context, url string) ([]byte, error) {
	build := func() (*http.Request, error) {
		signed, err := c.signer.SignRequest(http.MethodGet, url, nil, nil, time.Now())
		if err != nil {
			return nil, fmt.Errorf("sign: %w", err)
		}
		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
		if err != nil {
			return nil, err
		}
		for k, v := range signed {
			req.Header.Set(k, v)
		}
		return req, nil
	}

	res, err := httputil.Do(ctx, build, httputil.Config{
		Client:         c.httpClient,
		TransientCodes: transientCodes,
	})
	// ... check status, return res.Body
}

httputil.Do consumes the response body, and GetObject must return the open body to the caller. So GetObject has its own retry loop. It reuses httputil.DefaultRetryDelays and httputil.Sleep, so tests set one zero delay:

func (c *Client) GetObject(ctx context.Context, bucket, key string) (io.ReadCloser, error) {
	u := c.objectURL(bucket, key)
	var lastErr error
	for i, delay := range httputil.DefaultRetryDelays {
		// ... re-sign, build request
		resp, err := c.streamClient.Do(req)
		// ... on transient error or transient code, sleep and continue
		if resp.StatusCode/100 != 2 {
			resp.Body.Close()
			return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
		}
		return resp.Body, nil // caller closes
	}
	// ... return lastErr
}

Tests

endpointFn routes requests to an httptest.NewServer, so the client runs without AWS. The signer has its own tests that pin the signature against known inputs.

← All articles