go / sentry

Sentry tracks errors in production services. Most of the official sentry-go SDK is tracing, profiling, breadcrumbs, and framework integrations.

I use Sentry only for error tracking, so a few hundred lines cover it. I name the package errs because it also provides the error-wrapping helpers.

API

errs.Init(errs.Options{DSN: os.Getenv("SENTRY_DSN")})
defer errs.Flush(2 * time.Second)

errs.CaptureMessage("did something weird", errs.WithExtra("k", v))
errs.CaptureException(err, errs.WithFingerprint("apollo", "429"))

Functional options (WithTag, WithExtra, WithFingerprint, WithLevel) add per-call metadata:

type Option func(*event)

func WithExtra(key string, value any) Option {
	return func(e *event) {
		if e.Extra == nil {
			e.Extra = map[string]any{}
		}
		e.Extra[key] = value
	}
}

func WithFingerprint(parts ...string) Option {
	return func(e *event) { e.Fingerprint = parts }
}

No-op without a DSN

Each process holds at most one client in an atomic pointer. A nil pointer means no DSN, and every capture is a no-op:

var active atomic.Pointer[client]

func Init(opts Options) error {
	if opts.DSN == "" {
		active.Store(nil)
		return nil
	}
	d, err := parseDSN(opts.DSN)
	if err != nil {
		return err
	}
	// ... build client, start worker
	active.Store(c)
	return nil
}

func CaptureMessage(msg string, opts ...Option) string {
	c := active.Load()
	if c == nil {
		return ""
	}
	e := c.newEvent("info", opts)
	e.Message = &messagePayload{Formatted: truncate(msg)}
	c.dispatch(e)
	return e.EventID
}

Init returns an error only for a malformed DSN. An empty DSN keeps dev runs and tests quiet.

DSN

A Sentry DSN encodes the project ID, public key, and host:

https://{public_key}@{host}/{project_id}

Init parses the DSN to build the envelope endpoint and the authorization header:

func (d dsn) envelopeEndpoint() string {
	port := ""
	if d.port != "" && !isDefaultPort(d.scheme, d.port) {
		port = ":" + d.port
	}
	return fmt.Sprintf("%s://%s%s%s/api/%s/envelope/",
		d.scheme, d.host, port, d.path, d.projectID)
}

func (d dsn) authHeader(now int64, client string) string {
	return fmt.Sprintf(
		"Sentry sentry_version=%s, sentry_timestamp=%d, sentry_key=%s, sentry_client=%s",
		protocolVersion, now, d.publicKey, client,
	)
}

Attaching a stack

A Go error has no stack trace. Wrap and Errorf capture the program counters at the call site:

type stackErr struct {
	err error
	pcs []uintptr
}

func (e *stackErr) Error() string         { return e.err.Error() }
func (e *stackErr) Unwrap() error         { return e.err }
func (e *stackErr) StackTrace() []uintptr { return e.pcs }

func Wrap(err error, msg string) error {
	if err == nil {
		return nil
	}
	return &stackErr{
		err: fmt.Errorf("%s: %w", msg, err),
		pcs: callers(2),
	}
}

I use errs.Wrap(err, "...") where fmt.Errorf("...: %w", err) would appear. CaptureException walks the error chain for the first stack. If it finds none, it uses the capture-site stack:

func stackFromError(err error) []uintptr {
	for e := err; e != nil; {
		if s, ok := e.(stackCarrier); ok {
			return s.StackTrace()
		}
		switch u := e.(type) {
		case interface{ Unwrap() error }:
			e = u.Unwrap()
		case interface{ Unwrap() []error }:
			for _, c := range u.Unwrap() {
				if pcs := stackFromError(c); pcs != nil {
					return pcs
				}
			}
			return nil
		default:
			return nil
		}
	}
	return nil
}

Async dispatch

The calling goroutine must not block on a slow Sentry. The client puts events on a bounded channel, and a worker POSTs them:

func (c *client) dispatch(e *event) {
	if c.syncDispatch {
		c.send(e)
		return
	}
	c.mu.Lock()
	defer c.mu.Unlock()
	if c.closed.Load() {
		return
	}
	c.wg.Add(1)
	select {
	case c.queue <- e:
	default:
		// Queue is full; drop the event and undo the wg.Add so Flush
		// will not block waiting for an event that never enqueued.
		c.wg.Done()
	}
}

func (c *client) worker() {
	defer c.wg.Done()
	for e := range c.queue {
		c.send(e)
		c.wg.Done()
	}
}

The queue holds 1,000 events. Flush stops intake and waits for the queue to drain. I call it from the shutdown path in main. In tests, SyncDispatch sends on the calling goroutine.

Sending events

An envelope is three newline-separated JSON objects: envelope header, item header, and event:

func envelopeBody(e *event) ([]byte, error) {
	envelope := map[string]any{
		"event_id": e.EventID,
		"sent_at":  clock.Now().Format(time.RFC3339),
	}
	item := map[string]any{
		"type":         "event",
		"content_type": "application/json",
	}

	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	for _, v := range []any{envelope, item, e} {
		if err := enc.Encode(v); err != nil {
			return nil, fmt.Errorf("encode envelope: %w", err)
		}
	}
	return bytes.TrimRight(buf.Bytes(), "\n"), nil
}

The worker sends the payload with the backoff helper. On a 429, a callback reads the Retry-After header to delay the retry:

res, err := httputil.Do(context.Background(), build, httputil.Config{
	Client:         c.httpClient,
	TransientCodes: transientCodes,
	RetryAfter: func(resp *http.Response) time.Duration {
		return parseRetryAfter(resp.Header.Get("Retry-After"))
	},
})

After the retries run out, the worker logs the failure and drops the event.

Stack frames

runtime.Callers returns program counters newest-first and Sentry wants them newest-last, so the client reverses the list. It marks frames inside the module in_app:

func resolveFrames(pcs []uintptr, root string) []frame {
	var out []frame
	cf := runtime.CallersFrames(pcs)
	for {
		f, more := cf.Next()
		if f.Function == "" || strings.HasPrefix(f.Function, "runtime.") {
			if !more {
				break
			}
			continue
		}
		out = append(out, frame{
			AbsPath:  f.File,
			Filename: relativeTo(f.File, root),
			Function: f.Function,
			Lineno:   f.Line,
			InApp:    inApp(f.Function),
		})
		if !more {
			break
		}
	}
	for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
		out[i], out[j] = out[j], out[i]
	}
	return out
}

func inApp(fn string) bool {
	return strings.HasPrefix(fn, "app/") || strings.HasPrefix(fn, "app.")
}

relativeTo strips the project directory prefix, so the UI shows apollo/refresh_person.go.

One policy for a 500

My web handlers once answered a failed read with a fixed message and dropped the error. A Sentry issue then held only "fetch overview failed". Nobody could tell a database fault from a user who left the page.

Now every handler passes the error value to one function:

func WriteServerError(w http.ResponseWriter, err error) {
	if err == nil {
		err = errors.New("internal server error")
	}
	if !errors.Is(err, context.Canceled) {
		errs.CaptureException(err)
	}
	log.Printf("WriteServerError: %s", sanitizeLogValue(err.Error()))
	http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
}

The caller wraps the cause with fmt.Errorf and %w, so Sentry gets the whole chain. The stack is the first one that errs.Wrap attached in the chain, or else the capture-site stack (see Attaching a stack). The user still sees a generic 500.

A request that the browser cancels writes a log line and opens no issue. A request that runs past its deadline opens an issue, because it shows a slow query or a full connection pool. sanitizeLogValue strips control characters, so an error string cannot forge a log line.

The function that writes a 4xx message sends a stray 5xx to WriteServerError, so the app has one 5xx policy. A test fails on a handler that writes a 5xx any other way.

Slow requests

The request middleware sends a Sentry message for a request over 750 ms. The message carries duration_ms, and three numbers that say where the time went: db_ms, db_queries, and acquire_ms. See go / postgres for how the pgx tracer counts them.

Trade-offs

This client has no tracing, so no Performance dashboard.

It sends no automatic breadcrumbs. The SDK's default integrations send request params, SQL, and HTTP headers with every event, which leaks PII. This client sends only what each Capture* call passes in.

Tests

Tests point the HTTP client at an httptest.NewServer that records each envelope, and set SyncDispatch:

func TestCaptureMessagePostsEnvelope(t *testing.T) {
	srv := newCaptureServer(t, 200)
	defer srv.Close()

	err := Init(Options{
		DSN:          testDSN,
		HTTPClient:   srv.Client(),
		SyncDispatch: true,
	})
	tu.OK(err == nil)

	id := CaptureMessage("hello", WithExtra("a", 1))

	tu.OK(len(id) == 32)
	ev := srv.events[0]
	msg := ev["message"].(map[string]any)
	tu.OK(msg["formatted"] == "hello")
}

See also

cmd/sentry reads production issues from the Sentry web API.

← All articles