go / errs

errs is my error client for Go services. It uses the standard library only. It sends errors to sockeye.

I use sockeye for error tracking, so a few hundred lines cover the client.

API

errs.Init(errs.Options{
	DSN:         os.Getenv("SOCKEYE_DSN"),
	Environment: "production",
	Release:     os.Getenv("RENDER_GIT_COMMIT"),
})
defer errs.Flush(2 * time.Second)

errs.CaptureMessage("did something weird", errs.WithExtra("k", v))
errs.CaptureException(err, errs.WithFingerprint("apollo", "429"))

WithTag, WithExtra, WithFingerprint, WithRequestID, and WithLevel add metadata to one event.

An empty DSN makes every capture a no-op, so dev runs and tests stay quiet. Init returns an error only for a malformed DSN.

Attaching a stack

A Go error has no stack trace. Wrap and Errorf capture the program counters where they run:

func Wrap(err error, msg string) error {
	if err == nil {
		return nil
	}
	return &stackErr{
		err: fmt.Errorf("%s: %w", msg, err),
		pcs: callers(2),
	}
}

I write errs.Wrap(err, "...") where fmt.Errorf("...: %w", err) would go. CaptureException walks the error chain, through errors.Join too, and sends the first stack it finds. With none, it sends the stack where it was called. In a deferred recover, PanicError keeps the frames of the line that panicked.

Titles and grouping

errors.New gives plain errors one type, *errors.errorString. fmt.Errorf can also wrap another error. errs titles a plain error by the phrase before its first ": ". For example, save note reminder: no recipient is titled save note reminder.

A phrase with a digit in it makes no title, because sockeye groups by title when an event has no stack frames.

When an error carries a stack trace, sockeye groups by the error type and the top three in-app stack frames. It ignores line numbers so code edits above a function do not split the group.

Never block the caller

A capture puts the event on a channel of 1,000 and returns. When the channel is full, the capture drops the event and increments a counter. One background worker sends each event, retries transient codes with backoff, and obeys Retry-After.

The client protects sockeye from error storms:

Flush in main sends what is left at shutdown. Tests set SyncDispatch, so each capture sends before the test reads it.

A frame is in_app when its function is in the main module. errs reads the module path from runtime/debug.ReadBuildInfo, so a new app needs no setting.

One policy for a 500

Every handler sends a failure to one function:

func WriteServerError(w http.ResponseWriter, err error) {
	if !errors.Is(err, context.Canceled) {
		errs.CaptureException(err)
	}
	log.Printf("WriteServerError: %s", sanitizeLogValue(err.Error()))
	http.Error(w, http.StatusText(500), 500)
}

The caller wraps the cause, so sockeye gets the whole chain, and the user sees a plain 500. A request that the browser cancels writes a log line and opens no group in sockeye. A test fails on a handler that writes a 5xx any other way.

Request metrics

RecordRequest sends one served request to sockeye's perf page, with the key in the same DSN. The request middleware calls it where it writes the request log line:

errs.RecordRequest(errs.Request{
	Route:    route,
	Pattern:  r.Pattern,
	Status:   status,
	Duration: time.Since(start),
	DB:       stats.Query,
	Acquire:  stats.Acquire,
	DBTimed:  true,
	Elapsed:  stats.Split(),
	Time:     start,
})

Pattern is the pattern the http.ServeMux matched, so sockeye charts one row for each route whatever ids the path holds.

DBTimed says that the service measured its query time and pool wait, as a pgx query tracer does (see go / postgres). A request with no queries then sends db_ms: 0. Without DBTimed, the client sends no db_ms, and sockeye shows the value as not measured, not as zero.

DB and Acquire are sums: queries that run at the same time each count. Elapsed is the wall time in which only queries ran, only pool waits ran, or both. The tracer counts the queries and waits in flight, and it starts and stops three clocks as the counts move. The three never overlap, so with the time in which neither ran they add up to the request.

The records travel apart from the errors. RecordRequest queues up to 5,000 records and never blocks. A sender posts a batch at 50 records, or every 10 seconds, and an empty batch for a quiet interval, so sockeye can tell quiet from missing. Each batch carries an Idempotency-Key, and a retry sends the same key and bytes, so sockeye counts a batch once. The sender has its own circuit breaker, so a broken perf endpoint does not stop errors. Sockeye refuses the batches with 409 until an operator names the service's producer, and the client discards them.

The service can also report slow requests as error events. RecordRequest itself sends metrics only.

Trade-offs

errs has no tracing and sends no automatic breadcrumbs. It sends only what each call passes in.

soc error reads the error groups back in the terminal.

← All articles