go / errs
errs is my error client for Go services. It uses the standard library only. It sends errors to sockeye.
I use sockeye for error tracking, so a few hundred lines cover the client.
API
errs.Init(errs.Options{
DSN: os.Getenv("SOCKEYE_DSN"),
Environment: "production",
Release: os.Getenv("RENDER_GIT_COMMIT"),
})
defer errs.Flush(2 * time.Second)
errs.CaptureMessage("did something weird", errs.WithExtra("k", v))
errs.CaptureException(err, errs.WithFingerprint("apollo", "429"))
WithTag, WithExtra, WithFingerprint, WithRequestID, and
WithLevel add metadata to one event.
An empty DSN makes every capture a no-op, so dev runs and tests stay
quiet. Init returns an error only for a malformed DSN.
Attaching a stack
A Go error has no stack trace. Wrap and Errorf capture the program
counters where they run:
func Wrap(err error, msg string) error {
if err == nil {
return nil
}
return &stackErr{
err: fmt.Errorf("%s: %w", msg, err),
pcs: callers(2),
}
}
I write errs.Wrap(err, "...") where fmt.Errorf("...: %w", err)
would go. CaptureException walks the error chain, through
errors.Join too, and sends the first stack it finds. With none, it
sends the stack where it was called. In a deferred recover,
PanicError keeps the frames of the line that panicked.
Titles and grouping
errors.New gives plain errors one type, *errors.errorString.
fmt.Errorf can also wrap another error. errs titles a plain error
by the phrase before its first ": ". For example,
save note reminder: no recipient is titled save note reminder.
A phrase with a digit in it makes no title, because sockeye groups by title when an event has no stack frames.
When an error carries a stack trace, sockeye groups by the error type and the top three in-app stack frames. It ignores line numbers so code edits above a function do not split the group.
Never block the caller
A capture puts the event on a channel of 1,000 and returns. When the
channel is full, the capture drops the event and increments a counter.
One background worker sends each event, retries transient codes with
backoff, and obeys Retry-After.
The client protects sockeye from error storms:
- The first 10 events of one error in a minute go out whole.
- After that, the client sends one event every 10 seconds with a count of held events.
- After 5 failed sends in a row, the circuit breaker opens. It pauses for 30 seconds, 1 minute, 2 minutes, 5 minutes, and 10 minutes on consecutive probe failures.
Flush in main sends what is left at shutdown. Tests set
SyncDispatch, so each capture sends before the test reads it.
A frame is in_app when its function is in the main module. errs
reads the module path from runtime/debug.ReadBuildInfo, so a new
app needs no setting.
One policy for a 500
Every handler sends a failure to one function:
func WriteServerError(w http.ResponseWriter, err error) {
if !errors.Is(err, context.Canceled) {
errs.CaptureException(err)
}
log.Printf("WriteServerError: %s", sanitizeLogValue(err.Error()))
http.Error(w, http.StatusText(500), 500)
}
The caller wraps the cause, so sockeye gets the whole chain, and the user sees a plain 500. A request that the browser cancels writes a log line and opens no group in sockeye. A test fails on a handler that writes a 5xx any other way.
Request metrics
RecordRequest sends one served request to sockeye's
perf page, with the key in the
same DSN. The request middleware calls it where it writes the request
log line:
errs.RecordRequest(errs.Request{
Route: route,
Pattern: r.Pattern,
Status: status,
Duration: time.Since(start),
DB: stats.Query,
Acquire: stats.Acquire,
DBTimed: true,
Elapsed: stats.Split(),
Time: start,
})
Pattern is the pattern the http.ServeMux matched, so sockeye
charts one row for each route whatever ids the path holds.
DBTimed says that the service measured its query time and pool
wait, as a pgx query tracer does (see
go / postgres). A request with no queries
then sends db_ms: 0. Without DBTimed, the client sends no db_ms,
and sockeye shows the value as not measured, not as zero.
DB and Acquire are sums: queries that run at the same time each
count. Elapsed is the wall time in which only queries ran, only pool
waits ran, or both. The tracer counts the queries and waits in flight,
and it starts and stops three clocks as the counts move. The three
never overlap, so with the time in which neither ran they add up to
the request.
The records travel apart from the errors. RecordRequest queues up to
5,000 records and never blocks. A sender posts a batch at 50 records,
or every 10 seconds, and an empty batch for a quiet interval, so
sockeye can tell quiet from missing. Each batch carries an
Idempotency-Key, and a retry sends the same key and bytes, so
sockeye counts a batch once. The sender has its own circuit breaker,
so a broken perf endpoint does not stop errors. Sockeye refuses the
batches with 409 until an operator names the service's producer, and
the client discards them.
The service can also report slow requests as error events.
RecordRequest itself sends metrics only.
Trade-offs
errs has no tracing and sends no automatic breadcrumbs. It sends only
what each call passes in.
soc error reads the error groups back in the terminal.