apps / sockeye
sockeye is a development platform that I run on my own VMs. It hosts
git repos, runs checks, reviews changes, and merges them. It records
each deploy, and tracks logs, performance, errors, and uptime.
It replaces GitHub, a CI service, Sentry, a log search tool, and an
uptime monitor. The Go binary soc is the server, the runner, and the
CLI. Postgres holds the state, and bare git repos hold the code. An
instance serves one organization.
I build features so an agent can investigate issues. When a route slows after a deploy, an agent reads the change, the deploy, the errors, the logs, and the metrics. Stable IDs and timestamps connect each record to the next.
Checks
A Checkfile defines the checks for its directory. Each line has a
name and a command, separated by a colon:
gotest: go test -race -cover ./...
lint: test -z "$(go tool goimports -l .)"
A push runs the Checkfile in each directory that holds a changed
file, and in each directory above it. In this repo, a change to
sdk/go/account.go runs the checks in sdk/go/, sdk/, and the root.
It runs nothing in sdk/ruby/ or server/:
.
├── Checkfile
├── sdk
│ ├── Checkfile
│ ├── go
│ │ ├── Checkfile
│ │ └── account.go
│ └── ruby
│ └── Checkfile
└── server
└── Checkfile
When a runner is free, a check starts within 1 second of a push. The
runner runs each command in /bin/bash -eo pipefail. A check reports
failed on a non-zero exit, and errored when the runner could not
set up. The server keeps the output for 30 days. A repo pins its own
tools (see go / checks).
A check whose name starts with daily_ or weekly_ runs on a schedule
against main. A push does not run it. soc scheduled runs one on a
change's head.
An artifact: line names a build artifact. When the check passes, the
runner uploads the file. The server stores it under the hash of the
commit's tree, which the squash merge keeps. So a deploy installs the
bytes that CI tested.
Changes
A change is the review unit. Its id, such as APP-42, is also its
branch name and its worktree directory. A change goes from start to
merge like this:
soc checkout # allocate APP-42, cut a worktree, print its path
soc edit # title and description, before the work
git commit -am "retry: stop after five attempts"
soc push --wait # push, then block until the checks finish
soc merge # squash onto main
soc checkout cuts the worktree at ~/.worktrees/<repo>/<ID> and
symlinks an untracked .env from the main clone. Then it runs the
command that the clone's tree.setup git config key names. One repo
uses it to give each worktree its own database (see
postgres / dev test clusters).
The key is config, not a tracked hook, so a checkout of somebody else's
change cannot run their code.
Review
A review is one comment from the terminal. It has no inline comment, no thread, and no approval state. Each action has one way to do it, so an agent does it the same way every time:
soc show APP-42
git fetch origin && git diff origin/main...origin/APP-42
echo "the retry loop needs a ceiling" | soc comment APP-42
The server refuses a comment when the reviewer's commit is behind the change's head.

The web UI shows the state. For a write, it shows the command to run. A write wakes every open page, and each live section fetches itself again (see go / wakeups and web / live regions).
Merge
soc merge checks that the change has a title, that it merges cleanly,
and that every check on its head passed. soc merge --when-green
waits for the checks first, and syncs the change again if main moved.
The server writes a squash commit with Co-Authored-By and
Reviewed-by trailers. It uploads the backup before it moves main.
A Postgres advisory lock for each repo keeps two merges from running
at once.
git update-ref <ref> <new> <old> moves main. Git applies it only
if the ref still holds <old>, so a merge built on a stale base moves
nothing. If a later step fails, the same call in reverse restores
<old>:
// Abridged from the server's merge function.
if _, err := git.Run(ctx, repo, "update-ref", baseRef, sha, baseSHA); err != nil {
return nil, fmt.Errorf("updating %s: %w", base, err)
}
rollback := func() {
git.Run(ctx, repo, "update-ref", baseRef, baseSHA, sha)
}
if target == "github" {
if err := pushMirror(ctx, repo, sha, baseRef); err != nil {
rollback()
return nil, err
}
}
if _, err := recordMerge(ctx, repo, num, sha); err != nil {
rollback()
return nil, err
}
Deploys
A repo can deploy on merge. The server queues the merged SHA in
Postgres, the jobs process runs the repo's deploy command, and
soc merge prints its output. An interrupted deploy has an unknown
outcome, so I inspect it before a retry.
Other repos deploy with their own tool, which posts the SHA to sockeye when each service is live. sockeye records only the deploys that a tool posts.
soc deploy live # per service: the live SHA and the changes in it
soc status # each live deploy, with recent requests, errors, and p95
A change is live when its merge commit is an ancestor of the live SHA.
So soc show APP-42 says where each change runs.
Logs
soc logs enable --unit names the systemd units that a service logs
to. Its logs page then shows the journal with a live tail, a time
range, and a search. The search reads key=value fields from logfmt and
JSON lines, so status>=500 path=/login finds the failed logins.
soc logs takes the same terms.

The server cannot read the journal itself. A root relay reads the units in an allowlist (see unix / systemd).
Performance
The perf page draws requests by status class, the p50, p95, and p99
latency, and the database time per request, with a marker at each
deploy. Its table has one row per route: the pattern that the Go
http.ServeMux matched, such as GET /repos/{slug}. A click on a chart
opens the log lines for that span (see
web / charts).

The server parses the request lines in a service's journal, or a Go
service posts its own records through errs.
The latency charts leave out Server-Sent Events connections, because
their duration is how long a client watched. A long poll, such as the
one behind soc push --wait, gets a route row of its own.
Errors
A service posts its errors to the server with a key of its own. The server groups the events by fingerprint. The frame key has the function and the file, not the line, so an edit above a frame does not split its group:
// Abridged from the server's errorFingerprint function.
if fp := ownFingerprint(e.Fingerprint); fp != "" {
return "own:" + hash(service, fp)
}
typ, value, frames := innermost(e)
var top []string
for i := len(frames) - 1; i >= 0 && len(top) < 3; i-- {
if frames[i].InApp {
top = append(top, frames[i].Function+"@"+frames[i].Filename)
}
}
if len(top) > 0 {
return "frames:" + hash(service, typ, strings.Join(top, "\x00"))
}
return "message:" + hash(service, typ, normalize(value))
The repo's errors tab and soc error show the groups:
soc error list # open error groups
soc error show <ID> # events, extras, and stack trace
A commit message with Fixes: err-12 resolves that group when the
change merges. An event from a later release that holds the fix opens
the group again.
Errors in Slack
I send error groups to Slack with a bot token and a channel. The Go service reports through errs, which sends to sockeye, not Slack. The service needs no Slack credential.
The image opens at full size. In Slack, each alert links to its own error group.
The link opens the group in sockeye. The message shows its title, event count, last-seen time, and release. The command reads its events and stack in the terminal. The server builds the Slack message from those fields:
// From the server's errorMessage function.
return slack.Message{
Text: state + ": " + sockeye.SlackLink(link, label),
Attachment: slack.Attachment{
Color: color,
Text: title + "\n" + strings.Join(parts, " · "),
Fallback: state + ": " + sockeye.SlackEscape(label) + ": " + title,
},
}
One parent message represents one error group. It is red while the group is open, and green when I resolve or ignore it. A state change also sends a thread reply, because Slack does not notify on an edit. A reopened error broadcasts that reply to the channel. More events update the count at most once a minute, without a reply for each event.
Every 10 seconds, the jobs process compares each group with the state
recorded for its Slack message. It saves that state after Slack
accepts the call. A failed call remains for a later pass, and a 429
pauses the loop for Slack's Retry-After. A restart keeps the saved
state. This is not exactly-once delivery: an accepted post whose
response is lost can produce a duplicate on a later pass.
I use this for error capture, grouping, and alerts. It does not add tracing or automatic breadcrumbs to errs.
Uptime and jobs
Each service has HTTP and TCP checks of its public URLs, which the
deploy tool syncs with soc probe sync. Runners and Cloudflare Workers
in other regions request each check every minute. A check's status is
the worst runner answer in its newest minute. The server sends each
alert to a webhook as JSON, with a guard against a check that flaps. A
scheduled job requests /ping/<key> when it ends, and a missed ping
opens an alert.
An outage uses the same Slack loop as an error group. Its message is red while the check fails and green when it recovers. State changes also send thread replies. Code that opens an alert does not call Slack.
The server cannot report its own outage. A systemd timer on a runner
host runs soc uptime each minute, which requests a public page of
the server. After two failures in a row, it posts the outage to Slack
itself.
The jobs page lists each attempt at sockeye's own background work, such as the database snapshot and the restore rehearsal. A missed deadline opens an alert.
How it runs
One host runs one server process behind Caddy. Postgres holds the
jobs, the changes, and what each runner does. The live check output
stays in the server's memory, and a runner sends its file again after
a restart. A separate soc jobs process holds an advisory lock and
runs the recurring work.
A deploy migrates the database while the old build serves, then
restarts the process in place. Caddy holds the requests that arrive
during the restart, so a reader sees a pause of about a second. The
server rolls back to the previous build if the new build never answers
/healthz. A systemd watchdog restarts a server that runs and stops
answering. One host keeps the site up across a restart, not across a
host failure.
The database is a managed Postgres with a standby. After a failover, the pool ends each connection to the old primary in seconds and dials the host name again (see go / postgres). If the database stays stuck for more than a minute, the watchdog restarts the server (see unix / systemd).
I build the server binary on my laptop and copy it to the server. Each
runner host fetches the new build over the private subnet, and checks
its sha256. I build the CLI with go install ./cmd/soc.
Runners
A runner long polls the server for jobs. For each job, it checks out the job's tree and runs the command. A runner pings every second. After 15 seconds of silence, the server gives its job to another runner.
The runner starts each check with systemd-run as a transient unit,
under a user for that runner slot. These properties close the paths
out of the job:
--property=User=<slot user>
--property=ProtectSystem=strict
--property=ReadWritePaths=<job dir> <slot Go build cache>
--property=BindReadOnlyPaths=<repo objects>
--property=InaccessiblePaths=<runner key> <token file> <runner state>
--property=PrivateTmp=yes
--property=NoNewPrivileges=yes
--property=KillMode=control-group
--property=RuntimeMaxSec=<timeout>
When the check ends, systemd kills each process in the unit's cgroup,
including a setsid child or a double fork.
The runner fills Go's module cache with go mod download, which checks
each module against go.sum. The check reads that cache with
GOPROXY=off.
Each runner slot keeps its own Go build cache between jobs. So
go build and go vet compile only what changed, and go test
replays unchanged packages (see go / test). One slot
runs one job at a time, so a cache has one writer. A check can still
write an entry that gives a later check on its slot a false pass. I
accept that risk, because a review gates each merge. A job that builds
a deploy artifact gets an empty cache.
The runners run on a larger
Ubicloud VM, in the server's private
subnet. A runner's git fetch reaches the server over that subnet with
the libcurl resolve option. A check command does not get that option,
so a test of the public URL takes the public path.
Git and login
Each repo is a bare repo on the server's disk. The server runs
git http-backend behind Caddy, which adds TLS. There is
no ssh git transport. A pre-receive hook rejects direct writes to
main. Only the server's merge process writes it, and that process
does not pass through the hook:
if [ "$old" != "$zero" ]; then
echo "sockeye: refusing to update $protected directly." >&2
echo "sockeye: it is append-only, written only by the server's merge process." >&2
exit 1
fi
On each machine, I point the CLI at the server, then log in:
soc git setup https://sockeye.example.com/git
soc login
soc login signs a challenge with a key from my ssh-agent and trades
the signature for a token. The token lives seven days, and git reads it
through soc auth git-credential. A server with WorkOS SSO signs in a
browser through the organization's identity provider. A runner mints
its read-only token with its host SSH key.
I run separate instances for personal projects and for work. They share
no VM, database, or key. Inside a clone, the CLI talks to origin.
Outside one, SOCKEYE_HOST names the server.
Mirrors and backups
For most repos, the server puts a git bundle of each squash commit in
object storage, and a weekly job restores one (see
git / restore rehearsals). A repo that
go get or a deploy platform reads from GitHub gets a mirror of main
there instead.
I develop hml, highlight,
is, and errs on sockeye this way.
GitHub receives main and the tags, never a change branch, so I close
pull requests there.
