apps / sockeye

updated October 30, 2026

sockeye is a development platform for private monorepos that I run on my own VMs. It hosts the git repos, runs their checks, and reviews and merges changes. It records each deploy, and it shows each service's logs, performance, errors, and uptime. Its site, sockeye.sh, has the install steps and the CLI reference.

It does the work that I would otherwise split across GitHub, a CI service, Sentry, a log search tool, and an uptime monitor. One Go binary, soc, is the server, the runner, and the CLI. One Postgres database holds the state, and bare git repos hold the code. An instance serves one organization, so it has no tenants and no billing tiers.

I judge a feature by what it lets an agent investigate. An agent starts with a question, such as why a route got slow after a deploy. It reads the change, the deploy, the errors, the logs, and the request metrics, and it follows one record to the next. So each record has a stable id, each read states the time it covers and what it dropped, and each export is complete.

An agent can compute more from the data that sockeye keeps, but it cannot recover a request that sockeye did not capture. So I state each limit: sockeye has no distributed tracing, and its request metrics are rollups for each route.

Checks

A Checkfile defines the checks for its directory. Each line has a name and a command, separated by a colon:

gotest: go test -race -cover ./...
lint: test -z "$(go tool goimports -l .)"

A push runs the Checkfile in each directory that holds a changed file, and in each directory above it. In this repo, a change to sdk/go/account.go runs the checks in sdk/go/, sdk/, and the root. It runs nothing in sdk/ruby/ or server/:

.
├── Checkfile
├── sdk
│   ├── Checkfile
│   ├── go
│   │   ├── Checkfile
│   │   └── account.go
│   └── ruby
│       └── Checkfile
└── server
    └── Checkfile

When a runner is free, a check starts within 1 second of a push. A runner runs each command in /bin/bash -eo pipefail. A check reports failed on a non-zero exit, and errored when the runner could not set up. The server keeps the output for 30 days. A repo pins its own tools, so a check runs the versions that the repo chose (see go / checks).

A check whose name starts with daily_ or weekly_ runs on a schedule against main, and a push does not run it. soc scheduled runs one on a change's head.

An artifact: line names a file that a build leaves. When the check passes, the runner uploads the file. The server stores it under the hash of the commit's tree, which the squash merge keeps. So a deploy installs the bytes that CI tested.

Changes

A change is the review unit. It has an id such as APP-42, which is also its branch name and its worktree directory. A change goes from start to merge like this:

soc checkout       # allocate APP-42, cut a worktree, print its path
soc edit           # title and description, before the work
git commit -am "retry: stop after five attempts"
soc push --wait    # push, then block until the checks finish
soc merge          # squash onto main

soc checkout cuts the worktree at ~/.worktrees/<repo>/<ID>. It symlinks an untracked .env from the main clone. Then it runs the command that the clone's tree.setup git config key names. One repo uses it to give each worktree its own database (see postgres / dev test clusters). The key is config, not a tracked hook, so a checkout of somebody else's change can't run their code.

I write the title and description before the work, so the dashboard says what each open change is.

Review

A review is one comment from the terminal. It has no inline comment, no thread, and no approval state. Each action has one way to do it, so an agent does it the same way every time:

soc show APP-42
git fetch origin && git diff origin/main...origin/APP-42
echo "the retry loop needs a ceiling" | soc comment APP-42

The server refuses a comment when the reviewer's commit is behind the change's head.

The web UI shows the state. Its only forms activate or deactivate an account, revoke a token, and sign in with a one-time code. For each other write, it shows the command to run. A write wakes every open page, and each live section fetches itself again (see go / wakeups and web / live regions).

Merge

soc merge checks that the change has a title, that it merges cleanly, and that every check on its head passed. An instance can also require a title pattern. soc merge --when-green waits for the checks first, and syncs the change again if main moved.

The server writes a squash commit with Co-Authored-By and Reviewed-by trailers, moves main, and backs the commit up. If the backup fails, main stays where it was.

Deploys

A repo can deploy on merge. The server queues the merged SHA in Postgres, the jobs process runs the repo's deploy command, and soc merge prints its output. An interrupted deploy has an unknown outcome, so I inspect it before a retry.

Other repos deploy with their own tool, which posts the SHA to sockeye when each service is live. sockeye records only the deploys that a tool posts.

soc deploy live    # per service: the live SHA and the changes in it
soc status         # each live deploy, with recent requests, errors, and p95

A change is live when its merge commit is an ancestor of the live SHA. So soc show APP-42 says where each change runs.

Logs

soc logs enable --unit names the systemd units that a service logs to. Its logs page then shows the journal with a live tail, a time range, and a search. The search reads key=value fields from logfmt and JSON lines, so status>=500 path=/login finds the failed logins. soc logs takes the same terms.

The server can't read the journal itself. A root relay reads it for the units in an allowlist (see unix / systemd).

Performance

The perf page draws requests by status class, the p50, p95, and p99 latency, and the database time per request, with a marker at each deploy. Its table has one row per route: the pattern that the Go http.ServeMux matched, such as GET /repos/{slug}. A click on a chart opens the log lines for that span (see web / charts).

The server parses the request lines in a service's journal, or a Go service posts its own records through errs. The latency charts leave out Server-Sent Events connections, because their duration is how long a client watched. A long poll, such as the one behind soc push --wait, gets a route row of its own.

Errors

A service posts its errors to the server with a key of its own. The server groups the events by fingerprint, or by type and top frames. The repo's errors tab and soc error show the groups:

soc error list           # open error groups
soc error show <ID>      # events, extras, and stack trace

A commit message with Fixes: err-12 resolves that group when the change merges. An event from a later release that holds the fix opens the group again.

Uptime and jobs

Each service has HTTP and TCP checks of its public URLs, which the deploy tool syncs with soc probe sync. Runners and Cloudflare Workers in other regions request each check every minute. A check's status is the worst runner answer in its newest minute. The server sends each alert to a webhook, with a guard against a check that flaps. A scheduled job requests /ping/<key> when it ends, and a missed ping opens an alert.

The jobs page lists each attempt at sockeye's own background work, such as the database snapshot and the restore rehearsal. The backups and the restore rehearsal each have a deadline, and a missed deadline opens an alert.

How it runs

One host runs two server processes, and Caddy sends requests to both. Postgres holds what the two share: the jobs, the live check output, and what each runner does. So either process can answer any request. A Postgres advisory lock serializes merges, and main moves by compare-and-set.

A deploy restarts one process at a time. It takes the process out of Caddy's list, waits for /healthz and /readyz on the new build, and puts the process back. Both run on one host, so this keeps the site up across a restart, not across a host failure. A separate soc jobs process holds another advisory lock and runs the recurring work.

Runners

A runner long polls the server for jobs. For each job, it checks out the job's tree and runs the command. A runner pings every second. After 15 seconds of silence, the server gives its job to another runner.

The runners share Go's build and module caches on a larger Ubicloud VM, in the server's private subnet. A runner's git fetch uses the libcurl resolve option to reach the server over that subnet, with the same HTTPS URL, certificate check, and token. A check command does not get that option, so a test of the public URL takes the public path.

Git and login

Each repo is a bare repo on the server's disk. The server runs git http-backend behind Caddy, which adds TLS. There is no ssh git transport. A pre-receive hook rejects direct writes to main.

On each machine, I point the CLI at the server, then log in:

soc git setup https://sockeye.example.com/git
soc login

soc login signs a challenge with a key from my ssh-agent and trades the signature for a token. The token lives seven days, and git reads it through soc auth git-credential. A server with WorkOS SSO signs in a browser through the organization's identity provider. A runner mints its read-only token with its host SSH key.

I run separate instances for personal projects and for work. They share no VM, database, or key. Inside a clone, the CLI talks to origin. Outside one, SOCKEYE_HOST names the server.

Mirrors and backups

For most repos, the server puts a git bundle of each squash commit in object storage, and a weekly job restores one (see git / restore rehearsals). A repo that go get or a deploy platform reads from GitHub gets a mirror of main there instead.

I develop hml, highlight, is, and errs on sockeye this way. GitHub receives main and the tags, never a change branch, so I close pull requests there.

← All articles