apps / sockeye

sockeye is a development platform that I run on my own VMs. It hosts git repos, runs checks, reviews changes, and merges them. It records each deploy, and tracks logs, performance, errors, and uptime.

It replaces GitHub, a CI service, Sentry, a log search tool, and an uptime monitor. The Go binary soc is the server, the runner, and the CLI. Postgres holds the state, and bare git repos hold the code. An instance serves one organization.

I build features so an agent can investigate issues. When a route slows after a deploy, an agent reads the change, the deploy, the errors, the logs, and the metrics. Stable IDs and timestamps connect each record to the next.

Checks

A Checkfile defines the checks for its directory. Each line has a name and a command, separated by a colon:

gotest: go test -race -cover ./...
lint: test -z "$(go tool goimports -l .)"

A push runs the Checkfile in each directory that holds a changed file, and in each directory above it. In this repo, a change to sdk/go/account.go runs the checks in sdk/go/, sdk/, and the root. It runs nothing in sdk/ruby/ or server/:

.
├── Checkfile
├── sdk
│   ├── Checkfile
│   ├── go
│   │   ├── Checkfile
│   │   └── account.go
│   └── ruby
│       └── Checkfile
└── server
    └── Checkfile

When a runner is free, a check starts within 1 second of a push. The runner runs each command in /bin/bash -eo pipefail. A check reports failed on a non-zero exit, and errored when the runner could not set up. The server keeps the output for 30 days. A repo pins its own tools (see go / checks).

A check whose name starts with daily_ or weekly_ runs on a schedule against main. A push does not run it. soc scheduled runs one on a change's head.

An artifact: line names a build artifact. When the check passes, the runner uploads the file. The server stores it under the hash of the commit's tree, which the squash merge keeps. So a deploy installs the bytes that CI tested.

Changes

A change is the review unit. Its id, such as APP-42, is also its branch name and its worktree directory. A change goes from start to merge like this:

soc checkout       # allocate APP-42, cut a worktree, print its path
soc edit           # title and description, before the work
git commit -am "retry: stop after five attempts"
soc push --wait    # push, then block until the checks finish
soc merge          # squash onto main

soc checkout cuts the worktree at ~/.worktrees/<repo>/<ID> and symlinks an untracked .env from the main clone. Then it runs the command that the clone's tree.setup git config key names. One repo uses it to give each worktree its own database (see postgres / dev test clusters). The key is config, not a tracked hook, so a checkout of somebody else's change cannot run their code.

Review

A review is one comment from the terminal. It has no inline comment, no thread, and no approval state. Each action has one way to do it, so an agent does it the same way every time:

soc show APP-42
git fetch origin && git diff origin/main...origin/APP-42
echo "the retry loop needs a ceiling" | soc comment APP-42

The server refuses a comment when the reviewer's commit is behind the change's head.

The page for one change: its title and description, the services where it is live, the diff by file, and each check with its time

The web UI shows the state. For a write, it shows the command to run. A write wakes every open page, and each live section fetches itself again (see go / wakeups and web / live regions).

Merge

soc merge checks that the change has a title, that it merges cleanly, and that every check on its head passed. soc merge --when-green waits for the checks first, and syncs the change again if main moved.

The server writes a squash commit with Co-Authored-By and Reviewed-by trailers. It uploads the backup before it moves main. A Postgres advisory lock for each repo keeps two merges from running at once.

git update-ref <ref> <new> <old> moves main. Git applies it only if the ref still holds <old>, so a merge built on a stale base moves nothing. If a later step fails, the same call in reverse restores <old>:

// Abridged from the server's merge function.
if _, err := git.Run(ctx, repo, "update-ref", baseRef, sha, baseSHA); err != nil {
	return nil, fmt.Errorf("updating %s: %w", base, err)
}
rollback := func() {
	git.Run(ctx, repo, "update-ref", baseRef, baseSHA, sha)
}
if target == "github" {
	if err := pushMirror(ctx, repo, sha, baseRef); err != nil {
		rollback()
		return nil, err
	}
}
if _, err := recordMerge(ctx, repo, num, sha); err != nil {
	rollback()
	return nil, err
}

Deploys

A repo can deploy on merge. The server queues the merged SHA in Postgres, the jobs process runs the repo's deploy command, and soc merge prints its output. An interrupted deploy has an unknown outcome, so I inspect it before a retry.

Other repos deploy with their own tool, which posts the SHA to sockeye when each service is live. sockeye records only the deploys that a tool posts.

soc deploy live    # per service: the live SHA and the changes in it
soc status         # each live deploy, with recent requests, errors, and p95

A change is live when its merge commit is an ancestor of the live SHA. So soc show APP-42 says where each change runs.

Logs

soc logs enable --unit names the systemd units that a service logs to. Its logs page then shows the journal with a live tail, a time range, and a search. The search reads key=value fields from logfmt and JSON lines, so status>=500 path=/login finds the failed logins. soc logs takes the same terms.

The logs page in live tail: one request line per row, with the time, the server process, the status, the path, and the duration

The server cannot read the journal itself. A root relay reads the units in an allowlist (see unix / systemd).

Performance

The perf page draws requests by status class, the p50, p95, and p99 latency, and the database time per request, with a marker at each deploy. Its table has one row per route: the pattern that the Go http.ServeMux matched, such as GET /repos/{slug}. A click on a chart opens the log lines for that span (see web / charts).

The perf page for a web app over seven days: requests by status class, the p50, p95, and p99 response time, and the database work per request, with a marker at each deploy

The server parses the request lines in a service's journal, or a Go service posts its own records through errs. The latency charts leave out Server-Sent Events connections, because their duration is how long a client watched. A long poll, such as the one behind soc push --wait, gets a route row of its own.

Errors

A service posts its errors to the server with a key of its own. The server groups the events by fingerprint. The frame key has the function and the file, not the line, so an edit above a frame does not split its group:

// Abridged from the server's errorFingerprint function.
if fp := ownFingerprint(e.Fingerprint); fp != "" {
	return "own:" + hash(service, fp)
}
typ, value, frames := innermost(e)
var top []string
for i := len(frames) - 1; i >= 0 && len(top) < 3; i-- {
	if frames[i].InApp {
		top = append(top, frames[i].Function+"@"+frames[i].Filename)
	}
}
if len(top) > 0 {
	return "frames:" + hash(service, typ, strings.Join(top, "\x00"))
}
return "message:" + hash(service, typ, normalize(value))

The repo's errors tab and soc error show the groups:

soc error list           # open error groups
soc error show <ID>      # events, extras, and stack trace

A commit message with Fixes: err-12 resolves that group when the change merges. An event from a later release that holds the fix opens the group again.

Errors in Slack

I send error groups to Slack with a bot token and a channel. The Go service reports through errs, which sends to sockeye, not Slack. The service needs no Slack credential.

Two Sockeye error alerts in Slack, each with a red bar, an error message, an event count, a release, and a command to inspect the group

The image opens at full size. In Slack, each alert links to its own error group.

The link opens the group in sockeye. The message shows its title, event count, last-seen time, and release. The command reads its events and stack in the terminal. The server builds the Slack message from those fields:

// From the server's errorMessage function.
return slack.Message{
	Text: state + ": " + sockeye.SlackLink(link, label),
	Attachment: slack.Attachment{
		Color:    color,
		Text:     title + "\n" + strings.Join(parts, " · "),
		Fallback: state + ": " + sockeye.SlackEscape(label) + ": " + title,
	},
}

One parent message represents one error group. It is red while the group is open, and green when I resolve or ignore it. A state change also sends a thread reply, because Slack does not notify on an edit. A reopened error broadcasts that reply to the channel. More events update the count at most once a minute, without a reply for each event.

Every 10 seconds, the jobs process compares each group with the state recorded for its Slack message. It saves that state after Slack accepts the call. A failed call remains for a later pass, and a 429 pauses the loop for Slack's Retry-After. A restart keeps the saved state. This is not exactly-once delivery: an accepted post whose response is lost can produce a duplicate on a later pass.

I use this for error capture, grouping, and alerts. It does not add tracing or automatic breadcrumbs to errs.

Uptime and jobs

Each service has HTTP and TCP checks of its public URLs, which the deploy tool syncs with soc probe sync. Runners and Cloudflare Workers in other regions request each check every minute. A check's status is the worst runner answer in its newest minute. The server sends each alert to a webhook as JSON, with a guard against a check that flaps. A scheduled job requests /ping/<key> when it ends, and a missed ping opens an alert.

An outage uses the same Slack loop as an error group. Its message is red while the check fails and green when it recovers. State changes also send thread replies. Code that opens an alert does not call Slack.

The server cannot report its own outage. A systemd timer on a runner host runs soc uptime each minute, which requests a public page of the server. After two failures in a row, it posts the outage to Slack itself.

The jobs page lists each attempt at sockeye's own background work, such as the database snapshot and the restore rehearsal. A missed deadline opens an alert.

How it runs

One host runs one server process behind Caddy. Postgres holds the jobs, the changes, and what each runner does. The live check output stays in the server's memory, and a runner sends its file again after a restart. A separate soc jobs process holds an advisory lock and runs the recurring work.

A deploy migrates the database while the old build serves, then restarts the process in place. Caddy holds the requests that arrive during the restart, so a reader sees a pause of about a second. The server rolls back to the previous build if the new build never answers /healthz. A systemd watchdog restarts a server that runs and stops answering. One host keeps the site up across a restart, not across a host failure.

The database is a managed Postgres with a standby. After a failover, the pool ends each connection to the old primary in seconds and dials the host name again (see go / postgres). If the database stays stuck for more than a minute, the watchdog restarts the server (see unix / systemd).

I build the server binary on my laptop and copy it to the server. Each runner host fetches the new build over the private subnet, and checks its sha256. I build the CLI with go install ./cmd/soc.

Runners

A runner long polls the server for jobs. For each job, it checks out the job's tree and runs the command. A runner pings every second. After 15 seconds of silence, the server gives its job to another runner.

The runner starts each check with systemd-run as a transient unit, under a user for that runner slot. These properties close the paths out of the job:

--property=User=<slot user>
--property=ProtectSystem=strict
--property=ReadWritePaths=<job dir> <slot Go build cache>
--property=BindReadOnlyPaths=<repo objects>
--property=InaccessiblePaths=<runner key> <token file> <runner state>
--property=PrivateTmp=yes
--property=NoNewPrivileges=yes
--property=KillMode=control-group
--property=RuntimeMaxSec=<timeout>

When the check ends, systemd kills each process in the unit's cgroup, including a setsid child or a double fork.

The runner fills Go's module cache with go mod download, which checks each module against go.sum. The check reads that cache with GOPROXY=off.

Each runner slot keeps its own Go build cache between jobs. So go build and go vet compile only what changed, and go test replays unchanged packages (see go / test). One slot runs one job at a time, so a cache has one writer. A check can still write an entry that gives a later check on its slot a false pass. I accept that risk, because a review gates each merge. A job that builds a deploy artifact gets an empty cache.

The runners run on a larger Ubicloud VM, in the server's private subnet. A runner's git fetch reaches the server over that subnet with the libcurl resolve option. A check command does not get that option, so a test of the public URL takes the public path.

Git and login

Each repo is a bare repo on the server's disk. The server runs git http-backend behind Caddy, which adds TLS. There is no ssh git transport. A pre-receive hook rejects direct writes to main. Only the server's merge process writes it, and that process does not pass through the hook:

if [ "$old" != "$zero" ]; then
  echo "sockeye: refusing to update $protected directly." >&2
  echo "sockeye: it is append-only, written only by the server's merge process." >&2
  exit 1
fi

On each machine, I point the CLI at the server, then log in:

soc git setup https://sockeye.example.com/git
soc login

soc login signs a challenge with a key from my ssh-agent and trades the signature for a token. The token lives seven days, and git reads it through soc auth git-credential. A server with WorkOS SSO signs in a browser through the organization's identity provider. A runner mints its read-only token with its host SSH key.

I run separate instances for personal projects and for work. They share no VM, database, or key. Inside a clone, the CLI talks to origin. Outside one, SOCKEYE_HOST names the server.

Mirrors and backups

For most repos, the server puts a git bundle of each squash commit in object storage, and a weekly job restores one (see git / restore rehearsals). A repo that go get or a deploy platform reads from GitHub gets a mirror of main there instead.

I develop hml, highlight, is, and errs on sockeye this way. GitHub receives main and the tags, never a change branch, so I close pull requests there.

← All articles