git / restore rehearsals

updated October 14, 2026

I do not want my first restore of a backup to happen on the day I need it.

sockeye hosts my repositories on one VM, and the VM holds the only server copy. So once a week, sockeye restores its git backup and records the result. The same loop restores the database dump, which matters less.

What the bucket holds

The server sends two kinds of git bundle to the bucket through r2:

• Before a merge moves the base branch, the server bundles the new commits. This bundle keeps the squash commit, which no clone holds.
• Once a day, the server bundles every ref of each repository as full/<host>/<repo>-<hash>.bundle. The hash comes from the ref listing, so unchanged refs write to the same key. This bundle is the only copy of a change branch that has no clone.

Lifecycle rules on the bucket expire old bundles, so the bucket size stays steady.

The rehearsal

Once a week, for each repository, the server does these steps:

1. Download the newest full bundle to a temporary directory.
2. Run git clone --mirror on the bundle.
3. Run git fsck --no-dangling on the clone.
4. Compare the clone's git show-ref with the bare repository's.
5. Delete the temporary directory.

A mirror clone keeps every ref under its own name. So a missing branch shows as a ref that only one side has. The bundle is up to a day old, so some refs usually differ. The row records how many, and the rehearsal still passes. Only a failed step fails the rehearsal.

Each rehearsal has a ten-minute limit, so a stalled bucket causes a failure and not a hang. The server skips repositories that back up to GitHub.

The server records one row per rehearsal: the key, the size, the result, and one line of detail. A failure names the step that failed. /settings shows the newest row for each repository.

Why the server does it

A cron job can fail and nobody sees it. The rehearsal lives in the server, next to the code that writes the bundle, so the two change together. I already look at /settings.

The server restarts at every deploy. So it finds the next due time from the last row. A check based on uptime would rehearse at every deploy. A check based on a schedule would skip a week if the server was down at that hour.

Retry after a failure

The first version waited a week after a failed rehearsal. One rehearsal ran minutes after I created a repository, before its first daily bundle. It recorded "no full bundle in the bucket", and that row stayed for a week.

Now a failed row is due again after an hour, and a passing row after a week. A database read error also counts as due, because an extra rehearsal costs little.

What it proves

A pass proves that the newest bundle downloads, Git reads it, the objects pass fsck, and the refs match. It also proves that the key names, the bucket permissions, and the lifecycle rules still agree. If one of them changes, the next rehearsal fails before an emergency does.

A rehearsal does not move the recovery point. A restore from the bucket loses up to a day of pushes, minus what a developer clone holds. For my repositories, that loss is acceptable.

The database dump

The database runs on a managed Postgres service, and the provider's backups are the primary defense. I keep one more copy that the provider does not hold.

Every hour, the server runs pg_dump in custom format and uploads the file to an hourly prefix. The first dump of each UTC day also goes to a daily prefix. The weekly loop downloads the newest dump and restores it into a scratch database with pg_restore --no-owner --exit-on-error. Then it counts the rows of one table and drops the database.

Without --exit-on-error, pg_restore skips a failed statement and exits with success. A half-restored database then looks like a good one.

The server's database role owns the scratch database, so the role needs CREATEDB. A dump from a newer server needs a newer pg_restore, so the VM has the client for the server's major version.

When I wrote this, the loop had not yet rehearsed the Postgres dump. I proved the restore by hand when I moved the server to Postgres.

Beyond git

Nothing in the loop is specific to git. Any file that I back up to the bucket needs the same parts: a copy on a schedule, a restore on a schedule, and a row with the result. I may extend the loop to other files on my VMs.

The dev cluster habit

My development cluster restores a production dump each time I refresh it (see restoring production). I do this often, so it catches a bad dump for my other apps. No laptop touches the sockeye repositories or database, so the weekly rehearsal does that job for them.

← All articles