git / restore rehearsals

updated October 30, 2026

sockeye hosts my git repositories on one virtual machine. Once a week, sockeye restores its git backup and records the result. A similar loop tests the database backup.

What the bucket holds

The server sends two types of git bundles to object storage through r2:

Before a merge moves the base branch, the server bundles new commits to preserve the squash commit.
Once a day, the server bundles every ref of each repository as full/<host>/<repo>-<hash>.bundle. The hash comes from the ref listing, so unchanged refs write to the same key.

Lifecycle rules on the bucket expire old bundles to keep storage steady.

The rehearsal

Once a week, the server tests each repository:

func (fm *server) tryRestore(ctx context.Context, slug, key string) (string, error) {
	dir, err := os.MkdirTemp("", "sockeye-rehearsal-*")
	if err != nil {
		return "", err
	}
	defer os.RemoveAll(dir)

	bundle := filepath.Join(dir, "repo.bundle")
	cmd := exec.CommandContext(ctx, fm.r2Path, "get", key, bundle)
	if out, err := cmd.CombinedOutput(); err != nil {
		return "", fmt.Errorf("fetching %s: %w: %s", key, err, strings.TrimSpace(string(out)))
	}

	clone := filepath.Join(dir, "repo")
	if out, err := exec.CommandContext(ctx, "git", "clone", "--mirror", bundle, clone).CombinedOutput(); err != nil {
		return "", fmt.Errorf("cloning %s: %w: %s", key, err, strings.TrimSpace(string(out)))
	}
	if out, err := exec.CommandContext(ctx, "git", "-C", clone, "fsck", "--no-dangling").CombinedOutput(); err != nil {
		return "", fmt.Errorf("checking %s: %w: %s", key, err, strings.TrimSpace(string(out)))
	}
	out, err := exec.CommandContext(ctx, "git", "-C", clone, "show-ref").Output()
	if err != nil {
		return "", fmt.Errorf("reading refs: %w", err)
	}
	want, err := fm.runGit(ctx, slug, "show-ref")
	if err != nil {
		return "", fmt.Errorf("reading bare repo refs: %w", err)
	}
	if string(out) == want {
		return "restored; show-ref matches the bare repo", nil
	}
	return fmt.Sprintf("restored; bare repo moved: %d refs differ",
		countRefDiffs(string(out), want)), nil
}

A mirror clone preserves every ref. A bundle is up to a day old, so active repositories show newer refs. The rehearsal records the ref difference and marks success unless a command returns an error. Each run has a ten-minute timeout and logs status to /settings.

Scheduling and retries

The server schedules rehearsals from the database record of the last run:

func (fm *server) rehearsalDue(ctx context.Context, target string) bool {
	var last *Timestamp
	var ok bool
	err := fm.db.QueryRowContext(ctx, qFetchLastBackupRehearsal, target).Scan(&last, &ok)
	if errors.Is(err, sql.ErrNoRows) {
		return true
	}
	if err != nil {
		log.Error(ctx, err, "rehearsalDue of ", target)
		return true
	}
	if !ok {
		return time.Since(last.Time) >= rehearseWake
	}
	return time.Since(last.Time) >= rehearseEvery
}

A passing row waits one week (rehearseEvery). A failed row retries after one hour (rehearseWake). A database query error also triggers a retry.

What it proves

A pass proves that the bundle downloads, Git reads it, objects pass fsck, and refs match. It verifies key names, permissions, and bucket lifecycle rules. A restore from object storage recovers data up to the most recent daily bundle.

The database dump

Every hour, the server runs pg_dump and uploads the archive to object storage. The weekly rehearsal restores the dump into a temporary database:

func (fm *server) tryRestoreDatabase(ctx context.Context, key string) (string, error) {
	dir, err := os.MkdirTemp("", "sockeye-db-rehearsal-*")
	if err != nil {
		return "", err
	}
	defer os.RemoveAll(dir)

	dumpPath := filepath.Join(dir, "soc.dump")
	cmd := exec.CommandContext(ctx, fm.r2Path, "get", key, dumpPath)
	if out, err := cmd.CombinedOutput(); err != nil {
		return "", fmt.Errorf("fetching %s: %w: %s", key, err, strings.TrimSpace(string(out)))
	}

	scratch := rehearsalPrefix + strconv.FormatInt(time.Now().UnixNano(), 36)
	scratchURL, err := withDatabase(fm.dbURL, scratch)
	if err != nil {
		return "", err
	}
	if _, err := fm.db.ExecContext(ctx, "CREATE DATABASE "+scratch); err != nil {
		return "", fmt.Errorf("creating %s: %w", scratch, err)
	}
	defer fm.db.ExecContext(context.Background(), "DROP DATABASE IF EXISTS "+scratch+" WITH (FORCE)")

	restore := exec.CommandContext(ctx, "pg_restore", "--no-owner", "--exit-on-error",
		"--dbname="+scratchURL, dumpPath)
	if out, err := restore.CombinedOutput(); err != nil {
		return "", fmt.Errorf("restoring %s: %w: %s", key, err, strings.TrimSpace(string(out)))
	}

	db, err := openDB(scratchURL)
	if err != nil {
		return "", fmt.Errorf("opening restored database %s: %w", key, err)
	}
	defer db.Close()

	var changes int
	if err := db.QueryRowContext(ctx, qCountChanges).Scan(&changes); err != nil {
		return "", fmt.Errorf("reading changes from %s: %w", key, err)
	}

	return fmt.Sprintf("restored; %s", plural(changes, "change", "changes")), nil
}

--exit-on-error fails pg_restore on any statement error. The function verifies table rows by reading the change count, then drops the scratch database.

← All articles