cmd / r2
r2 is a backup tool and a Cloudflare R2 client. It snapshots SQLite
databases and bundles git repositories, and it puts, gets, and lists
objects. I wrote it in Go with the standard library only.
Why I wrote it
My VM holds the primary copy of my applications, and the provider sells no disk snapshots. A disk failure would take the databases and the repositories with it, so the copies have to live off the box.
The AWS SDK covers that in four API calls and a large dependency. I wrote the four calls instead.
Commands
r2 backup sqlite db.db # gzipped snapshot, uploaded as db-<UTCts>.db.gz
r2 backup git repo/ # bundle of the branch, as repo-<UTCts>.bundle
r2 put file [key] # upload a file
r2 get key [dest] # download an object
r2 ls [prefix] # list keys with sizes and timesThe SQLite backup shells out to sqlite3 .backup, so the snapshot stays
consistent while the application writes. I describe the database options
in go / sqlite. The git backup shells out to git bundle,
and git clone restores the bundle.
Signing requests from the spec
R2 speaks the S3 API, and an S3 request carries a Signature Version 4
signature. The SDK hides that work. r2 implements it from the AWS
documentation in 123 lines.
The signer builds a canonical request from the method, path, query, and
headers, then hashes it into a string to sign. It derives the signing
key through four HMAC rounds: date, region, service, and
aws4_request. It signs the string and sets the Authorization header.
R2 names its region auto.
AWS publishes test vectors for SigV4, and the repository keeps them
under testdata/. Each vector fixes the request and the clock, and the
test compares the signature byte for byte. A signer that passes the
vectors reaches R2 on the first request.
Design
One PUT per object, and no multipart upload. A single PUT carries a file up to 5 GB, and my artifacts are megabytes.
No pruning logic. A lifecycle rule on the bucket expires an object after 30 days, so retention lives next to the data.
One bucket per VM, with each application as a key prefix:
sports/db-<ts>.db.gz, sports/repo-<ts>.bundle,
r2/repo-<ts>.bundle. The API token accepts requests from the VM
addresses only.
Four environment variables configure the tool: R2_ENDPOINT,
R2_BUCKET, R2_ACCESS_KEY_ID, and R2_SECRET_ACCESS_KEY. An optional
R2_PREFIX prepends the application prefix to a generated key. The tool
exits when one of the four is missing.
How sports uses it
The scoreboard backs up through a shell script
that calls r2 put. Three triggers fire the script: a git post-commit
hook after each commit, the application after a clean ingest or a saved
summary, and an hourly cron job as the floor. A commit reaches the
bucket in seconds.
A restore reads the most recent bundle and snapshot:
r2 ls sports/
r2 get sports/repo-<ts>.bundle
git clone repo-<ts>.bundle sportsWhat it is not
r2 is not a sync tool and not a log streamer.
Litestream ships each write-ahead log frame
and restores to any second. r2 ships point-in-time artifacts and
restores to the last trigger. I tolerate a recovery point of minutes, so
I run the simpler tool.