Imagine we are software engineers at a devtool SaaS organization. The software we write to support the organization will take the shape of multiple programs.
We work incrementally, choose technologies suited to their tasks that we also enjoy, and are unafraid to write custom software that improves our happiness and productivity.
We initialize a monorepo on GitHub:
. └── README.md
We write a
Add to your shell profile: # Set environment variable to monorepo path export ORG="$HOME/org" # Prepend monorepo scripts export PATH="$ORG/bin:$PATH" Clone: git clone https://github.com/org/repo.git $ORG cd $ORG
$ORG environment variable will be used throughout the codebase.
We design an architecture like this:
- a "Dashboard" web interface for customers (Svelte, TypeScript)
- SDKs for customers (Go, Node, Ruby)
- an HTTP server (Go)
- a Postgres database backing the HTTP API
. ├── ... ├── cmd │ └── serverd │ └── main.go ├── dashboard ├── sdk │ ├── go │ ├── node │ └── ruby └── server └── http_handler.go
We use commit message conventions with a subject line prefix:
$ git log dashboard: redirect to new project form after sign up sdk/node: enable Keep-Alive in HTTP agent server: document access to prod db
The prefix refers to the module that is changing. It often matches a filesystem directory. Sometimes the change will touch files across modules, but the prefix should be "where the action is."
We try to keep the subject line under 50 characters and err on the side of brevity for module names.
To aid local development and help onboard teammates,
we provide a
setup.sh script for each module.
In turn, each
setup.sh script might invoke more general
Since everyone has
$ORG/bin on their
bin/ are available in everyone's shells
without a directory prefix.
. ├── ... ├── bin │ ├── setup-go │ ├── setup-node │ ├── setup-postgres │ └── setup-ruby ├── dashboard │ └── setup.sh ├── sdk │ ├── go │ │ └── setup.sh │ ├── node │ │ └── setup.sh │ └── ruby │ └── setup.sh └── server └── setup.sh
. ├── ... ├── .rubocop.yml └── .prettierrc.yml
To move quickly without breaking things, we write automated tests and run those tests continuously when we integrate our changes.
We want our Continuous Integration (CI) service to:
- begin running the tests in < 5s after opening or editing a pull request
- run tests in an environment with parity to production
- run only the tests relevant to the change
Common causes of slow-starting tests on hosted CI services are multi-tenant queues and containerization: noisy neighbors backlog the queues and containers have cache misses.
To meet our design goals, we write our own CI service,
. ├── ... ├── cmd │ └── testbot │ └── main.go ├── dashboard │ └── Testfile ├── sdk │ ├── go │ │ └── Testfile │ ├── node │ │ └── Testfile │ └── ruby │ └── Testfile ├── server │ └── Testfile └── testbot ├── Testfile ├── farmer │ ├── Procfile │ └── main.go └── worker └── main.go
We open a GitHub pull request to add a new feature to the SDKs:
sdk/go/account.go | 10 +++++----- sdk/go/account_test.go | 10 +++++----- sdk/node/src/account.ts | 10 +++++----- sdk/node/test/account.ts | 10 +++++----- sdk/ruby/lib/account.rb | 10 +++++----- sdk/ruby/spec/account_spec.rb | 10 +++++----- 6 files changed, 60 insertions(+), 60 deletions(-)
testbot farmer process on a server
responds to the GitHub webhook by:
- identifying the directories containing files that have changed
- walking up the file hierarchy to find
Testfiles for changed directories
- saving test jobs to its backing Postgres database
Testfile defines test jobs for its directory. Ours might be:
$ cat $ORG/sdk/go/Testfile tests: cd $ORG/sdk && go test -cover ./... $ cat $ORG/sdk/node/Testfile tests: cd $ORG/sdk/node && npm install && npm test $ cat $ORG/sdk/ruby/Testfile tests: $ORG/sdk/ruby/test.sh
Testfile can define multiple test jobs.
As test scripts become lengthier,
it is convenient to extract them to their own script.
Each line contains a name and command
separated by a colon.
The name appears in GitHub checks.
The command is run by a
which is continuously long polling
testbot farmer via HTTP over TLS,
asking for test jobs.
testbot worker runs on Heroku with Go, Node, and Ruby buildpacks.
We run more instances to increase test parallelism.
In this example,
the tests for the Go, Node, and Ruby SDKs
will begin to run almost simultaneously
testbot worker processes pick them up.
server will not run in this pull request
because no files in their directories were changed.
Testing across services
To make it convenient to test across service boundaries,
we write a
with-serverd script that:
- installs the
- migrates the database
- creates a team and credential
serverd servewithout blocking programs passed as arguments to
We place this script in
$ORG/bin to make it available on our
This script can be used in
$ cat $ORG/dashboard/Testfile tests: cd $ORG/dashboard && with-serverd ./test.sh $ cat $ORG/sdk/go/Testfile tests: cd $ORG/sdk && with-serverd go test -cover ./...
Tests that depend on
can avoid mocking on HTTP boundaries,
making actual requests to the backing service
and generating observable logs.
To broadly cover the product's surface area,
we move the Go SDK's
to the top of the file hierarchy
so its test jobs will run on all pull requests:
. ├── ... ├── Testfile ├── bin │ └── with-serverd └── sdk └── go
So, we mirror the SDK code to open source repos. This provides community access. Although the community patch workflow is a bit manual, these are SDKs tightly coupled to our product and we expect community patches to be rare.
To meet our design goals, we write a
. ├── ... ├── .mirrorbot.yml └── cmd └── mirrorbot ├── Procfile ├── Testfile └── main.go
mirrorbot runs as a command-line program
that copies relevant changes from the
to one or more target repos.
It can be run on a laptop or a server.
it clones the monorepo and each target repo.
Each copied commit includes an
upstream:[SHA] line (example).
mirrorbot reads that SHA from the latest commit
on the destination branch of each target repo
to get its cursor.
Mirror then fast-forwards the local monorepo, looking for new commits.
For each target repo, a patch file is produced for each new commit. The patch file is then filtered to remove anything not applicable to that repo. If anything remains, the patch is applied and committed. Any new commits are then pushed to GitHub.
.mirrorbot.yml file configures the program.
It maps monorepo branches to target repo branches
and monorepo directories and files to target repo directories and files.
github.com/org/org-sdk-node: - branch: - main: main - mirror: - sdk/node: / github.com/org/org-sdk-ruby: - branch: - main: main - mirror: - sdk/ruby: /
As our customers use the software, we better understand their needs and begin to design v2.
To do this well, we add new interfaces and deprecate old interfaces in the v1.x series of the SDKs. The server continues to support the v1.x series now and for a well-communicated time period past the release of v2.0 (such as 90 days).
As we release minor versions v1.1 and v1.2, patch versions v1.2.1 and v1.2.2, and eventually v2, we want to carefully ensure backwards compatibility.
To meet our design goals,
. ├── ... └── bin ├── with-go-sdk ├── with-node-sdk └── with-ruby-sdk
We use these scripts to run processes using a given version of our SDKs:
with-go-sdk [version] [command] with-node-sdk [version] [command] with-ruby-sdk [version] [command]
version can be a released version to a registry such as NPM or Rubygems
or a special value
monorepo to use the current
$ORG source code.
with-ruby-sdk 1.0 ruby -e "require 'org-sdk'; puts OrgSDK::VERSION"
We update the
Testfile at the top of the file hierarchy
to test supported releases
and the current source code (pre-release)
on every pull request:
$ cat $ORG/Testfile gohead: with-serverd with-go-sdk monorepo go test -cover ./... gov1: with-serverd with-go-sdk 1.5 go test -cover ./... gov2: with-serverd with-go-sdk 2 go test -cover ./... # ... etc. rubyv2: with-serverd with-ruby-sdk 2 $ORG/sdk/ruby/test.sh
The SDK scripts are composable with the
In addition to running on CI, they are useful for quickly testing bug reports from customers on a particular version, and hopefully providing a great customer experience for them.
which expects a
Procfile manifest for each program:
. ├── ... ├── cmd │ ├── mirrorbot │ │ ├── Procfile │ │ └── main.go │ ├── serverd │ │ ├── Procfile │ │ └── main.go │ └── testbot │ ├── Procfile │ └── main.go
Heroku uses a stack of "buildpacks" to set up its build system.
We use a "monorepo" buildpack first in the stack,
which uses an
APP_BASE=subdir config variable to tell Heroku
where to find the
heroku buildpacks:add -a <app> https://github.com/lstoll/heroku-buildpack-monorepo heroku config:set APP_BASE=cmd/serverd -a <app>
We use the Go buildpack second in the stack,
which will install our Go program as a binary in
heroku buildpacks:add -a <app> heroku/go
Procfile contains, for example:
Working in a monorepo can encourage a feeling of "tight integration", where service boundaries are well-defined and less likely to be mocked out. Some tasks are a particular pleasure, such as searching across projects for callsites of a function or RPC.
For other tasks, it can help to write custom tools. Writing those custom tools offers an opportunity to design an ideal experience for the engineering team.
The final directory structure looks like this:
. ├── .mirrorbot.yml ├── .prettierrc.yml ├── .rubocop.yml ├── .tool-versions ├── README.md ├── Testfile ├── bin │ ├── setup-go │ ├── setup-node │ ├── setup-postgres │ ├── setup-ruby │ ├── with-go-sdk │ ├── with-node-sdk │ ├── with-ruby-sdk │ └── with-serverd ├── cmd │ ├── mirrorbot │ │ ├── Procfile │ │ ├── Testfile │ │ ├── main.go │ │ └── setup.sh │ ├── serverd │ │ ├── Procfile │ │ └── main.go │ └── testbot │ ├── Procfile │ └── main.go ├── dashboard │ ├── Testfile │ └── setup.sh ├── sdk │ ├── go │ │ └── setup.sh │ ├── node │ │ ├── Testfile │ │ └── setup.sh │ └── ruby │ ├── Testfile │ └── setup.sh ├── server │ ├── Testfile │ ├── http_handler.go │ └── setup.sh └── testbot ├── Testfile ├── farmer │ └── main.go ├── setup.sh └── worker └── main.go