Heroku Postgres Read Replica with Rails
As a production Ruby on Rails app on Heroku scales, you may want to move some database read load to a replica in order to keep some web processes and their database connections free for other read and write throughput.
As of Rails 6, the framework offers this via built-in facilities. This article demonstrates one possible configuration of these facilities.
In this configuration, automatic connection switching is not enabled. Instead, the replica is used at targeted call sites in the Rails app.
For example, in ThingsController#index
:
def index
ApplicationRecord.read_only do
@things = Thing.order(created_at: :desc)
end
end
This targeted approach is well-suited for offloading admin actions or queries that don't need to be consistent up to the second.
Both this targeted approach and the automatic connection switching approach are not well-suited for user flows where a user makes a write action and needs to immediately see the read result because of the race condition created by a Heroku Postgres database under load replicating while the read query comes in.
This is app/models/application_record.rb
:
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
FOLLOWER_COLOR = ENV["DATABASE_FOLLOWER_COLOR"].to_s.upcase
FOLLOWER_URL = ENV["HEROKU_POSTGRESQL_#{FOLLOWER_COLOR}_URL"]
if FOLLOWER_URL.present?
connects_to database: {writing: :primary, reading: :follower}
end
def self.read_only
if FOLLOWER_URL.present?
ActiveRecord::Base.connected_to(role: :reading) do
# All code in this block will run against the follower database.
# If a write is attempted, an ActiveRecord::ReadOnlyError will raise.
yield
end
else
# All code in this block will run against the only configured database.
# Database roles `primary` and `follower` are not available.
yield
end
end
end
Heroku Postgres assigns a "color" identifier to each database
that is guaranteed to be unique within the Heroku app.
ActiveRecord::ConnectionHandling#connects_to
is only invoked if the follower database URL is present.
This is important because ActiveRecord will spawn a database pool for
each database "role" of writing
and reading
in this case.
If the configuration instead had a fallback for follower
to
the primary
Heroku database at DATABASE_URL
,
we would double our database connections,
which max out at 500 at the highest Heroku Postgres plans.
The primary
and follower
databases are therefore
only defined in config/database.yml
based on the same logic,
depending on the presence of the follower database URL:
<% pool = ENV.fetch("PUMA_THREADS", 5) %>
<% follower_color = ENV["DATABASE_FOLLOWER_COLOR"].to_s.upcase %>
<% follower_url = ENV["HEROKU_POSTGRESQL_#{follower_color}_URL"] %>
default: &default
adapter: postgresql
encoding: unicode
pool: <%= pool %>
development:
primary:
<<: *default
database: multi_db_development
follower:
<<: *default
database: multi_db_development
replica: true
test:
<<: *default
database: multi_db_test
production:
<% if follower_url.present? %>
primary:
<<: *default
follower:
<<: *default
replica: true
url: <%= follower_url %>
<% else %>
<<: *default
<% end %>
For simplicity,
the database pool is set to use PUMA_THREADS
and
all configurable environment variables are set to 5 by default.
DATABASE_FOLLOWER_COLOR=SILVER
PUMA_WORKERS=5
PUMA_THREADS=5
See config/puma.rb
:
workers ENV.fetch("PUMA_WORKERS", 5).to_i
threads_count = ENV.fetch("PUMA_THREADS", 5).to_i
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV.fetch("PORT", 3000)
environment ENV.fetch("RAILS_ENV", "development")
on_worker_boot do
ActiveRecord::Base.establish_connection
end
If your app is considering a read replica, you may also be using
Heroku's Performance dynos.
With PUMA_WORKERS=5
, at 400MB of memory for each Rails web
process,
you'd be at 80% capacity using Performance-M dynos (2.5GB memory each).
The cheapest plan with 500 connections is a
standard-3
Heroku Postgres database,
offering 15GB RAM and 512GB storage.
A calculation like the following can help set an upper limit
on the number of dynos for
Heroku Autoscaling.
500 maximum database connections =
(up to 25 connections for Heroku internal use) +
(19 web dynos * 5 Puma workers * 5 Puma threads)
Test the number of connections by opening a Postgres prompt
using heroku pg:psql
and running:
SELECT
count(*)
FROM
pg_stat_activity
WHERE
pid <> pg_backend_pid()
AND usename = current_user;
In another shell, send traffic to the app with a tool like Apache Bench:
ab -n 100 -c 8 https://tranquil-brushlands-56319.herokuapp.com/
As the traffic increases, you'll see the connections max out.