ruby / test framework

updated September 02, 2026

I used a custom test framework of about 250 lines of Ruby. It provided stubs, database transactions, and factories. The implementation is at the end of this article.

Test groups and test cases

Test groups inherited from a Test base class:

class MathTest < Test
  def test_greater_than
    ok { 10 > 5 }
  end

  def test_less_than
    ok { 3 < 7 }
  end
end

A test case was a public instance method named test_*. A file could hold one or more test groups.

Assertions

The ok method with a block was the only assertion:

# pass
ok { true }
ok { 2 + 2 == 4 }
ok { "hello" =~ /ello/ }

# fail
ok { false }
ok { nil }

ok required single-line brace syntax.

class NilTest < Test
  def test_nil
    val = nil
    ok { val == nil }
  end

  def test_not_nil
    val = "value"
    ok { val != nil }
  end
end

class RegexTest < Test
  def test_match
    got = "[email protected]"
    want = /\A[\w+\-.]+@[a-z\d\-.]+\.[a-z]+\z/i
    ok { got =~ want }
  end

  def test_no_match
    got = "text"
    ok { got !~ /[<>]/ }
  end
end

class ExceptionTest < Test
  def test_raised
    raised = false

    begin
      raise ArgumentError, "invalid argument"
    rescue ArgumentError
      raised = true
    end

    ok { raised }
  end

  def test_not_raised
    raised = false

    begin
      10 / 2
    rescue
      raised = true
    end

    ok { !raised }
  end
end

A failing ok printed a backtrace and exited.

Runner

I ran a test file directly:

ruby test/lib/db_test.rb

My framework randomized test order and printed a seed:

seed 1234
..
ok

Dots keep test output short. ENV["CI"] disables color codes.

Example output from failing tests:

seed 5678

MathTest#test_add

  fail:

    got == 6
    |   |
    |   false
    5

  path/to/file.rb:42:in `MathTest#test_add`

On failure, ok re-ran the block with tracing to build the message. The block can run twice, so I avoided side effects in ok blocks.

I re-ran with the same order using the seed:

ruby test/lib/db_test.rb --seed 1234

I ran a single test case from the command line:

ruby test/lib/db_test.rb --name test_fuzzy_like_pattern

Or from Vim with vim-test.

A suite file such as test/suite.rb required all test files:

require_relative "test_helper"

Dir["#{__dir__}/**/*_test.rb"].each { |f| require f }

Then I ran it:

ruby test/suite.rb

Before suite

One-time setup code went in test_helper.rb. It ran when Ruby loaded the file:

# before suite
DB.pool.exec(<<~SQL)
  INSERT INTO users (id, name, admin, email)
  VALUES (1, 'Admin', true, '[email protected]')
  ON CONFLICT DO NOTHING;

  ALTER SEQUENCE users_id_seq RESTART WITH 2;

  REFRESH MATERIALIZED VIEW cache_companies;
SQL

Database transactions

Each test case ran in a transaction that rolled back. The tests used a dedicated test Postgres cluster:

class TransactionTest < Test
  def test_insert
    co = insert_company(name: "Acme Inc")

    rows = db.exec("SELECT * FROM companies")
    ok { rows.size == 1 }
  end

  def test_another_insert
    # Database is clean (previous test rolled back)
    rows = db.exec("SELECT * FROM companies")
    ok { rows == [] }
  end
end

To test transaction behavior itself, I set @tx = false:

class TransactionBehaviorTest < Test
  def initialize
    super
    @tx = false
  end

  def test_rollback
    # Test actual transaction behavior
    # Changes cleaned up with DELETE after test
  end
end

Fresh database fixtures

Factory methods inserted fresh fixtures with DB:

class CompaniesTest < Test
  def test_create
    co = insert_company(name: "Acme Inc", status: "Active")

    ok { co.name == "Acme Inc" }
    ok { co.status == "Active" }
  end

  def test_with_relationships
    co = insert_company
    per = insert_person(name: "Jane Doe")
    pos = insert_position(
      person_id: per.id,
      company_id: co.id,
      company_name: co.name,
      title: "CTO"
    )

    ok { pos.person_id == per.id }
  end
end

Factories provided defaults and returned Data objects.

State-based

I preferred to assert on state in the database:

def test_create_company
  Companies::Create.new(db).call(name: "Acme Inc")

  row = db.exec("SELECT * FROM companies").first
  ok { row["name"] == "Acme Inc" }
end

Object stubs

Otherwise, I injected a stub for the collaborator:

module Companies
  class Import
    def initialize(db, client:)
      @db = db
      @client = client
    end

    def call(domain:)
      data, err = @client.fetch(domain)
      if err
        return "err: #{err}"
      end

      @db.exec(<<~SQL, [data["name"]])
        INSERT INTO companies (name)
        VALUES ($1)
      SQL

      "ok"
    end
  end
end

class CompaniesImportTest < Test
  def test_import
    client = stub(fetch: [{"name" => "Acme Inc"}, nil])

    got = Companies::Import.new(db, client: client).call(
      domain: "acme.com"
    )

    ok { got == "ok" }
    ok { client.called?(:fetch) }

    row = db.exec("SELECT * FROM companies").first
    ok { row["name"] == "Acme Inc" }
  end

  def test_api_error
    client = stub(fetch: [nil, "API rate limited"])

    got = Companies::Import.new(db, client: client).call(
      domain: "acme.com"
    )

    ok { got == "err: API rate limited" }
    ok { db.exec("SELECT * FROM companies") == [] }
  end
end

Stubs supported lambdas for transformations:

client = stub(
  transform: ->(text) { text.upcase },
  calculate: ->(a, b) { a + b }
)
ok { client.transform("hello") == "HELLO" }
ok { client.calculate(2, 3) == 5 }

Class method stubs

For class methods, I used stub_class:

class TimeTest < Test
  def test_frozen_time
    stub_class(Time, now: Time.at(0))

    ok { Time.now == Time.at(0) }
  end
end

My framework restored class method stubs after each test.

Class method stubs also supported lambdas:

# identity functions
stub_class(Convert::ExtractDomain, call: ->(host) { host })

# raise errors
stub_class(Aws::S3::Client, new: ->(*) {
  raise StandardError.new("auth failed")
})

# capture variables
err_msg = nil
stub_class(Sentry, capture_exception: ->(e) { err_msg = e.message })
some_code_that_raises
ok { err_msg == "expected error" }

Asserting stub calls

Every stub recorded its calls. I checked them with called?:

client = stub(fetch: [{"name" => "Acme Inc"}, nil])

Companies::Import.new(db, client: client).call(domain: "acme.com")

ok { client.called?(:fetch) }
ok { !client.called?(:delete) }

Or I asserted call count and arguments with calls:

ok { client.calls[:fetch].size == 2 }
ok { client.calls[:fetch][0][:args] == ["acme.com"] }
ok { client.calls[:fetch][0][:kwargs] == {domain: "acme.com"} }

Yielding stubs

For a method that yields, I used Object.new with def:

client = Object.new
def client.get_data(_)
  yield "chunk1"
  yield "chunk2"
end

got = []
client.get_data("http://example.com") { |chunk| got << chunk }
ok { got == ["chunk1", "chunk2"] }

For Object.new stubs, I captured values with instance variables:

client = Object.new
def client.process(_)
  @thread_ref = Thread.current
  yield "data"
end
def client.thread_ref
  @thread_ref
end

some_code_under_test(client)

ok { !client.thread_ref.alive? }

HTTP stubs

I stubbed HTTP requests to third-party services with WebMock:

class CompaniesImportTest < Test
  def test_import
    stub_request(:post, "https://api.example.com/companies")
      .with(body: {domain: "acme.com"})
      .to_return(status: 200, body: {name: "Acme Inc"}.to_json)

    got = Companies::Import.new(db).call(domain: "acme.com")

    ok { got == "ok" }
  end
end

WebMock stubs live in one global registry for the whole process. teardown called WebMock.reset!:

private def teardown
  # ... other cleanup ...
  WebMock.reset!
end

Style guide

I inlined code where I could. When a local variable helped, I used these names:

got: return value
want: expected value
row: single hash from query (.first)
rows: array of hashes from query

def test_length
  got = "hello".length
  want = 5
  ok { got == want }
end

Blank lines separated setup, exercise, and assertion:

def test_add
  a = 2
  b = 3

  got = a + b

  ok { got == 5 }
end

With several exercises, I grouped each exercise with its assertion:

def test_multiply
  got = 2 * 3
  ok { got == 6 }

  got = 4 * 5
  ok { got == 20 }

  got = 0 * 10
  ok { got == 0 }
end

I named fresh fixtures with abbreviations (co, per, u). A fixture re-read from the database got a db_ prefix:

co = insert_company(name: "foo")

Something.new(db).call("bar")

db_co = db.exec("SELECT * FROM companies WHERE id = $1", [co.id]).first
ok { db_co["name"] == "bar" }

A simple query went on one line:

note = db.exec("SELECT * FROM notes WHERE company_id = $1", [co.id]).first

A complex query used a heredoc:

job = db.exec(<<~SQL, [per.id]).first
  SELECT
    jobs.*
  FROM
    notes
    JOIN jobs ON jobs.args ->> 'note_id' = notes.id::text
  WHERE
    notes.person_id = $1
    AND jobs.queue = 'slack'
SQL

I asserted an empty result with rows == []:

rows = db.exec("SELECT * FROM tracking WHERE company_id = $1", [co.id])
ok { rows == [] }

For unordered comparisons, I mapped and sorted:

rows = db.exec("SELECT * FROM list_items WHERE company_id = $1", [co.id])
ok { rows.map { |r| r["list_id"] }.sort == [748, 541].sort }

For error messages, I used include?:

ok { got[:errs].include?("Company is required") }

Rack handler testing

For web framework handler tests, I extended Test with Rack::Test. See response modes for how a handler declared its modes.

require_relative "test_helper"
require "rack"
require "rack/test"

class HandlerTest < Test
  include Rack::Test::Methods

  def app
    Framework::Server.app(db)
  end

  def db
    @db ||= DB.new
  end

  def sign_in
    sign_in_as(insert_user(active: true))
  end

  def sign_in_as(user)
    jar = EncryptedCookieJar.new({}, ENV.fetch("SECRET_KEY_BASE"))
    jar.set(user.remember_token, expires: Time.now + 3600)
    headers = {}
    jar.write_to_headers(headers)
    full_cookie = headers["Set-Cookie"].first.match(/^([^;]+)/)[1]
    name, value = full_cookie.split("=", 2)
    encoded_value = URI.encode_www_form_component(value)
    set_cookie("#{name}=#{encoded_value}")
  end

  def flash
    Flash.new(last_request)
  end

  class Flash
    def initialize(request)
      @request = request
    end

    def [](key)
      rack_session = @request.env["rack.session"]
      if rack_session.nil?
        return nil
      end

      flash_hash = rack_session[:flash]
      if flash_hash.nil?
        return nil
      end

      flash_hash[key]
    end
  end
end

I used HandlerTest for handler tests:

class CompaniesHandlerShowTest < HandlerTest
  def test_index
    sign_in
    co = insert_company(name: "Acme Inc")

    resp = get("/companies/#{co.id}")

    ok { resp.status == 200 }
    ok { resp.body.include?("Acme Inc") }
  end
end

Test#run_group reused one instance per class, so the Rack::Test cookie jar carried session state from one test method into the next. teardown cleared it:

private def teardown
  super
  clear_cookies
end

Ping test coverage

A ping test ran the full handler path and asserted status 200. One test class mapped to one handler class:

CompaniesHandler::Show  → CompaniesHandlerShowTest
Admin::UsersHandler::Index → AdminUsersHandlerIndexTest

Each test class defined ping methods based on declared modes:

modes :html → test_ping_html
modes :ajax → test_ping_ajax
modes :html, :ajax → both

class CompaniesHandlerShowTest < HandlerTest
  def test_ping_html
    sign_in
    co = insert_company

    resp = get("/companies/#{co.id}")

    ok { resp.status == 200 }
    ok { resp.body.include?(co.name) }
  end

  def test_ping_ajax
    sign_in
    co = insert_company

    resp = get("/companies/#{co.id}", {}, {"HTTP_AH_REFERER" => "/"})

    ok { resp.status == 200 }
    ok { !resp.body.include?("<html") }
  end
end

class PingCoverageTest < HandlerTest
  def test_report_missing_ping_coverage
    routes_app = Framework::Server.routes(db)
    missing = []

    routes_app.instance_variable_get(:@routes).each do |method, entries|
      entries.each do |_, path, handler|
        if handler.is_a?(Array) # skip redirects
          next
        end

        klass = handler.class
        modes = klass.modes
        test_class = find_test_class(klass)

        if modes.include?(:html)
          if test_class.nil? || !test_class.method_defined?(:test_ping_html)
            missing << "#{method} #{path} (#{klass}) missing test_ping_html"
          end
        end

        if modes.include?(:ajax)
          if test_class.nil? || !test_class.method_defined?(:test_ping_ajax)
            missing << "#{method} #{path} (#{klass}) missing test_ping_ajax"
          end
        end
      end
    end

    ok { missing == [] }
  end

  private def find_test_class(klass)
    parts = klass.name.split("::")
    parts.shift # remove top-level module
    test_class_name = parts.join + "Test"

    begin
      Object.const_get(test_class_name)
    rescue NameError
      nil
    end
  end
end

A new handler or mode without a ping test failed the build.

Suite files

Handler tests had their own suite file:

# test/ruby_suite.rb
require_relative "test_helper"

Dir.glob(File.join(__dir__, "**", "*_test.rb"))
  .reject { |f| f.include?("/handle/") }
  .sort
  .each { |f| require f }

# test/rack_suite.rb
require_relative "rack_helper"

Dir.glob(File.join(__dir__, "lib", "handle", "**", "*_test.rb"))
  .sort
  .each { |f| require f }

I ran them separately:

ruby test/ruby_suite.rb  # fast, no Rack
ruby test/rack_suite.rb  # slower, loads Rack app

An at_exit hook in test/test_helper.rb ran the suite.

Implementation

The test/test_helper.rb file:

ENV["APP_ENV"] = "test"

require "prism"
require "webmock"
require_relative "../lib/db"
require_relative "factories"

WebMock.enable!
WebMock.disable_net_connect!(allow_localhost: true)

DB.configure do |c|
  c.pool_size = 1
  c.reap = false
end

class Test
  class Failure < StandardError; end

  include Factories
  include WebMock::API

  if ENV["CI"]
    GREEN = ""
    RED = ""
    RESET = ""
  else
    GREEN = "\e[32m"
    RED = "\e[31m"
    RESET = "\e[0m"
  end

  @@groups = []
  @@seed = nil
  @@name = nil

  i = 0
  while i < ARGV.length
    case ARGV[i]
    when "--seed"
      @@seed = ARGV[i + 1].to_i if i + 1 < ARGV.length
      i += 2
    when "--name"
      @@name = ARGV[i + 1] if i + 1 < ARGV.length
      i += 2
    else
      i += 1
    end
  end

  def self.inherited(c)
    @@groups << c
  end

  def self.run_suite
    seed = @@seed || rand(1000..9999)
    srand seed
    puts "seed #{seed}"

    @@groups.shuffle.each do |c|
      c.run_group
    end

    print "\n#{GREEN}ok#{RESET}\n"
  end

  def self.run_group
    group = new

    tests = public_instance_methods(false)
      .grep(/^test_/)
      .shuffle

    if @@name
      tests = tests.select { |t| t.to_s == @@name }
      if tests == []
        return
      end
    end

    if tests == []
      return
    end

    tests.each { |test| group.run_test(test) }
  end

  def db
    DB.pool
  end

  def initialize
    @tx = true
    @stubs = []
  end

  def run_test(test)
    setup
    send(test)
    print "#{GREEN}.#{RESET}"
  rescue => err
    puts "\n#{RED}#{self.class}##{test}#{RESET}"
    lines = err.backtrace.reject { |l| l.include?(__FILE__) }.join("\n  ")
    puts "\n  #{RED}fail:\n#{err}#{RESET}\n\n  #{lines}"
    exit 1
  ensure
    teardown
  end

  def ok(&block)
    if block.call
      return
    end

    rets = []
    tgt = Thread.current
    trace = TracePoint.new(:return, :c_return) do |tp|
      if Thread.current == tgt
        rets << [tp.callee_id.to_s, tp.return_value]
      end
    end
    trace.enable { block.call }

    file, lnum = block.source_location
    line = File.readlines(file)[lnum - 1]
    if line !~ /ok\s*\{\s*(.+)\s*\}/
      raise ArgumentError, "ok requires single-line brace syntax: ok { expr }"
    end
    src = $1.strip

    # method name => [column positions]
    mcols = Hash.new { |h, k| h[k] = [] }
    begin
      visit_ast_node(Prism.parse(src).value, mcols)
    rescue
      raise Test::Failure, "\n    #{src}\n    (unable to parse source)"
    end
    lvars = block.binding.local_variables.map(&:to_s)

    # match return values to column positions
    vals = []
    seen = Hash.new(0)
    rets.each do |name, value|
      cols = mcols[name]
      if cols.any? && seen[name] < cols.length
        vals << [cols[seen[name]], name, value]
        seen[name] += 1
      end
    end

    # Prism treats bare identifiers as method calls; add local variables
    mcols.each do |name, cols|
      if lvars.include?(name) && vals.none? { |_, n, _| n == name }
        cols.each do |col|
          vals << [col, name, block.binding.local_variable_get(name.to_sym)]
        end
      end
    end

    if vals.empty?
      raise Test::Failure, "\n    #{src}\n    (no values to show; try assigning to local variables)"
    end

    # rightmost first for building output lines
    vals.sort_by! { |col, _, _| -col }

    max = vals.map(&:first).max
    pline = " " * (max + 1)
    vals.each { |col, _, _| pline[col] = "|" }

    vlines = []
    vals.each_with_index do |(col, _, value), i|
      ins = begin
        value.inspect
      rescue
        "#<inspect failed>"
      end
      out = " " * (max + 1)
      vals.each_with_index do |(c, _, _), j|
        if j > i
          out[c] = "|"
        end
      end
      if col + ins.length > out.length
        out = out[0...col] + ins
      else
        out[col, ins.length] = ins
      end
      vlines << out.rstrip
    end

    msg = [src, pline.rstrip, *vlines].join("\n")
    raise Test::Failure, "\n    " + msg.gsub("\n", "\n    ")
  end

  def stub(methods)
    obj = Object.new
    calls = Hash.new { |h, k| h[k] = [] }

    methods.each do |meth, return_value|
      obj.define_singleton_method(meth) do |*args, **kwargs, &block|
        calls[meth] << {args: args, kwargs: kwargs}
        if return_value.is_a?(Proc)
          return_value.call(*args, **kwargs, &block)
        else
          return_value
        end
      end
    end

    obj.define_singleton_method(:called?) do |meth|
      calls[meth] != []
    end

    obj.define_singleton_method(:calls) do
      calls
    end

    obj
  end

  def stub_class(klass, methods)
    methods.each do |meth, return_value|
      orig = klass.method(meth)
      @stubs << [klass, meth, orig]

      klass.define_singleton_method(meth) do |*args, **kwargs, &block|
        if return_value.is_a?(Proc)
          return_value.call(*args, **kwargs, &block)
        else
          return_value
        end
      end
    end
  end

  private def setup
    if @tx
      db.exec("BEGIN")
    end
  end

  private def teardown
    @stubs.reverse.each do |klass, meth, orig|
      klass.define_singleton_method(meth, orig)
    end
    @stubs = []

    WebMock.reset!

    if @tx
      db.exec("ROLLBACK")
    else
      tablenames = db.exec(<<~SQL).map { |row| row["tablename"] }
        SELECT
          tablename
        FROM
          pg_tables
        WHERE
          schemaname = 'public'
          AND tablename != 'users'
        ORDER BY
          tablename
      SQL

      tablenames.each do |t|
        db.exec("DELETE FROM #{t}")
      end

      # app-specific cleanup of all users except admin fixture
      db.exec("DELETE FROM users WHERE id != 1")
    end
  end

  private def visit_ast_node(node, cols)
    if node.nil?
      return
    end

    if node.is_a?(Prism::CallNode)
      name = node.name.to_s
      c = node.message_loc&.start_column || node.location.start_column
      cols[name] << c
    end

    node.compact_child_nodes.each { |child| visit_ast_node(child, cols) }
  end
end

at_exit { Test.run_suite }

The test/factories.rb file:

module Factories
  class Sequence
    def initialize
      @counter = 0
    end

    def next
      @counter += 1
    end
  end

  SEQ = Sequence.new

  def insert_company(o = {})
    insert_into("companies", {
      name: o[:name] || "Company #{SEQ.next}"
    }.merge(o))
  end

  def insert_person(o = {})
    insert_into("people", {
      name: o[:name] || "Person #{SEQ.next}"
    }.merge(o))
  end

  def insert_position(o = {})
    insert_into("positions", {
      company_id: o[:company_id] || insert_company.id,
      person_id: o[:person_id] || insert_person.id,
      title: o[:title] || "CEO, Founder"
    }.merge(o))
  end

  private def insert_into(table, attrs)
    row = db.exec(<<~SQL, attrs.values).first
      INSERT INTO #{table} (
        #{attrs.keys.join(", ")}
      ) VALUES (
        #{(1..attrs.size).map { |i| "$#{i}" }.join(", ")}
      )
      RETURNING *
    SQL

    Data.define(*row.keys.map(&:to_sym)).new(*row.values)
  end
end

← All articles