North Star Metric

It is difficult to prioritize product and growth work.

Should we write a new feature? Remove a feature? Fix a bug? Design a user interface? Remove a step in the activation funnel? Write a blog post? Change pricing? Hire a customer support rep?

The "North Star" metaphor means following one thing as our guide. A North Star metric for startups answers the question:

How many people are getting authentic value from our product?

"Authentic value" means avoiding vanity metrics such as pageviews and sessions.

Daily Active Users (DAU) or Monthly Active Users (MAU) may not apply if the moment when a user gets authentic value is not necessarily when the user acts.

Example metric

An organization exists to serve a mission. A product exists to serve a job to be done.

Hound's mission is to enable better software. A job to be done to enable better software is to keep a project's code in a consistent style. Hound serves that job to be done by automatically reviewing GitHub pull requests for style violations.

The moment Hound reviews is called a "build." It is the moment the team gets authentic value and the product's job is done.

Hound's North Star metric is therefore:

Teams with builds per week

Calculate

We calculate the North Star metric with a SQL query on our read-only "follower" database using Heroku Dataclips.

SELECT
  week,
  teams,
  (
    teams::float /
    lag(teams) OVER (ORDER BY week) - 1
  ) growth
FROM (
  SELECT
    date_trunc('week', builds.created_at) AS week,
    count(DISTINCT repos.id) AS teams
  FROM
    repos
    INNER JOIN builds ON builds.repo_id = repos.id
  WHERE repos.full_github_name NOT LIKE 'houndci%'
    AND builds.created_at >= current_date - interval '14 weeks'
  GROUP BY week
) AS _;

The subquery aggregates builds data into weeks using the date_trunc function and GROUP BY statement. Within those weeks, we count the DISTINCT GitHub repos and alias them as teams. The WHERE clause filters the results in any appropriate ways for the particular product. We usually remove employees of the product's company and restrict the time window to the previous quarter, including this week in progress.

The outer query lags each week's teams OVER its previous week to calculate growth. lag() is a window function that compares rows of teams by an offset of 1.

The result set looks like this:

Growth rate

It is necessary to calculate both absolute numbers and growth rate in order to completely understand the product's current health. We want to know Hound currently has "266 teams with builds per week", and is growing at "9% per week" (the average of the trailing quarter, excluding the current week).

Y Combinator startups are expected to grow by 7% per week:

When I first meet founders and ask what their growth rate is, sometimes they tell me "we get about a hundred new customers a month." That's not a rate. What matters is not the absolute number of new customers, but the ratio of new customers to existing ones. If you're really getting a constant number of new customers every month, you're in trouble, because that means your growth rate is decreasing.

He also identifies the most-common follow-up question after learning the current North Star value and growth rate:

How many new versus retained users?

New versus retained users

A second Dataclip calculates retained users:

SELECT
  date_trunc('week', builds.created_at) AS week,
  count(DISTINCT repos.id) AS retained_teams
FROM
  repos
  INNER JOIN builds ON builds.repo_id = repos.id
WHERE repos.full_github_name NOT LIKE 'thoughtbot%'
  AND builds.created_at >= current_date - interval '14 weeks'
  AND repos.created_at < date_trunc('week', builds.created_at)
GROUP BY week;

It is almost exactly the same as the North Star's subquery except for the WHERE clause's AND repos.created_at < date_trunc('week', builds.created_at), which filters the data to include only builds from repos that were created prior to the current week.

Visualizing the data

We can use Heroku Dataclips' CSV export and Google Sheets' IMPORTDATA function to get a live import in Google Sheets:

=IMPORTDATA("https://dataclips.heroku.com/abc123.csv")

Which lets us chart the data and make other calculations, such as averaging the growth rate:

The new versus retained teams bottom chart is generated by importing the second Dataclip into a second Google Sheet tab. On that tab, we calculate new teams by subtracting the retained teams from the total teams in the first Dataclip.

The "Goals" section is calculated by multiplying last week's total teams, retained teams, and new teams by 1.07 (a 7% growth rate).

The "Current" section is calculated by dividing this week's total teams, retained teams, and new teams by their corresponding goals.

Data leads to the next question

The bottom chart and our current progress toward our goals helps us make decisions about whether to focus on customer acquisition (marketing) or customer retention (product).

The data is telling us that retention is good but acquisition is not.

The guidance from our North Star metric causes us to ask questions such as "which customer acquisition channels are working the best?", or "do we have an awareness problem or an activation problem?"

Prioritize work by answering questions

The questions lead us to look deeper into our data in places such as Mixpanel funnels, which helps reveal where in the product people are getting blocked.

Here's Hound's activation funnel in Mixpanel:

While both steps could be improved, it seems more likely that improving the first step would result in a bigger improvement of our North Star:

Teams with builds per week

Numbers identify the "what." Our team identifies the "why."

To answer "why?", we can critique the current home page for ways to improve the user interface design. For example, we might want to change the call to action from "Sign in with GitHub" to "Add Hound to your GitHub repo" or some other copy text.

Another way to answer "why?" is to look for qualitative data to compare to the quantitative data. We often use Intercom for user research.

In our conversations with users, we may discover that they are unhappy with the permissions we require in the GitHub OAuth step of the sign up process. That may lead us to change the OAuth permissions, removing functionality which relies on the more aggressive permissions.

Design an experiment, instrument, implement, observe

We now have ideas for changes to make. In this example, the changes are to the product, but our process could have as easily led us to make changes to our marketing.

The product changes are only one part of the design of the growth experiment. We also need to define how we will know if the changes are successful.

Changing the CTA text should improve the conversion rate of from "Viewed Home Page" to a new "Clicked to Authenticate" event in the funnel.

Requiring fewer GitHub permissions should improve the conversion rate of from "Clicked to Authenticate" to "Signed Up." That would represent the percent of users who have successfully authenticated with GitHub.

As sometimes happens in these iterations, we need to instrument the app with a more granular event. Then, we can compare the changes by A/B testing. Or, we could deploy the instrumentation and gather baseline data for a period of time before deploying changes.

After we feel good about our conversion rates, we might turn our attention away from product work, toward marketing work such as prioritizing customer acquisition channels.

Learn more

Once we begin looking at the North Star metric on a daily and weekly basis, we've given ourselves an organizing mechanism for deciding where to focus our time.

For more about North Star metrics, watch talks from Alex Schultz and Josh Elman.