What is a guardrail metric?

The definition

A guardrail metric is a measure an experiment is not allowed to make worse. It sits beside the primary metric, the one the experiment is trying to move, and it is read the same way, at the planned sample, with a threshold decided before the test starts. The difference is the question it answers. The primary metric asks whether the change worked. A guardrail asks what the change cost, somewhere the team was not looking.

Kohavi, Tang and Xu, whose book is the standard reference on online experiments, put it as identifying what the organisation is not willing to change [1]. Microsoft's experimentation platform describes guardrails as measures of aspects of the product that must not degrade but will not necessarily improve, and names page load time, crash rate and abandonment rate as its examples [2]. GrowthBook's documentation says the same thing in a product's words: guardrail metrics are the ones you are trying not to hurt, and if they move significantly, consider ending the experiment [4].

Three words in that definition carry the weight. Not to hurt: a guardrail has a direction, must not drop or must not rise, and no target. Significantly: it is read statistically, not eyeballed, so a guardrail that moves by noise does not end a good test. Consider: a breach is a decision, and the decision rule should have said in advance what a breach does.

Goals, drivers and guardrails

Metrics on an experiment come in three kinds, and the taxonomy is worth keeping because teams confuse them. A goal metric is the business outcome the experiment is for: paid activation, upgrade rate, retained revenue. A driver metric is closer to the change and moves sooner: first project created, signup completion, a click on the plan. A guardrail is neither; it is a boundary. Kohavi, Tang and Xu give the taxonomy a chapter of its own, with guardrail metrics as a sidebar to it [1].

The confusion that costs the most is a driver mistaken for a goal. A signup test that raises completion while activation falls has moved its driver and hurt its goal, and if activation was not a guardrail the team ships a loss and calls it a win. The signup experiment in this library holds activation within fourteen days as a guardrail for exactly that reason, and its hub says why signup tests are the ones most likely to need it.

Examples, by stage

The right guardrails depend on where the experiment sits, because each stage has a way for a win to hide a loss.

• Signup. Activation within a couple of weeks, because an easier signup can admit accounts that never intended to use the product. Support conversations about signing in. Abuse and spam signups.
• Onboarding. Support volume, because a shortened path can leave people lost. Downstream completion of the step that was removed or delayed, because optional setup that never gets done is a leak moved rather than fixed. Cancellation in the first cycle. The onboarding experiment carries all three.
• Pricing. Refund requests in the first month and cancellation before the first renewal, because a presentation that reads as a trick is paid for later. Support conversations about billing. And the upgrade rate itself, held as a guardrail on any test whose real question is the annual and monthly mix, so a framing that only shifts the mix cannot pass as a win. The pricing experiment is written that way.
• Any stage. Page load time and error rate, whenever the change touches code, because a variant that is slower or that crashes is a bad experience whatever its metric says. Kohavi, Tang and Xu single out crashes: if a feature increases crashes, the experience is so bad that other factors pale beside it [1].

The trust guardrail

One guardrail is different in kind: it does not say whether the change hurt something, it says whether the test can be believed. The most important is the sample ratio mismatch. If an experiment was set to split traffic fifty to fifty and the observed split is off by more than chance allows, something upstream of the metrics is broken: a redirect that drops slow clients from one arm, a bot filter that treats the arms differently, a logging bug. Microsoft's platform treats a sample ratio mismatch as invalidating the test, with any metric movement untrustworthy until the cause is found [2]. A study of the causes at Microsoft warns that ignoring one can make a bad change look good and ship, or the reverse [3].

The practical rule: check the ratio before reading a single metric, and treat a mismatch as a stop, not a footnote. Kohavi, Tang and Xu give sample ratio mismatch and the other trust guardrails a chapter of their own [1], which is a fair measure of how often teams skip them.

How to choose them

Three guardrails is a good number for most experiments, and five is the most a team will actually read. Choose them by asking, for the change in front of you, three questions.

What could this win be hiding? A driver that rises while a goal falls, a mix that shifts without the total moving, a cohort admitted that will not stay.

What downstream cost would this change create? Support, refunds, cancellations, the step that is now optional and might not get done.

What could break? Speed, errors, the logging the test itself depends on.

Then write each one down with its direction and its definition before the experiment starts, and write into the decision rule what a breach does. On the experiments in this library, a breach reverts the change at any point in the run, and the primary metric can only be read at the planned sample with every guardrail holding. That order, guardrails first, is what makes a result worth acting on.

What Athren watches

Every experiment Athren proposes carries its guardrails with their directions and thresholds, and it watches them while the experiment runs. A guardrail that crosses its threshold turns the flag off and marks the change reverted, versioned so the revert can be read back later. A person approves every experiment before it reaches traffic and every winner before it ships, so the guardrails decide when to stop, and people decide what to keep.