Onboarding experiments

What experimentation on onboarding is

Onboarding runs from the first session to the first moment the product does something the user came for. Experiments here change the order, the length and the demands of that path: which steps are required, what is asked before the first action, what the empty state offers, and when the product asks for other people, data or configuration.

The metrics

The primary metric is almost always activation: the share of new accounts that reach the event the product treats as first value, in a window of days. Paid activation is the version to prefer when the product converts to paid soon after activating, because it is closer to revenue and further from a local click. Secondary metrics sit closer to the change: first project created, setup completed, time to first value. They explain a result. They should not decide it.

The bottlenecks

The recurring ones: a setup step required before any value is shown, an empty state that offers nothing to do, an invitation asked for before the user has anything to invite people to, a data import demanded up front, and permissions or verification that arrive at the wrong moment. Each is a demand placed before the product has proved itself.

The guardrails

Support volume, because a shortened path can leave people lost. Downstream completion of the step that was removed or delayed, because optional setup that never gets done is a leak moved rather than fixed. Cancellation in the first cycle, because an account that activated on a thinner path may hold less.

When a test here is feasible

Onboarding tests count signups, not visits, so the population is smaller than a signup screen's and larger than a pricing page's, and activation rates are moderate, which helps. A product with a few thousand signups a month can usually detect a relative change in the tens of percent within weeks. One with a few hundred cannot, and should decide these questions by judgment and ship them rather than test them. Size the test before it starts, with the calculator and the sample size guide, and read the feasibility guide before deciding to run it at all.

The patterns

Delaying a required setup step until after the first value is the pattern this stage keeps returning to, and it is the example written out in full below. Starting from a template rather than a blank project, inviting teammates after the first project rather than before it, and a nudge at the second session are the same idea at other points on the path, and they are written when they are.