Land returning users on their work in progress

A returning user lands on the same project list as a first-time user and has to remember what they were doing before the product is useful again. The cost is small and it is paid on every return, which is the shape of a habit that does not form. It is also invisible in the journey the front page draws, because that example ends at the first payment and this behaviour starts after it.

A returning session is where a product either becomes a habit or stops being opened, and the surface it opens on decides how much work a user does before the product is worth anything to them. The reason to test this rather than ship it on judgment is that a new screen on a returning surface is where an effect is most likely to appear early and fade: treatment effects in online experiments are not always stable over time and can show increasing or decreasing patterns as users learn a change [1]. That is a measurement problem before it is a design one, and it is why the read here is at day 28 and not at day 3. Nothing on this page reports a result.

Workspaces with at least one project and at least one session in the 14 days before assignment. Workspaces created during the test are excluded, because their first weeks are onboarding rather than return, and workspaces whose only sessions are from an integration or a bot are excluded, because nothing about them is a returning person.

Opening the workspace home on work already in progress raises week four retention, because the cost of a return falls to a single click and a product that is useful immediately is opened again. A project list asks the user to remember what they were doing before it gives them anything.

Control

The workspace home opens on the project list, ordered by name, the same view in a first session and a fiftieth.

Variant

The workspace home opens on the projects the user opened or edited most recently, the last one first, with the full project list one click away in the place it has always been. A user with no recent activity of their own sees the list, as before.

Name

Week four retention

Definition

The share of assigned workspaces with at least one qualifying session in the fourth week after assignment, where a qualifying session contains an edit and not only a page view

Window

28 days from assignment, read once at day 28 and never before

Secondary: Return within seven days

The share of assigned workspaces with a qualifying session in the seven days after assignment, 7 days from assignment.

Secondary: Active days per workspace

Distinct days with a qualifying session, per assigned workspace, 28 days from assignment.

Secondary: Time to the first action in a returning session

Median seconds from the workspace home rendering to the first edit in the same session, 28 days from assignment.

New projects created

Must not drop. New projects per hundred assigned workspaces in the window, against its baseline rate

Support conversations

Must not rise. Support conversations per hundred assigned workspaces in the window

Cancellation within the window

Must not rise. The share of assigned paid workspaces that cancel within the 28 days

Assignment balance

Must not rise. The gap between the observed and the planned split of assigned workspaces, checked daily; a sample ratio mismatch on a test this long invalidates the read

This is an example. No expected impact is stated because there is no evidence to state one from.

Allocation

50 to 50 on eligible workspaces, because a retention read needs every unit it can get

Eligibility

An eligible workspace is assigned once, at the first workspace home render while the flag is on, and keeps its arm for the whole 28 day window across every member and every device

Ramp

Five days at 10 percent to catch a broken home screen, then the full 50 to 50. The ramp cohort is read on its own and does not join the planned sample, because it was assigned under a different allocation.

Minimum sample

The calendar is the constraint here, not the traffic. The last workspace assigned still needs its own 28 days, so the test runs for the enrolment period plus four weeks, and a product that needs six weeks of enrolment to reach the sample is running a ten week test. Size it before it starts and decide whether ten weeks is a price worth paying; a retention test read at day 3 measures a new screen's novelty and nothing about retention.

Size it with

A/B test sample size calculator

Size it with

Experiment feasibility checker

The workspace home's default view moves behind the customer's existing feature flag. The project list, its ordering, its components and its data are the customer's own, and both arms reach it from the same place. Exposure is recorded at the first workspace home render after assignment, so the arms are compared on the same population, and the arm is stored on the workspace rather than in a cookie, so a 28 day window survives a cleared browser.

Read the primary metric once, at day 28 after the last assignment, and ship when its interval clears zero with every guardrail holding. Guardrails are read daily throughout, and a breach reverts the test at any point; that is the only read before day 28. If the interval does not clear zero at the planned sample the result is inconclusive, and the change is then decided on judgment rather than run on until it clears.