Experiments

An experiment in Quoin moves through a fixed lifecycle: draft → running → stopped/concluded — four states, not five. "Planned" is a UI-only label over a draft row after Quoin reads a plan for it. It is never a stored status. When the experiment is running, its rules are frozen. Nothing can change the weights, the variations or the coverage. The only actions left are a kill switch or a force override.

Reading a report

Read a report from top to bottom. The order is important:

1. The health checks come first. If the sample ratio is off (a p-value below 0.001) or the smaller arm has fewer than 100 units, the report says so and shows no lift. See "Health checks, before any number," below.
2. Then the verdict sentence. One sentence names the verdict word and the number behind it.
3. Then the sequential 95% interval. It is the headline number. Quoin draws it one time on a ruler against zero. It stays valid every time that you look. So it is safe to check a running experiment every day.

A report is one snapshot for each read. Reload to see a newer one. "The six verdicts" lists the six words that a verdict can take. "A worked example: trial-reminders" shows one report, from the start to its decision.

Pre-registration

Before an experiment can start, it locks a pre-registration. The pre-registration has these items:

• The planned number of weeks.
• The planned sample size for each arm and in total.
• The baseline rate.
• The metrics that the experiment uses. Quoin fingerprints them, so a later edit to the definition of a metric cannot silently change what the running experiment measures.
• A power warning, if the plan looks underpowered.

After the lock, Quoin shows the pre-registration as a locked block. The block has a surface-2 background, a lock icon, and the text "locked 28 Jul, 10:04 · sam". You cannot edit it unless you first stop the experiment.

The planner prices both arms at the own variance of control. It has no data on the treatment yet, so this is the only honest guess. For this reason, Quoin describes every plan number as "about", never as an exact promise. When the experiment runs, the actual detectable lift can differ from the estimate of the plan. This is more likely when the treatment is much more variable, or much less variable, than control.

Planning, locking and starting (as of QN-06)

/experiments/new is a four-step wizard, plus a result screen. The steps are Hypothesis & flag, Metrics, Plan, and Review & lock. The result screen is Started (or, on a rollback, back to draft).

The Plan step has no field to fill in. Quoin chooses weeks automatically. It moves enrolment forward at the current traffic of the workspace until the plan can detect a default target lift of 25%. The maximum is 26 weeks. Traffic always drives the sample size (eligibleUnits over the chosen window). Quoin never derives it from the target lift. The arms are always the two boolean values of the flag, split 50/50. You edit the margin of a guardrail in points of its own baseline. Quoin stores it as the relative fraction of checkDesign. The first experiment of a workspace shows a caution banner that suggests an A/A test first. This is never a refusal. You can still lock and start the experiment.

Locking runs two writes in sequence. Quoin creates the draft, and then starts it. If the second write is refused or throws an error, Quoin rolls the experiment back to draft. Nothing is exposed, and the flag is not changed. The wizard shows this as a "Couldn't start, rolled back to draft" screen with a re-fetched, current record. A "Back to draft" link reopens the wizard at Review & lock with the own pre-registration of the draft.

If the rollback write also fails, Quoin says so. It does not claim a draft that it does not have. These things happen:

• The error has start_rollback_failed.
• The screen reads "Not rolled back" and names the state: running, no rule on the flag, recording nothing.
• Quoin sends a structured experiments.rollback_failed line to the logs of the worker.
• The experiment stays as it is until you stop it.

The numbers in "What happens next" on the Started screen come from the plan that you just locked. The first-read estimate is the number of days that both arms need to clear the 100-per-arm floor at the eligible-per-day rate of the plan. The planned end is the own weeks of the plan.

Health checks, before any number

Before a report shows a result, two checks must pass:

• Sample ratio mismatch (SRM). A chi-squared test compares the split of units across arms with the planned ratio. If the p-value of that comparison is below 0.001, Quoin treats assignment as broken and shows no number. See the broken verdict, below.
• Minimum sample size. Every arm must have at least 100 units before a report tries to read a difference. See the too-early verdict.

Both checks run before Quoin computes the verdict, every time that you open the report. You cannot see a headline number for an experiment that fails either check.

The sequential interval is the headline number

Quoin computes two 95% confidence intervals for every comparison:

• A fixed interval. It is valid only at the single look at the planned end of the experiment.
• A sequential (always-valid) interval. It is valid at every look, including today.

The headline of the report is always the sequential interval. For this reason, it is safe to open a running experiment on any day and read the number honestly, with no "you looked too early" penalty.

The six verdicts

A report reaches exactly one of six verdict states. Quoin computes them in this order:

Only broken and too-early are unreadable. For those two states, Quoin shows no lift, no interval and no projection. The other four states all show the number.

A worked example: trial-reminders

The billing.trial_reminder_cadence experiment of Tessellate ran trial-reminders. It compared a control cadence with an earlier reminder. It concluded on 8 September with control n 3,412 (11.6% converted) against early n 3,521 (14.1% converted). Quoin always writes control first, "3,412 / 3,521,". The sample ratio p = 0.19. This is a healthy split, well above the 0.001 threshold.

The primary metric read clear: converted.14d rose 21.6% [+5.2, +37.9]. But the guardrail, unsubscribed.14d, also read clear in the wrong direction: +1.6 points [+0.7, +2.5] against a ±0.5-point margin. The whole interval was outside the margin. This is a confirmed guardrail harm. The decision record is rolled back. Quoin shows the projection for the one case where a harmed metric gets a projection, which is the own cost of the guardrail: "Projected at conclusion: 30–220 more conversions a month, against 35–125 more unsubscribes a month." If you accept the rollback, you avoid roughly 80 unsubscribes a month and you forgo roughly 125 conversions a month.

Concluding an experiment

An experiment concludes exactly one time. It writes a permanent decision record. On the web, "Conclude now" opens the conclude dialog over the report. "Conclude now" is on a running report that is clear early, or on any stopped report for an editor. The dialog has these items:

• One option card for each arm that you could ship, one card for roll back and one card for inconclusive. Each card has the same projected impact that the next-step cards use.
• A required decision note.
• One Confirm.

Quoin selects the pre-registered default for you. Quoin writes nothing until you confirm.

quoin experiments conclude trial-reminders --rollback --why "guardrail harm confirmed, whole interval outside ±0.5pt margin" --env production

One write, all or nothing. Concluding changes the flag and writes the decision record in the same atomic batch, with one flag revision and one audit row. These are the changes to the flag:

• The rule of the experiment ends.
• A ship serves the chosen arm to everyone.
• A roll back or an inconclusive also drops any rollout rule that was under the experiment. So "the flag returned to control for everyone" is what the flag does, not only what the record says.
• Pins for named people stay.

If anything moved since you opened the dialog, Quoin writes nothing. Something moved if someone else concluded the experiment, or the flag changed. The page then re-reads what the server holds. It never shows a "concluded" that then disappears. If you confirm two times, or two people confirm the same choice at the same time, Quoin still writes one record, one revision and one audit row. Quoin refuses a ship while the kill switch of the flag is off, because the ship would serve the arm to nobody. The dialog greys out the ship card and says so, and roll back is preselected. Turn the flag back on first, or roll back.

Frozen at the moment it stopped. The record keeps the report exactly as it read at conclusion. This is the state as of when the experiment stopped, or now if you conclude straight from running. A concluded report is that frozen report and nothing else. Later data can never move it. Quoin derives its projected impact again from the inputs that are frozen with it, never from today's traffic.

Shipping despite a harmed guardrail is typed, not clicked. If the arm that you would ship has harmed a guardrail (its whole interval past zero in the wrong direction), the dialog asks you to type the name of that guardrail. The server checks it again (confirmHarm). The server refuses a client that skips the dialog in the same way.

The concluded report. The header has the stamp of the decision, ROLLED BACK or SHIPPED. This is the one place where Quoin writes in capitals. The header also has "concluded 8 Sep · sam". Below the frozen verdict and rulers is the decision record. It has the rationale, quoted. It also has the line that says who wrote the record, when, and what happened to the flag ("Sam Reyes · 8 Sep, 18:00 · outcome: rolled back — the flag returned to control for everyone").

A clear experiment that did not ship also shows what it would have done: "Projected at conclusion, had it shipped," for trial-reminders +125 conversions/month (range +30 to +220) against +80 unsubscribes/month (range +35 to +125). If Quoin measured no reachable population (no identified traffic yet), Quoin projects no number, in the dialog or on the record. The dialog says why.

If a guardrail was harmed, the headline says so: "The difference is up — and the guardrail is harmed, past its margin." The same headline shows on a stopped or running report before you conclude it. It always has the own ruler of the guardrail under it, so the claim never stands without its number. The pre-registration under Details keeps every guardrail and its margin ("unsubscribed.14d, a rise of no more than 0.5 pts"). It also says when and by whom the plan was locked ("locked 28 Jul, 10:04 · sam"). Nothing on a concluded report is interactive. The stamp presses one time, directly after you conclude. It is there on every later read.

After Quoin writes a decision record, nobody can re-open it. No product endpoint and no CLI command edits a record. If you change your mind later, start a new experiment. Two operator paths are outside this guarantee. A quoin-admin import can replace the rows of a workspace, and the 30-day purge of a deleted workspace removes them.

Quoin keeps the decision note permanently on the record. Quoin copies it into the audit row and exports it with the audit log. Do not put personal data in it. Only an editor or an owner can conclude. You can instead discard a draft, or a run whose start never reached a single person. Use quoin experiments discard <key> --env production or the API (experiments.discard). No screen offers this action. Anything that ran keeps its record.

What a report will never show

A report never shows a chart of the interval drawn over time, because such a chart invites a reader to see a trend in noise. It never shows a per-user drill-down. It never shows a bare p-value as the headline. It never shows "chance to win." A report does show cumulative exposure over time. This is about how much data arrived, not about the result.

As of QN-07a

A report is a plain JSON object, schemaVersion: 1. Quoin increases this number each time the shape changes. A stale reader can then tell that it reads an old contract. It does not silently misparse a new one. A report has these fields, in addition to key, flagKey, status and verdict:

• phase: an experiment gets a new phase each time it restarts after an edit. A report and its history stay attributable to the run that produced them (see phases, below).
• asOf: the moment when the numbers were computed. It is null for a running experiment that reads "now".
• preregHash: you can check a report against the pre-registration that locked it.
• nobodyExposed and flagEnabled: you can read both without a single number, for a blank or paused experiment.
• sampleRatio and exposed: the health checks and the raw counts for each arm.
• plannedTotalN and day/days: the own target of the plan and where today is against it.
• primary and guardrails: each is a full metric report with control, arms, and both intervals.

A report is one snapshot that Quoin computes new when you read it. There is no live update. Quoin caches nothing across reads. By design, you reload the page to see a newer snapshot. No interval ever moves under you during a read.

experiments.phases lists every phase that an experiment ran through, with its date range and the n that it reached. It is a separate read from the report itself. So a report stays cheap for a viewer who never opens the phase history.

A stop of a running experiment (experiments stop, or the Stop of the report) always takes a required reason. Quoin refuses an empty or whitespace-only string before it writes anything. Quoin stores the reason permanently on the revision and the audit row of the stop. You cannot stop an experiment silently.

Next steps

Every report, in every verdict state, ends in a row of next-step cards. It never shows a spinner and never shows a blank space while something loads. Quoin draws the recommended action first. Each card ends in a quoin CLI command (copy-and-run, always --env production) or, where no action applies, a plain "(no action…)" line. A report never writes anything on its own. A card is never a button that fires a mutation directly.

broken and too-early show no card with a number. They show only "no figure on this report can be trusted" and "not enough readers in either arm yet". not-yet and no-difference show cards but no projected number, for the same reason. Only clear and significant-at-plan can project.

If a card can show a number, the number comes from one formula: impact = control.mean × liftBound × eligibleUnitsPerMonth × Δcoverage. The formula uses both the low bound and the high bound of the sequential interval, never the point estimate alone. The result is a range, never a single promised outcome. eligibleUnitsPerMonth is the same reach estimate that the planner already uses. Quoin carries it over unchanged.

If a guardrail is harmed on an otherwise-clear report, Quoin does the same against the own absolute interval of the guardrail. So "ship anyway despite the harm" always carries the worst-case cost of the guardrail, not only the upside of the primary metric. On a concluded report, the row is the own row of the record. You can keep what the record decided (no action, because it is already the recorded state) or re-test under a new key.

Next