Guards

A guard is attached to one flag in one environment. Every hour, the guard reads one rate. An example is failed exports over started exports. The guard compares the rate with a fixed ceiling. If the rate is over the ceiling on two consecutive hourly checks, the guard turns the flag off. The guard keeps the rules of the flag. It writes one revision under its own name. It stops its checks until a person resumes the flag. A guard never turns anything on.

What a guard watches

The signal of a guard is { event, window, numerator, denominator }. The rate is the count of the numerator event over the count of the denominator event in the trailing window for the environment of the flag. The window is 1h, 1d or 7d. Quoin skips backfilled events, so an import of old data cannot trip a guard and cannot clear a guard. An hour with no denominator events has no rate. Quoin records that check as skipped. The count of consecutive checks over the ceiling stays where it was.

The ceiling is an absolute rate that Quoin stores on the guard. It is 2.0%, not "five times the baseline". Quoin keeps the baseline (0.3% in the example below) for the chart on /rollouts/<key> and for the person who reads the chart. The check is a plain comparison of the rate with the ceiling. It has no interval and no test of significance. A guard is not an experiment, and it never reads an experiment.

When it pauses

Checks run on a schedule, one time each hour, if a person uses the flag or not. Each check records one row: clear, over, tripped, would_pause, skipped or error. Two consecutive over checks trip the guard. A check under the ceiling resets the count.

A trip writes off with the actor guard:<name>. The guard composes a note. For the example below, the note is error_rate.1h 4.6% against a 2.0% ceiling, two consecutive checks. The trip also writes the revision id and the audit row. All of these writes are in one atomic batch with the check that tripped the guard. Quoin writes all of the batch or none of it. If two runs overlap, only one of them writes. The other run finds that the hour is already checked and writes nothing.

If a check cannot read its signal or cannot write its result, it records error. It leaves the flag exactly as it was. A guard that cannot read its signal never pauses on a guess. It also never turns a flag back on.

This is the example that the docs and the screens use: exports.csv_v2 is rolled out to 10% in production. A guard named error-rate watches exports.failed over exports.started in a one-hour window. The baseline is 0.3% and the ceiling is 2.0%. Two consecutive hourly checks read 4.6%, so the guard turned the flag off in revision r-0418.

Resuming

A paused flag shows this state on its own page: "Paused at 10% by guard:error-rate on 17 Sep. The rules are kept." It also shows the state on /rollouts/<key> and on Home. If you resume at the coverage that tripped the guard, Quoin asks you to type resume anyway. When a person resumes, a one-hour rate has usually fallen back under the ceiling. So the confirmation is about "a guard turned this off and nothing has changed since". It is not about whether the rate still holds.

To resume without the typed phrase, first roll out at a lower coverage, for example quoin flags rollout exports.csv_v2 5 --env production. Then resume. The resume re-arms the guard with a fresh count. This is true only if the lower coverage is the one change since the pause. Only the coverage of the first rollout rule can go down. Quoin asks for resume anyway in these cases:

• A second rollout rule has a changed coverage.
• Someone added or widened a force rule after the pause. Such a rule is for a tier, a client, a sign-up date or an email domain.
• Someone removed a force rule.
• Someone scoped a rollout to more people, however low the rollout is.

A force rule that names people does not count. Such a rule names emails, user ids or the owner. The same gate holds when the flag comes back on in other ways. quoin flags revert to a pre-pause revision and quoin flags promote from another environment ask for --confirm "resume anyway" on the same terms.

A resume re-arms every guard on the flag in that environment, not only the guard that paused it. A person who turns the flag back on after the kill switch does the same. The next pause again takes two consecutive checks. See Flags and rules for the write path.

Automatic pausing is off until a dry run says it's safe

Every workspace starts with automatic pausing off. When it is off, the guard still checks every hour. A trip records would_pause and writes nothing else. The flag stays on. This is the dry run.

An operator turns automatic pausing on for one workspace with this command:

quoin-admin flags set guards.auto_pause true --workspace tessellate

Quoin refuses the command unless every guard in that workspace, in every environment, has a continuous seven-day run of hourly checks. This run is seven times 24 checks that read their signal, with no gap longer than two hours. The refusal names the guard that is short and gives its number of checks. Quoin never gates the change to off. Every change writes an audit row. No customer and no screen can make this change. It is a decision of an operator, made after the operator reads a week of would_pause rows.

Quoin checks the same seven-day rule again at every trip, guard by guard. If a guard is attached after the change, or its check history has a gap, it records would_pause. It leaves the flag on until it has its own continuous seven-day run. So a new guard never pauses a real flag on its first two checks. The log line of the run gives the reason: reason: "dry_run_incomplete", or reason: "auto_pause_off" when the switch of the workspace is off.

Monitoring the guards

Quoin logs three events as single JSON lines to the logs of the worker. Quoin does not decide to route them to a pager. This is a deployment decision for you.

• guard.pause_unacknowledged: a guard paused a flag 30 minutes ago and no person has written a revision to that flag since. Quoin logs it one time for each pause. A revision from a person of any kind acknowledges the pause. Such a revision is a resume, a rollback or the kill switch.
• guard.meta_alert: the guards have a fault. More than 1% of the checks in the last 15 minutes recorded error, or a run could not start (reason: "run_failed"), or the monitor could not read (reason: "monitor_failed"). Checks run on the hour. The monitor runs again at quarter past, half past and quarter to.
• guard.monitor: the summary of every run. It gives the median time from a pause to the first revision of a person on the flag, for every pause that the check log still holds. It also gives the number of pauses that a person acknowledged and the number that are still open.

The check log keeps rows for as long as Quoin keeps events. Then the hourly sweep deletes the rows.

Runbook: a guard fired — what do I check

1. Open the rollout page of the flag, /rollouts/<key>. It shows the name of the guard, the rate at the trip, the rate now, the baseline and the ceiling. It also shows the three next steps with the commands to run. The flag is off for everyone in that environment. Nobody gets the new path.
2. Check the numerator, not only the rate. A rate over the ceiling can be a real regression or a quiet hour. A quiet hour has a few failures over very few starts. Look at the events behind the numerator and the denominator of the guard for the last two hours.
3. Check what changed. The history of the flag (on /flags/<key>) shows the revision of the guard and every revision of a person before it. A deploy in the same hour as the first over check is the usual cause.
4. Decide, and write the reason down. The recommended path is to fix the cause, roll out at half the coverage, and then resume. This path re-arms the guard with a fresh count. If you resume at the same coverage, Quoin asks you to type resume anyway. The guard already judged this path one time. To leave the flag off is also a decision. The flag is already off.
5. If the pause looks wrong, look for guard.meta_alert lines in the same hour before you trust the reading of a guard. Also look for error rows in the checks of the guard. A guard that has an error never pauses, so a pause always came from a reading. But a wrong signal definition reads wrongly every hour. Detach the guard and attach a corrected guard.
6. Nobody acknowledged it for 30 minutes? Quoin logged guard.pause_unacknowledged. Any revision that you write to the flag acknowledges it. A resume at a lower coverage also acknowledges it.

Next