Flaky Test Quarantine: Unreliable Tests Out of the Blocking Run, Each With an Owner

A red build that nobody believes any more

Flaky tests teach a team to rerun and ignore. A quarantine policy pushes back. A test that fails intermittently above the threshold leaves the blocking run and gets a ticket for its owner, a test in a critical path suite is never quarantined and stays blocking, and each team gets a capped number of new flaky tickets a week so the backlog stays honest. The weekly digest reports what was quarantined and the CI minutes those tests were costing.

Test history lives in Airtable, fixes and ownership changes show up in GitHub pull requests, tickets go into ClickUp, and the policy, including the critical path suite list, is a Google Drive document. The digest is assembled by hand. When an owner says a test is fixed, an incident ties a test to a production failure or the suite list changes, the digest and the tickets stop agreeing, and the minutes figure quietly includes tests whose tickets were never opened.

Who works in the room

In the CI health room, Fin answers by default and owns classes and tickets, while the room guidelines give the digest audit and every owner message to Ada. Fin uses a lasting grant on the Airtable base that the build lead approved for the room; Ada drafts in Gmail as the engineer it is working for, checked on every call. Teams that use a coding agent such as Codex can add it to the room too: once a quarantine ticket exists, the owner can ask it to attempt a fix, and the owner reviews that pull request like any other.

Classification to closeout in one week

The week runs in a fixed order. Airtable and ClickUp change only after the build lead approves, and the quarantine list in the repository changes only through a pull request an engineer merges.

Scoping every change to the tests it touches

Replies trickle in through Slack and email all week. One owner says a pull request merged yesterday fixed the test, another says it belongs to a different team. Fin re-runs only the test in question, checks the fix against the pull request in GitHub, and counts a reassigned ticket against the new team's cap. An incident link works in one direction only: a test tied to a production failure goes back to blocking whatever its flake rate.

A change to the critical path list is scoped by suite. When the list adds the checkout suite and drops the admin suite, Fin re-runs the tests in those two suites and leaves every other class alone. Tests that leave the critical path can now be quarantined, and their tickets compete for the cap like any other. The closeout figure comes last, from the records as they stand, and counts only quarantined tests whose tickets actually exist.

What the checks catch

• CI minutes flagged that include quarantined tests whose tickets are still queued over the cap
• A fifth ticket for a team capped at four, including one that arrived by reassignment
• A digest line with more than one problem that carries a lower-precedence code, and tests the digest left out entirely
• A test in a critical path suite that was quarantined anyway because its flake rate was high
• Tests outside the changed suites that moved after the suite list update

What stays with people

• Approving each class and ticket before Airtable or ClickUp changes
• Merging any change to the quarantine list, and any fix a coding agent proposes
• Confirming that a test is tied to an incident
• Sending owner messages, which Ada prepares as Gmail drafts

Integrations in this workflow

Kylon is not a CI system or a test runner, and it does not replace either. It reads from the test history, the code host and the tracker the team already uses, and keeps the quarantine decisions between them in one place.

• Airtable
• ClickUp
• GitHub
• Gmail
• Google Drive
• Slack

• Airtable holds each flaky test with its run history, owner, class and CI minutes
• GitHub shows the pull requests that fixed, touched or moved ownership of a test
• ClickUp carries the quarantine tickets, created only after the build lead approves
• Google Drive keeps the quarantine policy, the critical path suite list and the weekly digest
• Gmail carries owner messages once a person sends Ada's draft
• Slack is where owners reply and incident links are posted

Where to start

One repository's flaky list and one week is enough to begin. Connect the Airtable base, share the policy with the room, and add a single rule to the room guidelines: a test in a critical path suite is never quarantined, and no ticket opens over a team's cap unless the build lead says so.

Set it up with your agent

Open a new Room, add your agent and paste this prompt. It connects the tools, builds the App with you and asks for your rules one step at a time, and nothing goes out until a person approves it.