--- title: "Flaky Test Quarantine Workflow With AI Agents | Kylon" description: "How a build team classifies flaky CI tests, opens capped quarantine tickets, audits the weekly digest and handles suite list changes in one Kylon room." url: "https://kylon.io/use-cases/flaky-test-quarantine" language: "en" --- [HOME](https://kylon.io/)\>[Use Cases](https://kylon.io/use-cases)\>Flaky Test Quarantine: Unreliable Tests Out of the Blocking Run, Each With an Owner Use Case8 min read # Flaky Test Quarantine: Unreliable Tests Out of the Blocking Run, Each With an Owner Flaky tests are classified against run history and the critical path suite list, quarantine tickets stay inside each team's cap, and owner replies, incident links and suite changes re-run only the tests they touch. The build lead approves every ticket. [Get started](https://app.kylon.io/sign-up) Kylon TeamProduct November 6, 2026 ## A red build that nobody believes any more Flaky tests teach a team to rerun and ignore. A quarantine policy pushes back. A test that fails intermittently above the threshold leaves the blocking run and gets a ticket for its owner, a test in a critical path suite is never quarantined and stays blocking, and each team gets a capped number of new flaky tickets a week so the backlog stays honest. The weekly digest reports what was quarantined and the CI minutes those tests were costing. Test history lives in Airtable, fixes and ownership changes show up in GitHub pull requests, tickets go into ClickUp, and the policy, including the critical path suite list, is a Google Drive document. The digest is assembled by hand. When an owner says a test is fixed, an incident ties a test to a production failure or the suite list changes, the digest and the tickets stop agreeing, and the minutes figure quietly includes tests whose tickets were never opened. ## Who works in the room People - Build and release lead Owns the quarantine policy and the per-team caps, and approves every class and ticket - Developer productivity engineer Writes the weekly digest and keeps the critical path suite list - Test owners Fix or reassign their flaky tests and reply when something changes - Incident manager Confirms when a test is tied to a production incident Agents - FinQuarantine desk Classifies flaky tests in Airtable from run history, GitHub pull requests and the suite list, and proposes ClickUp tickets within each team's cap AirtableGitHubClickUpGoogle Drive - AdaDigest audit and owners Audits the digest with the policy's error codes, checks the closeout figure and drafts owner messages for a person to send Google DriveAirtable[Gmail](https://kylon.io/integrations/gmail) [Slack](https://kylon.io/integrations/slack) In the CI health room, Fin answers by default and owns classes and tickets, while the room guidelines give the digest audit and every owner message to Ada. Fin uses a lasting grant on the Airtable base that the build lead approved for the room; Ada drafts in Gmail as the engineer it is working for, checked on every call. Teams that use a coding agent such as Codex can add it to the room too: once a quarantine ticket exists, the owner can ask it to attempt a fix, and the owner reviews that pull request like any other. ## Classification to closeout in one week The week runs in a fixed order. Airtable and ClickUp change only after the build lead approves, and the quarantine list in the repository changes only through a pull request an engineer merges. ```mermaid flowchart TD classify["Fin classes each flaky test as
quarantine, blocking or watch"] ok1["Build lead approves the classes"] tickets["Fin proposes tickets per team up to
the cap, worst flake rate first"] ok2["Build lead approves before
any ClickUp ticket is created"] audit["Ada gives each digest line the
highest-precedence error code"] fixd["Developer productivity engineer
corrects it, the build lead confirms"] reply([An owner reports a fix
or a reassignment]) suite([The critical path
suite list changes]) incident["Incident manager confirms a link,
the test stays blocking"] rerun["Fin re-runs only the tests
the change touches"] ok3["Build lead approves each change"] draft["Ada drafts the owner reply
for a person to send"] close["Fin totals each team, counting CI
minutes only where tickets exist"] sign["Build lead signs off"] classify --> ok1 --> tickets --> ok2 --> audit --> fixd --> reply --> rerun suite & incident --> rerun rerun --> ok3 --> draft --> close --> sign class classify,tickets,audit,rerun,draft,close agent class ok1,ok2,fixd,incident,ok3,sign person class reply,suite event classDef agent fill:#FFFFFF,stroke:#DCD3C8,color:#1B1712 classDef person fill:#EEF0E3,stroke:#6E7447,color:#1B1712 classDef event fill:#F4ECDD,stroke:#F4ECDD,color:#564C42 ``` | Stage | What happens | Who acts | Who decides | | --- | --- | --- | --- | | Classification | Each flaky test is classed quarantine with ticket, blocking or watch from its run history, recent pull requests and the critical path list | Fin, from Airtable run history, GitHub and the Drive policy | The build lead approves the classes | | Tickets | Quarantine tickets open per team up to the weekly cap, worst flake rate first, and the rest queue | Fin | The build lead approves before any ClickUp ticket is created | | Digest audit | Each digest line is checked and given the highest-precedence error code, including MISSING\_ENTRY for tests the digest left out | Ada | The digest author corrects it and the lead confirms | | Owner replies | A fix or a reassignment re-runs only that test, and a reassigned ticket counts against the new team's cap | Fin, then Ada for the reply | The build lead approves the change | | Incident links | A test tied to an incident comes out of quarantine and stays blocking | Fin | The incident manager confirms the link | | Suite list update | Only tests in suites that joined or left the critical path list are re-run | Fin | The build lead approves the changed classes | | Closeout | Totals per team and CI minutes flagged, counted only for quarantined tests whose tickets were created | Fin, with Ada checking the figure | The build lead signs off | Stages of a weekly flaky test cycle. Tests, teams, caps and figures in this example are illustrative. ## Scoping every change to the tests it touches Replies trickle in through Slack and email all week. One owner says a pull request merged yesterday fixed the test, another says it belongs to a different team. Fin re-runs only the test in question, checks the fix against the pull request in GitHub, and counts a reassigned ticket against the new team's cap. An incident link works in one direction only: a test tied to a production failure goes back to blocking whatever its flake rate. A change to the critical path list is scoped by suite. When the list adds the checkout suite and drops the admin suite, Fin re-runs the tests in those two suites and leaves every other class alone. Tests that leave the critical path can now be quarantined, and their tickets compete for the cap like any other. The closeout figure comes last, from the records as they stand, and counts only quarantined tests whose tickets actually exist. ## What the checks catch - CI minutes flagged that include quarantined tests whose tickets are still queued over the cap - A fifth ticket for a team capped at four, including one that arrived by reassignment - A digest line with more than one problem that carries a lower-precedence code, and tests the digest left out entirely - A test in a critical path suite that was quarantined anyway because its flake rate was high - Tests outside the changed suites that moved after the suite list update ## What stays with people - Approving each class and ticket before Airtable or ClickUp changes - Merging any change to the quarantine list, and any fix a coding agent proposes - Confirming that a test is tied to an incident - Sending owner messages, which Ada prepares as Gmail drafts ## Integrations in this workflow Kylon is not a CI system or a test runner, and it does not replace either. It reads from the test history, the code host and the tracker the team already uses, and keeps the quarantine decisions between them in one place. - Airtable - ClickUp - GitHub - [Gmail](https://kylon.io/integrations/gmail) - Google Drive - [Slack](https://kylon.io/integrations/slack) - **Airtable** holds each flaky test with its run history, owner, class and CI minutes - **GitHub** shows the pull requests that fixed, touched or moved ownership of a test - **ClickUp** carries the quarantine tickets, created only after the build lead approves - **Google Drive** keeps the quarantine policy, the critical path suite list and the weekly digest - **Gmail** carries owner messages once a person sends Ada's draft - **Slack** is where owners reply and incident links are posted ## Where to start One repository's flaky list and one week is enough to begin. Connect the Airtable base, share the policy with the room, and add a single rule to the room guidelines: a test in a critical path suite is never quarantined, and no ticket opens over a team's cap unless the build lead says so. ## Set it up with your agent Open a new Room, add your agent and paste this prompt. It connects the tools, builds the App with you and asks for your rules one step at a time, and nothing goes out until a person approves it. Set-up prompt Set up our flaky test quarantine desk in this Room. Take it one step at a time and wait for my OK before you move on. 1. Connect Airtable, ClickUp, GitHub, Gmail, Google Drive and Slack. Tell me which ones still need connecting. 2. Ask me who works this desk: the build and release lead, the developer productivity engineer, the test owners and the incident manager. Help me invite them to this Room. 3. Build a Flaky Tests App with Test, Team, Class and Status columns, one record per test. Pull the current flaky list from the Airtable base. 4. Check each test against its run history in Airtable, recent pull requests in GitHub and the critical path suite list in Google Drive. Propose a class of quarantine with ticket, blocking or watch for each test, and show me the ones that need a decision. 5. Never create a ClickUp ticket and never write to Airtable until the build and release lead approves each class and each ticket. Draft owner messages in Gmail for a person to send, and never send them directly. 6. Write our rules and who approves what into this Room's guideline, including our critical path check and our per-team cap check. 7. After one clean week, offer to turn the weekly quarantine review into an Automation. Before you start, ask me for our rules one at a time. [Get started](https://app.kylon.io/sign-up) Your first company harness. Where humans and agents run your business together. [Get started](https://app.kylon.io/sign-up) ## More Use Cases [View all](https://kylon.io/use-cases) [ ### API Deprecation Migration: Cutoff Waves That Stay Right as Integrations Move Retiring an API version runs through one Kylon room. Agents set a disposition for each customer integration in Attio, schedule cutoff waves from the deprecation policy and track blocking bugs in GitHub, while the API product lead approves every disposition and notice. Software & SaaSEngineering operationsEmail operations ](https://kylon.io/use-cases/api-deprecation-migration)[ ### Backup Restore Verification: Backups Proven Restorable Within Their Class Interval A backup counts only once someone has proved it restores. Agents in the team's Kylon room read each backup-job issue in GitHub against restore reports in Box and live targets in Vercel and propose a verdict by class. The reliability lead approves before any issue changes. Software & SaaSIT & security operations ](https://kylon.io/use-cases/backup-restore-verification)