Experiment Readouts: Ship Calls That Stay Right When the Data Is Corrected

Ship calls that are right on Monday and wrong by Friday

Readouts follow a rule order. A guardrail metric that moved the wrong way stops a ship call before anything else is read, a sample below the minimum makes the result inconclusive, a primary metric past the ship threshold is a ship, and everything else is iterate or stop. Each call then gets a follow-up tier, from ship this sprint to archive. Applying the rules to a sprint of experiments takes a morning. Keeping the calls right while the data is corrected takes the rest of the cycle.

Experiments are registered in Notion, metrics come from Google Analytics, and the flag pull requests in GitHub show when each variant actually went live. The readout summary is a Google Drive document an analyst writes from all three. When a sample is recounted, a guardrail is retracted or the ship threshold moves, the summary, the Notion records and what each team thinks it is shipping drift apart.

Who works in the room

There is one Kylon room per quarter for experimentation, and any message that names no agent reaches Ryan, which owns the calls and is the room's default. Myco audits the analyst's summary and checks every re-derived list, and Fin writes the team pages and drafts replies, as the room guidelines set out. After a correction, Ryan re-derives the affected calls and mentions Fin with the changed experiment ids, so replies and pages are written from the corrected list.

From closed experiments to team pages

The cycle starts when the sprint's experiments close and ends with one Notion page per team. Calls reach Notion only after the experimentation lead approves them.

Recounts, retractions and a higher bar

A recount is narrow and stays narrow. When the analyst reprocesses one experiment's sample, Ryan re-runs that experiment alone, writes the new call to its Notion record with a note once the lead approves, and Fin drafts the reply on the recount thread naming the corrected experiment id. Nothing else moves.

A retraction or a threshold change is wide by design. When the checkout guardrail is retracted, it applies to every checkout experiment that used it, including ones the notice did not list. When the ship threshold is raised, every call is re-derived under the whole rule order, because a rule higher in the list can now decide a result the threshold used to decide. With a quarter of experiments in play, Ryan splits the re-derivation across parallel helpers and posts one list of changed calls, which Myco checks before the lead approves.

What the checks catch

• A ship call that survived a guardrail moving the wrong way because the threshold was read first
• Checkout experiments that used the retracted guardrail but were missing from the retraction notice
• An experiment whose window was measured from its scheduled start instead of the day its flag pull request merged
• A call that only changes when the full rule order is re-run after the threshold revision
• A team page with fields or sections the agreed format does not have

What stays with people

• Approving every call and follow-up tier before Notion is updated
• Retracting a guardrail or moving the ship threshold
• Deciding what actually ships, which stays with each product manager and their engineering lead
• Sending the replies on the recount and threshold threads, which Fin drafts in Gmail

Integrations in this workflow

Kylon is not an experimentation platform and it does not replace one. It sits across the analytics, the code host and the documents the team already uses, and keeps the readout coordination in one room.

• GitHub
• Gmail
• Google Analytics
• Google Drive
• Notion
• Slack

• Notion holds the experiment records, the rules page and the team pages
• Google Analytics supplies the primary and guardrail metrics for each variant
• GitHub shows the flag pull requests that mark when each variant went live
• Google Drive keeps the analyst's summary and the recount files
• Gmail carries replies on the recount and threshold threads; Fin drafts them and a person sends
• Slack is where analysts flag data problems and where the team pages are announced after review

Where to start

Take one sprint's closed experiments. Connect the Notion database where experiments are registered and the Google Analytics property that holds their metrics, and set one rule in the room guidelines: the rule order is applied in full every time, and no call changes in Notion until the experimentation lead approves it.

“The threshold moved once this quarter. Every call was re-derived under the whole rule order, and the lead reviewed only the ones that changed.”

Set it up with your agent

Open a new Room, add your agent and paste this prompt. It connects the tools, builds the App with you and asks for your rules one step at a time, and nothing goes out until a person approves it.