Google ran the numbers: 84% of their test failures that flip from pass→fail are flaky — not real bugs. Flaky tests are the tax almost every CI pays and almost no CI fixes. What flaky tests actually cost, and how to kill them — a thread 🧵

The better question isn't "re-run or mute?" — it's "is this red signal or noise?" Testhide answers it with a Flakiness Predictor: gradient-boosted trees over 41 features, scoring each failure — its history (top signal: recent fail-rate) build context. Real regression → block the PR. Noise → smart-quarantine. 👇

Your CI is green. Your ML models are quietly going wrong anyway. A model trained on last quarter's data drifts as the real world shifts under it — accuracy decays, retraining never fires, and not one test turns red. The blind spot almost no CI/CD covers 🧵

Why Testhide is different: • plain CI (Jenkins, GitHub Actions, TeamCity) → no model monitoring at all • eval tools (Braintrust, Langfuse) → a separate dashboard, not your pipeline, no per-model input drift or retrain trigger We run it in-pipeline, per real input, self-hosted.

Your AI agent passed 41 unit tests. The eval suite was green. Your teammate merged and went home. By Monday it was quietly denying refunds it should approve — in ~1 of 20 real conversations. No error. No alarm. Nothing "failed." Why green CI lies about agents 🧵

But your golden set goes stale the day you freeze it. So the same models keep watching prod. An OOD detector flags inputs that look like nothing it's seen → you turn that into a fixture → the next PR's eval defends against a failure your users found for you. The loop closes.