Table Of Contents
Key Takeaways
- Flaky tests pass and fail on the same code without any change — the cause is always deterministic, never random.
- The 5 root causes are async timing issues, brittle selectors, shared test state, CI environment differences, and external service dependencies.
- Diagnose by pattern: fails only in CI points to environment issues, fails only in the full suite points to shared state, fails on first attempt but passes on retry points to async timing.
- Fixes include dynamic waits instead of hard-coded sleeps, stable data-testid selectors, per-test setup and teardown, pinned CI browser versions, and mocked external APIs.
- Testsigma’s Healer Agent repairs broken selectors automatically at runtime, while the Analyzer Agent classifies failures as real defects, flaky tests, or environment issues before manual triage.
What Makes a Test Flaky?
A flaky test (also called a non-deterministic test) passes sometimes and fails sometimes on the same code. That’s different from a test that always fails, which means you broke something. Flaky tests are harder to deal with precisely because they don’t fail consistently. You can’t reproduce the failure on demand, so you can’t confirm a fix actually worked.
The practical symptom: you re-run the failed test, it passes, and you merge the PR assuming it was a blip. Two days later it fails again in a different pipeline on different data. The code hasn’t changed. Nothing obvious has changed. That’s the signature.
What’s actually happening is that some external condition your test depends on isn’t stable. It might be the timing of an API response, the state left behind by the test that ran before this one, or a CSS selector tied to a class name that gets regenerated on every build. The test was written assuming a controlled environment, and the environment isn’t fully controlled.
The harder question is why it passes at all. Something in the timing, execution environment, or test design is introducing variability. Pinning down test flakiness to a single source is the whole job. The five causes below cover the vast majority of cases.
The 5 Most Common Root Causes
1. Async Timing and Race Conditions
Your test moves faster than the UI or API. It asserts on a button before it’s rendered, or checks data before the request resolves. Replace sleep() with dynamic waits: in Selenium, use WebDriverWait with an explicit condition; in Playwright, use waitForSelector or waitForResponse.
Hard-coded delays mask the problem. Dynamic waits remove it.
2. Brittle Selectors
Auto-generated class names like .css-1x7cjk4 change whenever the component re-renders or the build toolchain updates. Your test worked last week because the class name was different last week. Add data-testid attributes to the elements your tests touch. They’re stable by design. A developer has to explicitly change them, which means selector breakage shows up in code review instead of your CI logs at 2am.
3. Shared Test State
One test creates a user record. Another reads it. A third deletes it. Now execution order determines whether the suite passes, which means you don’t have a test suite: you have a sequence of steps that only works one way.
Each test needs to own its setup and teardown. Seed your database per test, not once before the suite. The ISTQB Software Testing Standard calls this “test independence” and treats it as foundational precisely because shared state failures are so hard to reproduce. If full isolation isn’t feasible, scope state to the test file so failures don’t spread.
4. Environment Differences
Your test passes locally and fails in CI. CI runners use different CPU and memory allocations, different browser versions, different OS configurations. A Playwright scroll interaction that works on a local M2 Mac can time out on a 2-core CI runner.
Pin browser versions in your CI config. Log the full runtime environment on each run. Match your local memory limits to what the runner actually has. Most teams discover this the hard way, usually the afternoon before a release.
5. External Service Dependencies
Your test calls a live payment API, a real SMS provider, or a third-party auth service. That service returns a 429, slows down, or quietly changes a response field. Your code is fine. The test fails anyway.
Mock external APIs at the test boundary. In Playwright, use page.route() to intercept and stub responses. In Selenium, point the service layer at a local mock server. The W3C WebDriver specification defines a clean protocol for controlling the browser, but anything outside the browser is your responsibility to control. External services introduce variance you can’t fix; mocks give you a stable contract.
How to Diagnose Your Cause
Run through this before you start fixing:
Fails only in CI, passes locally? Environment issue. Check browser versions, available memory, and network differences between your laptop and the CI runner.
Fails only when run with the full suite, passes in isolation? Shared state. Add logging around the setup and teardown of the test that runs immediately before the failing one.
Fails on the first attempt but passes on retry? Async timing. The test is outrunning the application. Add an explicit wait targeted at the specific condition you need. Don’t add a retry policy and call it fixed.
Fails on specific data inputs but not others? Test data problem. Seed the exact records you need in the test setup rather than depending on whatever happens to be in the shared database.
How Testsigma’s Healer Agent Fixes Flakiness Automatically
Diagnosing and fixing flaky tests manually works. It also takes time you probably don’t have when a release is queued.
What Self-Healing Actually Means at the Platform Level
Self-healing test automation means the test repairs itself when something changes: a selector, a page structure, a component name. Testsigma does this at runtime. When a locator fails, the Healer Agent finds the element through alternate attributes, updates the test step, and flags the change for your review. The suite keeps running. You review the repairs in bulk rather than hunting each one down.
Analyzer Agent: Classifying Failures before They Waste Your Time
Testsigma AI agents include an Analyzer that classifies each failure as a real defect, a flaky test, or an environment issue before you open a single log. Instead of triaging 40 failed reports to find the 3 that matter, you get a sorted breakdown: actual bugs, environment failures, flakiness. That separation cuts triage time by hours per release cycle.
Up to 90% Reduction in Test Maintenance
Teams using Testsigma report up to 90% less time spent on test maintenance, mostly from eliminating the repair loop on selector changes and layout shifts.
FAQ’s
The 5 root causes are async timing errors, brittle selectors, shared test state, environment differences between CI and local machines, and external service dependencies. Test flakiness always has a deterministic cause. The challenge is identifying which one applies rather than adding retries and hoping it goes away.
Start with the diagnosis: does it fail only in CI, only in a full suite run, only on retry, or only on specific data? Each pattern points to a different cause. Fix the cause directly. Adding retries masks the problem; fixing the selector or adding a proper async wait removes it.
Almost always an environment difference. CPU, memory, browser version, and network speed all vary between a local machine and a CI runner. Pin your browser version in CI, match resource limits to the runner, and log the runtime environment in your test output so you can trace what changed.
