Fix Failing or Flaky Tests
Diagnose and fix a Playwright test that fails or passes intermittently using a systematic taxonomy.
Input
$ARGUMENTS contains:
- A test file path:
e2e/login.spec.ts - A test name: ""should redirect after login"`
- A description:
"the checkout test fails in CI but passes locally"
Steps
1. Reproduce the Failure
Run the test to capture the error:
npx playwright test <file> --reporter=listIf the test passes, it's likely flaky. Run burn-in:
npx playwright test <file> --repeat-each=10 --reporter=listIf it still passes, try with parallel workers:
npx playwright test --fully-parallel --workers=4 --repeat-each=52. Capture Trace
Run with full tracing:
npx playwright test <file> --trace=on --retries=0Read the trace output. Use /debug to analyze trace files if available.
3. Categorize the Failure
Load flaky-taxonomy.md from this skill directory.
Every failing test falls into one of four categories:
| Category | Symptom | Diagnosis |
|---|---|---|
| Timing/Async | Fails intermittently everywhere | --repeat-each=20 reproduces locally |
| Test Isolation | Fails in suite, passes alone | --workers=1 --grep "test name" passes |
| Environment | Fails in CI, passes locally | Compare CI vs local screenshots/traces |
| Infrastructure | Random, no pattern | Error references browser internals |
4. Apply Targeted Fix
Timing/Async:
- Replace
waitForTimeout()with web-first assertions - Add
awaitto missing Playwright calls - Wait for specific network responses before asserting
- Use
toBeVisible()before interacting with elements
Test Isolation:
- Remove shared mutable state between tests
- Create test data per-test via API or fixtures
- Use unique identifiers (timestamps, random strings) for test data
- Check for database state leaks
Environment:
- Match viewport sizes between local and CI
- Account for font rendering differences in screenshots
- Use
dockerlocally to match CI environment - Check for timezone-dependent assertions
Infrastructure:
- Increase timeout for slow CI runners
- Add retries in CI config (
retries: 2) - Check for browser OOM (reduce parallel workers)
- Ensure browser dependencies are installed
5. Verify the Fix
Run the test 10 times to confirm stability:
npx playwright test <file> --repeat-each=10 --reporter=listAll 10 must pass. If any fail, go back to step 3.
6. Prevent Recurrence
Suggest:
- Add to CI with
retries: 2if not already - Enable
trace: 'on-first-retry'in config - Add the fix pattern to project's test conventions doc
Output
- Root cause category and specific issue
- The fix applied (with diff)
- Verification result (10/10 passes)
- Prevention recommendation