How to Investigate Flaky Automated Tests: Reproduction Steps and Practical Fixes

Flaky automated tests pass in some runs and fail in others even when the relevant code and intended inputs are unchanged. They slow delivery, undermine confidence, and can hide genuine product races. This guide gives a repeatable investigation process with Playwright and pytest examples. For wider pipeline strategy, see Continuous Testing.

How Do You Investigate a Flaky Automated Test?

Capture the original failure, including the commit, test ID, environment, logs, trace, test data, and timing. Reproduce the same test on the same versions. Vary one factor at a time, such as worker count, test order, network behavior, time, or data isolation. Use evidence to identify the root cause, implement a targeted fix, and rerun the original scenario. A passing retry does not prove the failure was harmless.

Step 1: Preserve the First Failure

Record the exact assertion and stack trace, commit SHA, CI runner image, browser and runtime versions, dependency lockfile, random seed, locale, timezone, and relevant fixture identifiers. Capture sanitized application logs, HTTP responses, screenshots, and traces. Retain the failed artifact before any automatic rerun replaces it. Never expose tokens or personal data in shared debugging artifacts.

Step 2: Reproduce the Test in Isolation and Under CI Conditions

Start with the same commit and locked dependencies as the failing job. Run the individual test with one worker. Repeat it to observe intermittent behavior, then use the original CI worker count and surrounding suite. Change one variable per experiment and record the outcome. Playwright documents its runner options in the CLI reference.

npx playwright test tests/booking.spec.ts --workers=1 --trace=on
npx playwright test tests/booking.spec.ts --repeat-each=20 --workers=1
npx playwright test tests/booking.spec.ts --repeat-each=20 --workers=4

For a Python test, use the same interpreter and installed dependency versions as CI. This illustrative shell loop stops on the first failure and assumes a Unix-like shell.

python -m pytest tests/test_booking.py::test_confirm_booking -vv -s
for i in $(seq 1 20); do
  python -m pytest tests/test_booking.py::test_confirm_booking -q || break
done

Step 3: Classify the Failure

Symptom	Likely hypothesis	Investigation
UI element missing after click	Asynchronous rendering	Inspect action and network timeline
Fails only with multiple workers	Shared fixtures or data	Give workers isolated resources
Fails after another test	Order dependency	Reverse test order and inspect cleanup
Intermittent 429 or 503	External service instability	Inspect response and dependency logs
Fails near midnight	Clock or timezone assumption	Freeze time and test boundaries
CI-only timeout	Resource contention or version drift	Compare CI and local environments

These are hypotheses, not confirmed diagnoses. A flaky test can reveal a real application bug, including duplicate writes and concurrency defects.

Step 4: Replace Fixed Sleeps With State-Based Assertions

A fixed delay guesses when the application will be ready. Prefer an assertion that waits for a meaningful state. The following fictional Playwright example illustrates the change.

// Fragile: assumes the response arrives in 500 ms
await page.getByRole('button', { name: 'Confirm booking' }).click();
await page.waitForTimeout(500);
expect(await page.getByText('Booking confirmed').count()).toBe(1);

// Better: auto-retrying assertion
await page.getByRole('button', { name: 'Confirm booking' }).click();
await expect(page.getByText('Booking confirmed')).toBeVisible();

A visible confirmation is not sufficient if the application displays it before the database commit. Verify the authoritative booking record when the business outcome matters. See Playwright assertions.

Step 5: Remove Shared State and Test-Order Dependencies

Give tests unique user accounts, identifiers, database rows, and temporary files. Avoid shared staging credentials and global mutable fixtures. Reset state between tests or use isolated transactions where appropriate. Check cleanup after failure as well as success. Use deterministic service doubles for tests that do not need live providers, while retaining separate contract and integration coverage.

Step 6: Investigate Async Work and Real Product Races

An accepted background job is not necessarily complete. Poll a documented status endpoint within a bounded deadline and inspect the final business effect. For duplicate webhooks and ambiguous timeouts, verify idempotency. If two concurrent reservations create the same booking, fix the database or transaction invariant rather than increasing the test timeout. See Acadify's API Integration Release Checklist.

Worked Example: An Intermittent Booking Confirmation Failure

This is a fictional debugging scenario, not an Acadify client result. A browser test fails only occasionally after clicking Confirm. The trace shows the assertion ran before the booking response completed. The engineer reproduces it under one worker and four workers, replaces the immediate count assertion with a web-first assertion, and checks that exactly one booking exists in the database. The surrounding suite is rerun to exclude fixture interference.

Step 7: Verify the Fix Without Hiding the Failure

Repeat the original scenario under its previous failure conditions, run neighboring tests, and inspect CI over subsequent executions. Report the number of attempts, failures, and environments tested. No finite set of clean runs proves that an intermittent defect is impossible. Automatic retries may help diagnosis but should remain visible in reports. Quarantine only with a named owner, tracking issue, review date, and alternative coverage.

Practical Flaky-Test Investigation Checklist

• Original failure evidence and environment are preserved.
• Reproduction steps are documented and one variable is changed at a time.
• The root cause is supported by traces, logs, or controlled experiments.
• Synchronization uses actual application state rather than arbitrary sleeps.
• Fixtures, data, and external dependencies are appropriately isolated.
• Real product races have deterministic regression tests.
• The fix is verified under original CI conditions and remaining risks are documented.

Frequently Asked Questions

Should every flaky test be rerun automatically?

No. Limited visible reruns can collect evidence, but a passing rerun does not cancel the initial failure or replace a root-cause fix.

Is increasing a timeout ever the right fix?

Yes, if measurements show the legitimate operation needs a larger deadline. It will not repair shared-state races, invalid selectors, or incorrect application behavior.

How many clean runs prove a fix?

None prove impossibility. Choose repeated-run coverage based on the previous failure frequency and impact, and continue monitoring CI.

References

Official documentation: Playwright Best Practices, Assertions, Trace Viewer, CLI, and pytest Usage. The commands and booking scenario are illustrative; adapt them to your test runner, CI infrastructure, and data security requirements.