---
title: "How to Investigate Flaky Automated Tests: Reproduction Steps and Practical Fixes"
author: "Acadify Engineering Team"
date: "October 11, 2026"
description: "Learn to investigate flaky automated tests with Playwright and pytest commands, root-cause diagnosis, synchronization fixes, data isolation and CI verification."
categories: ["Quality Assurance"]
---

Canonical URL: https://acadifysolution.com/blogs/post/investigate-flaky-automated-tests-reproduction-fixes

# How to Investigate Flaky Automated Tests: Reproduction Steps and Practical Fixes

By **Acadify Engineering Team** on October 11, 2026

Flaky automated tests pass in some runs and fail in others even when the relevant code and intended inputs are unchanged. They slow delivery, undermine confidence, and can hide genuine product races. This guide gives a repeatable investigation process with Playwright and pytest examples. For wider pipeline strategy, see [Continuous Testing](https://acadifysolution.com/blogs/post/continuous-testing-business-growth-2026).




## How Do You Investigate a Flaky Automated Test?




Capture the original failure, including the commit, test ID, environment, logs, trace, test data, and timing. Reproduce the same test on the same versions. Vary one factor at a time, such as worker count, test order, network behavior, time, or data isolation. Use evidence to identify the root cause, implement a targeted fix, and rerun the original scenario. A passing retry does not prove the failure was harmless.




## Step 1: Preserve the First Failure




Record the exact assertion and stack trace, commit SHA, CI runner image, browser and runtime versions, dependency lockfile, random seed, locale, timezone, and relevant fixture identifiers. Capture sanitized application logs, HTTP responses, screenshots, and traces. Retain the failed artifact before any automatic rerun replaces it. Never expose tokens or personal data in shared debugging artifacts.




## Step 2: Reproduce the Test in Isolation and Under CI Conditions




Start with the same commit and locked dependencies as the failing job. Run the individual test with one worker. Repeat it to observe intermittent behavior, then use the original CI worker count and surrounding suite. Change one variable per experiment and record the outcome. Playwright documents its runner options in the [CLI reference](https://playwright.dev/docs/test-cli).




```
npx playwright test tests/booking.spec.ts --workers=1 --trace=on
npx playwright test tests/booking.spec.ts --repeat-each=20 --workers=1
npx playwright test tests/booking.spec.ts --repeat-each=20 --workers=4
```




For a Python test, use the same interpreter and installed dependency versions as CI. This illustrative shell loop stops on the first failure and assumes a Unix-like shell.




```
python -m pytest tests/test_booking.py::test_confirm_booking -vv -s
for i in $(seq 1 20); do
  python -m pytest tests/test_booking.py::test_confirm_booking -q || break
done
```




## Step 3: Classify the Failure




| Symptom | Likely hypothesis | Investigation |
| --- | --- | --- |
| UI element missing after click | Asynchronous rendering | Inspect action and network timeline |
| Fails only with multiple workers | Shared fixtures or data | Give workers isolated resources |
| Fails after another test | Order dependency | Reverse test order and inspect cleanup |
| Intermittent 429 or 503 | External service instability | Inspect response and dependency logs |
| Fails near midnight | Clock or timezone assumption | Freeze time and test boundaries |
| CI-only timeout | Resource contention or version drift | Compare CI and local environments |




These are hypotheses, not confirmed diagnoses. A flaky test can reveal a real application bug, including duplicate writes and concurrency defects.




## Step 4: Replace Fixed Sleeps With State-Based Assertions




A fixed delay guesses when the application will be ready. Prefer an assertion that waits for a meaningful state. The following fictional Playwright example illustrates the change.




```
// Fragile: assumes the response arrives in 500 ms
await page.getByRole('button', { name: 'Confirm booking' }).click();
await page.waitForTimeout(500);
expect(await page.getByText('Booking confirmed').count()).toBe(1);

// Better: auto-retrying assertion
await page.getByRole('button', { name: 'Confirm booking' }).click();
await expect(page.getByText('Booking confirmed')).toBeVisible();
```




A visible confirmation is not sufficient if the application displays it before the database commit. Verify the authoritative booking record when the business outcome matters. See [Playwright assertions](https://playwright.dev/docs/test-assertions).




## Step 5: Remove Shared State and Test-Order Dependencies




Give tests unique user accounts, identifiers, database rows, and temporary files. Avoid shared staging credentials and global mutable fixtures. Reset state between tests or use isolated transactions where appropriate. Check cleanup after failure as well as success. Use deterministic service doubles for tests that do not need live providers, while retaining separate contract and integration coverage.




## Step 6: Investigate Async Work and Real Product Races




An accepted background job is not necessarily complete. Poll a documented status endpoint within a bounded deadline and inspect the final business effect. For duplicate webhooks and ambiguous timeouts, verify idempotency. If two concurrent reservations create the same booking, fix the database or transaction invariant rather than increasing the test timeout. See Acadify's [API Integration Release Checklist](https://acadifysolution.com/blogs/post/api-integration-release-checklist-authentication-timeouts-retries-recovery).




## Worked Example: An Intermittent Booking Confirmation Failure




This is a fictional debugging scenario, not an Acadify client result. A browser test fails only occasionally after clicking Confirm. The trace shows the assertion ran before the booking response completed. The engineer reproduces it under one worker and four workers, replaces the immediate count assertion with a web-first assertion, and checks that exactly one booking exists in the database. The surrounding suite is rerun to exclude fixture interference.




## Step 7: Verify the Fix Without Hiding the Failure




Repeat the original scenario under its previous failure conditions, run neighboring tests, and inspect CI over subsequent executions. Report the number of attempts, failures, and environments tested. No finite set of clean runs proves that an intermittent defect is impossible. Automatic retries may help diagnosis but should remain visible in reports. Quarantine only with a named owner, tracking issue, review date, and alternative coverage.




## Practical Flaky-Test Investigation Checklist



- Original failure evidence and environment are preserved.
- Reproduction steps are documented and one variable is changed at a time.
- The root cause is supported by traces, logs, or controlled experiments.
- Synchronization uses actual application state rather than arbitrary sleeps.
- Fixtures, data, and external dependencies are appropriately isolated.
- Real product races have deterministic regression tests.
- The fix is verified under original CI conditions and remaining risks are documented.




## Frequently Asked Questions




### Should every flaky test be rerun automatically?




No. Limited visible reruns can collect evidence, but a passing rerun does not cancel the initial failure or replace a root-cause fix.




### Is increasing a timeout ever the right fix?




Yes, if measurements show the legitimate operation needs a larger deadline. It will not repair shared-state races, invalid selectors, or incorrect application behavior.




### How many clean runs prove a fix?




None prove impossibility. Choose repeated-run coverage based on the previous failure frequency and impact, and continue monitoring CI.




## References




Official documentation: [Playwright Best Practices](https://playwright.dev/docs/best-practices), [Assertions](https://playwright.dev/docs/test-assertions), [Trace Viewer](https://playwright.dev/docs/trace-viewer), [CLI](https://playwright.dev/docs/test-cli), and [pytest Usage](https://docs.pytest.org/en/stable/how-to/usage.html). The commands and booking scenario are illustrative; adapt them to your test runner, CI infrastructure, and data security requirements.


---
### About the Author
**Acadify Engineering Team**
The editorial team publishes practical guides about software development and AI evaluation.