iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Test automation proves its value after the application changes. When a test fails, determine first whether expected user behavior changed, the test relied on an incidental implementation detail, or the run was nondeterministic. Each cause calls for a different response: update an outdated expectation, make a brittle check more resilient, or investigate flakiness without masking a genuine defect.
Why tests fail after a change
A failing test is a signal to investigate, not an automatic verdict on the product. The same red result can point to three different problems:
- Behavior changed intentionally. The product now works differently, so the test’s old expectation may no longer be correct.
- The test was coupled to implementation. A CSS class, DOM hierarchy, timing assumption, or shared state changed even though the user-visible behavior did not.
- The run was nondeterministic. The same code can pass on one run and fail on another because of timing, dependencies, shared resources, or the execution environment.
Start by comparing the failing assertion with the intended user behavior and the product change. If the expected behavior changed, update the test as part of the same change. If it did not, inspect the test’s assumptions and the run conditions. Google’s testing guidance recommends checking application and test changes together, reviewing revision history, and rerunning suspected flaky tests independently. Google Testing Blog guidance on test flakiness.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteWhat makes an automated test resilient?
Assert what a user can observe
Prefer checks based on visible content, accessible names, and user actions over selectors tied to styling or internal markup. A test that confirms a user can submit a form and see a success message is less exposed to a harmless change in a container’s CSS class than one that asserts the class directly. Playwright’s guidance recommends testing user-visible behavior and using locators resilient to DOM changes. Playwright best practices.
Keep end-to-end tests focused on meaningful journeys. Duplicating many overlapping checks at the browser level can make a suite slower and increase the number of tests that need attention when a flow changes. Use the test layer that gives the clearest, most useful signal for each behavior.
Wait for a defined state, not a guessed delay
A fixed pause assumes the application will always finish within that interval. If it is too short, the test races the application; if it is unnecessarily long, every run is slower. Synchronize on an observable condition instead, such as a specific element becoming available or an expected response completing. The appropriate condition depends on the behavior under test; the key is to wait for the state the test needs rather than for an arbitrary number of seconds. Google Testing Blog warns that arbitrary delays can become flaky again while also slowing a test suite. Google Testing Blog guidance on test flakiness.
Keep tests independent
A test should not require a particular order, rely on leftovers from a prior run, or mutate data that another test assumes is unchanged. Give tests controlled data and arrange their starting conditions so they can run on their own. Independence makes a failure easier to reproduce and helps distinguish a defect in the target behavior from contamination by another test.
Choose framework features, but design the suite yourself
Frameworks can provide useful locators, interaction tools, waiting behavior, and diagnostics, but none removes the need for sound test design. The Selenium project documentation puts the boundary plainly: “Selenium provides tools to make functional user interaction easier, but does not help you write well-architected test suites.” Selenium test practices.
There is no evidence here for a universal Selenium-versus-Playwright winner. Compare the approaches against your actual needs: user-facing assertions, locator stability, state-based synchronization, failure diagnostics, isolation, browser and environment coverage, and fit with your team’s language and CI system. Playwright documents recommended practices; that is not proof its tests cannot become brittle. Selenium supplies interaction tools; that does not determine the architecture of the suite.
How to triage a failing test
- Check the intended behavior. Read the product change and the test assertion. Decide whether the user’s expected outcome changed. If so, update the expectation deliberately rather than weakening it until the test passes.
- Inspect the test’s assumptions. Look for selectors based on styling or markup structure, fixed waits, order dependencies, shared mutable data, and assumptions about a service or response that may not hold.
- Reproduce the failure in isolation. Run the suspected test independently and compare its result with the suite run. A change in outcome suggests a possible interaction, state, or environment issue; it does not by itself prove the product is correct.
- Check the whole execution path. Review test code and data, the runner and framework, the application and dependent services, and the operating system, hardware, network, and shared resources. Inspect logs and revision history, and confirm that the application and its dependencies started as expected.
- Choose a response that preserves the signal. Correct an obsolete expectation when behavior changed; remove accidental coupling when behavior did not; investigate intermittent results as flakiness. Retries may help gather evidence, but should not become a way to hide persistent failures.
Flakiness is not necessarily confined to test code. Google Testing Blog identifies potential causes across the test, framework, application and dependencies, operating system, hardware, network, and shared resources. Its discussion of flakiness and triage supports treating the test system as a whole rather than reflexively blaming one layer.
Rank #4
Why reliable failure signals matter
When tests repeatedly raise false alarms, people can become less likely to trust a failure. But suppressing failures or accepting retries without investigation can hide a real regression. The goal is not simply a green build; it is a result the team can use to make a sound release decision.
John Micco’s 2016 account of Google’s own test corpus illustrates the problem, but its figures are historical and Google-specific, not current industry rates: about 1.5% of all test runs were flaky; almost 16% of tests had some level of flakiness; and about 84% of observed transitions from passing to failing involved a flaky test. Micco’s account of flaky tests at Google.
Best Value
A separate 2017 Google Testing Blog article reported around 4.2 million tests running on Google’s continuous integration system at that time. That is historical scale context, not a current count or a measure of maintenance effort. Where do our flaky tests come from?
Build change validation into delivery
Tests are most useful when they participate in the flow of proposed changes, so failures arrive while the relevant code and context are still clear. Google Cloud describes frequent change proposals and validation, with presubmit checks that can include unit and integration tests as well as static and dynamic analysis. Google Cloud’s approach to change.
That is an example, not a mandatory checklist for every team. Choose checks according to the risks in your system, and make each check’s role clear: what behavior it protects, what evidence a failure provides, and who should investigate it. For further reading on brittle tests and maintenance, see the testing chapter of Software Engineering at Google.
Free tools Windows power users keep installed
One-click scans. No signup required.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
