What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Usually, no—not in bulk. A golden-file failure after a model change is a signal to investigate, not approval to replace every expected output. Regenerate only the affected snapshots, inspect the diff, and accept a new baseline only when the changed behavior is intended. Keep that workflow separate from updating a curated evaluation set used to compare models.
What a golden-file failure tells you
A golden file stores an expected output so a later run can be compared against it. Snapshot testing can reveal that output changed; it cannot decide whether the change is a bug, an acceptable consequence of the model update, or an accidental side effect. TensorFlow Federated describes golden testing as checking outputs against expected files, while the Go Golden library supports human approval of updated snapshots. TensorFlow Federated’s golden-testing guidance and the Go Golden project documentation illustrate those roles.
So “burn the golden files” is useful only as a provocation: investigate and selectively refresh expectations when warranted. It is not a reason to overwrite baselines before reviewing what changed.
How to handle a failed snapshot after a model change
- Identify the affected behavior. Find which tests failed and what output each snapshot represents. A model update may explain a difference, but the failure alone does not establish that the new result is acceptable. For agent evaluations, Google Cloud’s Agent Studio documentation describes comparing an agent’s output with expected results: Evaluation | Agent Studio.
- Regenerate only the relevant outputs. Use the narrowest update mechanism your project supports. Commands and flags vary: SCION documents package-level and repository-wide golden-file update examples, while TensorFlow Federated documents an argument for updating expected files. Do not assume one project’s flag applies to another. See SCION’s Golden Files documentation and TensorFlow Federated’s guidance.
- Review the diff before accepting it. Check whether each changed output is expected, whether unrelated snapshots moved, and whether important cases are missing. TensorFlow Federated specifically advises checking the resulting diff for unanticipated changes. The Go Golden library’s approval mode provides an example of keeping a test failing until a person accepts the new snapshot.
- Approve the baseline explicitly. Accept updated expectations only after deciding that they reflect the intended behavior. If the change violates a user-visible requirement, fix the model, prompt, code, or test rather than updating the golden file to hide the failure.
When output is nondeterministic
If the same test can produce different output across runs, a straightforward byte-for-byte snapshot may fail without a meaningful behavior change. First identify which fields or behaviors are unstable, then use the project’s supported policy for handling them—for example, comparing stable properties or isolating volatile data when the test design permits. Do not silently refresh snapshots until a passing run happens to produce a convenient result.
SCION documents a separate update flag for nondeterministic golden files, distinct from its ordinary golden-file updates. That is a project-specific example, not a universal convention; follow the test framework’s documented controls. SCION’s documentation describes its approach.
Snapshots are not the same as a curated evaluation set
A snapshot typically checks a particular test’s expected output. A curated evaluation set is a collection of inputs and expected outcomes maintained as a reference for assessing model or agent behavior. Replacing snapshots after a deliberate output change may be appropriate; changing an evaluation set is a separate act of test curation.
Golden-Eval describes freezing a specific version as the reference for an evaluation campaign. Keep the inputs, labels or expected outcomes, and version of the set stable while comparing models. Change the set when evidence justifies it—such as a feature change, a newly identified failure, or adversarial testing—and record that change as a new version rather than treating it as an automatic side effect of model updates. See Golden-Eval’s methodology.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep training separate from regression checks
Model testing has concerns that conventional snapshot testing does not capture. Google’s ML Test Score publication cautions against golden tests that partially train a model. Keep model training and regression evaluation conceptually distinct: a regression check should assess behavior against a reference, not quietly alter the model as part of the test. Google Research’s ML Test Score paper discusses this distinction.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteQuick Recap
Best Value
Rank #4
A practical decision rule
- Update a snapshot when its output changed, the change is intended, and the reviewed diff is limited to the affected behavior.
- Do not update it yet when the cause is unclear, the diff contains unrelated changes, or the new output violates expected behavior.
- Handle instability deliberately when output varies between runs; use an explicit project policy rather than treating every failure as a baseline-refresh request.
- Version an evaluation set separately when its inputs or expected outcomes need to change; preserve a stable reference for comparisons within an evaluation campaign.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
