iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
I had described a Coqui XTTS fine-tuning run as having “diverged on its own.” After two independent audits, I found that the log behind that claim ended at step 72,900—even though the run continued to step 125,039. The complete record changed what the evidence could support: held-out evaluations improved, while the training-loss series that trainproof flagged still ended above its own minimum. That is a finding about logged measurements, not proof that the model’s quality got worse—or better.
How the missing end of the run changed the claim
Before release 0.22.0, I had presented this XTTS run as the one failure in my validation gallery that had not been deliberately injected. The shipped log was only a prefix of the run, so its last “BEST MODEL” line named best_model_49880.pth. I used that checkpoint bookkeeping to support my interpretation.
| Evidence in the shipped prefix | Evidence in the complete run |
|---|---|
Stopped at step 72,900 of 125,039; three evaluations were visible. The last “BEST MODEL” line named best_model_49880.pth. Trainproof repository correction, 2026 |
Continued to step 125,039. The trainer promoted best_model_124700.pth at step 124,700, and six retained held-out evaluations improved from 4.8813 to 2.5894. These are held-out loss measurements, not perceptual-quality scores. Trainproof repository correction, 2026 |
The full record undermines the claim that the run as a whole had diverged. It does not justify the opposite claim that the model necessarily became better. I retained no audio, mean opinion score (MOS), or other perceptual evaluation for this run, so those outcomes are not established by the logged losses.
What the loss measurements do—and do not—show
The narrower statement supported by the evidence is that the per-micro-batch training display loss ended above its own minimum. That describes one series, not the model’s overall quality. Separately, the six retained held-out evaluations improved. The two observations can coexist; neither converts into a perceptual judgment without an evaluation of the model’s outputs.
#1 Best Overall
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
As I wrote in my repository correction, “What the retained evidence supports is narrower than the original claim: the per-micro-batch training display loss ended above its own minimum.”
Why trainproof still returned FAIL
Trainproof’s TP-DIVERGE rule examines one training series. In this event file, the reader selected the denser per-micro-batch series—1,251 points—and discarded the five-point epoch-aggregated series before evaluating the rule. The selected series ended at a loss 1.88 times its minimum; the epoch-aggregated series ended at its minimum. The improving held-out curve did not change which training series the rule evaluated. Trainproof repository correction, 2026
Rank #2
I left the verdict at FAIL rather than changing the threshold to make this single run pass. Adjusting a rule to fit a preferred answer in one case would not establish that the rule is calibrated or that the run had a particular model-quality outcome. Release 0.22.0 did not fix this series-selection limitation.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What else the audit changed
A missing optional dependency is not a failed training run
Previously, if sentencepiece was absent, trainproof tokenizer exited with code 1 and reported FAIL. The corrected behavior is NOT-CHECKED with exit code 2: the tool could not perform that check, which is different from finding a fault in the run. Trainproof repository correction, 2026
A check must not alter the training data order
The Hugging Face callback inspected labels in on_train_begin by opening a fresh iterator over the training data. With a map-style data loader and a random sampler using a generator, creating that iterator could consume generator state and shift the subsequent batch order. The callback’s objective_check option now defaults to false. Trainproof repository correction, 2026
Evidence text must match the measured value
Four evidence strings were corrected because their wording did not accurately describe the measurements. One example: a zero-learning-rate message claimed that 100.0% of steps had a zero learning rate, although the recorded values were all -1e-4. I reported 497 tests for release 0.22.0, with a test added for each correction that failed against the pre-repair code; that is my reported test count, not an independent audit of the tests. The release changed no rule ID, threshold, or detection predicate. Coder Legion republication, 2026
Rank #4
A further caveat remains in the repository: some rule messages describe mechanisms the available log cannot establish. For example, TP-NAN-GRAD says non-finite gradients “reached the optimizer,” although a GradScaler may have skipped the step. Those messages were not changed in 0.22.0; the repository describes correcting them as architectural work. Trainproof repository correction, 2026
Recommended Free Tools
What a deterministic linter can establish
Trainproof describes itself as a deterministic linter for training runs, installed with pip install trainproof. It applies rules to logs and prints evidence; a check that cannot run should be reported as NOT-CHECKED, not treated as a pass. Its findings concern mechanical signals visible in logs. They do not, by themselves, establish whether a model learned the intended task or whether its outputs are useful.
Best Value
That distinction matters because loss can improve while a model learns the wrong mapping. In a separate author-reported validation gallery, I ran 18 Qwen2.5-3B QLoRA experiments across six configurations and three seeds. A shuffled-label run reduced its loss by 62% while learning no useful mapping; comparing it with a known-good baseline exposed the failure. This is a controlled example I reported, not a representative population from which to calculate trainproof’s error rate. BedVibe Studios Engineering, 2026
The injected-fault gallery is a regression suite, not a representative sample of training runs. I have not calibrated the rules against a population with independently known outcomes, so no false-positive rate is established. A deterministic verdict can be repeatable without being a calibrated measure of model success.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to read a linter finding
- Read the evidence string first. Treat it as the reported measurement, then check which series and portion of the run it describes.
- Check the coverage of the log. Confirm that the file includes the full run, not just a prefix, and inspect whether the reported checkpoint and evaluations belong to that complete record.
- Separate measurement from interpretation. A training-loss pattern, held-out loss, or mechanical rule finding is not a perceptual evaluation unless the relevant outputs were actually assessed.
- Distinguish a failed check from a check that did not run. NOT-CHECKED means the tool lacked what it needed to evaluate that condition; it is not evidence of a training fault.
- Use independent evaluation for the intended task. If the question is whether the model produces useful results, inspect outputs or use an evaluation that measures that outcome rather than inferring it from a linter verdict.
My repository correction puts the distinction this way: “Read every finding’s evidence string as the measurement, and its message as an interpretation that may exceed it.”
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsQuick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
