iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes: if training weights are implemented by repeating records, a later deduplication pass can erase those weights. In an account published August 23, 2026, developer Seth Wheeler found that his pipeline removed repeated training examples—and also merged some distinct examples because it compared only their final 200 characters. His reported counts make the failure concrete, but they do not establish that it caused his model’s performance problems.
How the pipeline was supposed to shape the curriculum
Wheeler describes building a capture-the-flag arena where language models attack and defend containers. To improve a local Qwen2.5-3B-Instruct bot’s weaker defensive play, he extracted turns from game replays and trained with an MLX LoRA adapter described as rank 8 across 8 layers. He tried two curriculum approaches; the postmortem focuses on how the extraction pipeline represented their weights.
In his DEV Community post, Wheeler says training/defense_training_data.py on the custom-bot branch assigned different weights according to outcomes and vulnerability checks. It implemented those weights by appending the same example multiple times to the JSONL list:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- Successful defensive turns received weight 6.
- Defensive turns that patched a vulnerability but still lost a flag received weight 3.
- Neutral defensive activity received weight 1, while turns marked for dropping received weight 0.
- Attacker turns received weight 3 if a command captured a flag, otherwise weight 1.
Repeating a record can make it appear more often during training, so the representation depends on every later pipeline stage preserving those copies.
#1 Best Overall
How deduplication removed the repetitions
The same function later deduplicated examples using line[-200:] as its key: after seeing a key once, it skipped subsequent lines with that key. Because identical role, reasoning text, and command inputs produced identical example text, repeated copies had identical keys. The pass therefore retained one copy and discarded the extra rows that had encoded higher weights.
As Wheeler put it, the two stages were “the same operation with opposite signs”: one duplicated examples to increase their influence, then the next removed duplicates. A second problem came from the key itself. Two examples could have different reasoning text but the same final 200 characters, such as a shared ending command. Those distinct full lines collided and one was removed too.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
What Wheeler’s reported counts show
Wheeler reports that a run across 358 session directories (329 containing a complete game.json) produced 94,022 examples after weighting and before deduplication. Those comprised 55,034 distinct full lines. The suffix-key deduplication left 50,145 lines.
| Pipeline measure | Wheeler’s reported result |
|---|---|
| Weighted examples before deduplication | 94,022 |
| Distinct full lines before deduplication | 55,034 |
| Lines after suffix-key deduplication | 50,145 |
| Rows attributable to weighting repetitions | 38,988 (41.5% of the weighted corpus) |
| Additional distinct full lines removed by 200-character suffix collisions | 4,889 (8.9% of the distinct full lines) |
The intended weighted corpus contained these categories, according to Wheeler:
Rank #3
| Example category | Turns | Weight | Weighted lines |
|---|---|---|---|
| Successful defensive turns | 2,861 | 6 | 17,166 |
| Patched defensive turns that still lost a flag | 555 | 3 | 1,665 |
| Neutral defensive turns | 18,227 | 1 | 18,227 |
| Attacker commands that captured flags | 2,518 | 3 | 7,554 |
| Other attacker turns | 33,035 | 1 | 33,035 |
| Curated Gemini defensive exemplars | 3,275 | 5 | 16,375 |
He says the “gold” defensive turns were meant to account for 18.3% of weighted data but made up at most 5.7% of the post-dedup training set. The curated Gemini file had 3,275 lines but only 3,187 distinct 200-character tails. Increasing its configured repetition weight from 2 to 5 therefore did not increase the number of examples left after deduplication.
What reached the trainer—and what remains uncertain
Wheeler reports that the downstream split contained 38,448 training lines and 4,272 validation lines, 42,720 total. The training file’s length matched a 90/10 split, and every training line was unique both as a full line and by its 200-character tail. In the repository he describes, session logs, JSONL corpora, and adapters were gitignored; the extraction and retraining code was available, but the generated data files were not. The reported counts therefore come from his own analysis of his code and logs, not an independently reproduced run.
Rank #4
The figures do not prove that the bug caused the retrains to be rolled back. Wheeler says the needed controlled experiment—using the same data and evaluation while preserving the weights—had not been run, and correct weighting might still have regressed. He also notes that his extraction included more sessions than were available for the historical v7 training run, so the absolute counts are not directly comparable.
Free tools Windows power users keep installed
One-click scans. No signup required.
His reported self-play results point to a separate caution about evaluation. Across 11 generations, validation loss declined from 0.526 at generation 0 to 0.462 at generation 9, with a reported best of 0.377. Over that span, average score fell from 172.2 to 101.4 and flags lost per game rose from 2.7 to 4.7. Wheeler warns that matches among six copies of the same bot are close to zero-sum, not a general capability measure. The narrow lesson is that the validation metric improved while the behavior he wanted did not; those observations do not establish why.
Best Value
How to keep a later transform from silently undoing weights
The practical issue is not that duplication or deduplication is inherently wrong. It is that they encode different assumptions: duplication says repeated rows matter; deduplication says rows with the same key are interchangeable. If both appear in one pipeline, verify that the second operation does not contradict the first.
- Measure the corpus at each boundary. Record row counts and category or weight distributions before weighting, after weighting, after deduplication, and in the final train/validation split. A single upstream counter cannot show what the trainer actually receives.
- Inspect the exact uniqueness key. Check whether it compares the full example or only a suffix. Test examples with different reasoning but a shared ending command, and count full-line duplicates separately from key collisions.
- Choose an explicit representation for weights. If later stages need deduplication, store a weight or multiplicity as data and preserve or aggregate it deliberately instead of relying on duplicate rows to carry meaning. If examples are intentionally collapsed, define how their weights combine.
- Validate the final artifact. Before training, compare the intended distribution with the rows and effective weights in the exact corpus passed to the trainer. Then hold data, weighting, and evaluation constant when testing whether a curriculum changes behavior.
Wheeler summarized the missed check this way: “The number worth printing is the one I never printed, the corpus length before and after dedup.” His account is a useful warning about pipeline observability: inspect not only how examples are scored upstream, but also what survives every transform.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Recommended Free Tools
