Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Fixture leakage happens when expected test data gets copied into an AI coding prompt, giving a model the answer to reproduce instead of just the rule to implement. This small Python exercise searches prompt text for some expected values from pytest-style assertions and redacts matches before you send the prompt. It is a local teaching gate—not proof that a prompt is safe.
What fixture leakage looks like
Suppose a test expects a function to produce the name “Mira.” If you include that assertion in a prompt asking an AI to implement the function, the expected name is already visible. A generated draft might return “Mira” because it copied the fixture rather than because it correctly handles the underlying rule. Changing the test input can expose the mistake: the function still returns the old name.
As Alex Chen, author of the DEV Community tutorial, puts it: “Fixture leakage is not a vibe-coding slogan. It is a dataflow bug.” The useful question is whether expected values flow from test source into the prompt.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Build the small scrubber
The script below demonstrates the core idea: find expected fragments in pytest-style equality assertions, then replace matching prompt substrings. It uses Python’s standard library and is intended to run locally; the model call is optional.
#1 Best Overall
import re
import sys
from pathlib import Path
REDACTION = "[[REDACTED_EXPECT]]"
MIN_FRAGMENT_LENGTH = 2
def expected_fragments(test_source: str) -> list[str]:
"""Extract simple expected values from pytest-style equality asserts."""
fragments = []
pattern = re.compile(r"^s*asserts+.+?s*==s*(.+?)s*(?:#.*)?$")
for line in test_source.splitlines():
match = pattern.match(line)
if not match:
continue
value = match.group(1).strip()
# This teaching example handles quoted string literals only.
if len(value) >= 2 and value[0] == value[-1] and value[0] in ""'":
fragment = value[1:-1]
if len(fragment) >= MIN_FRAGMENT_LENGTH:
fragments.append(fragment)
return sorted(set(fragments), key=len, reverse=True)
def scrub(prompt: str, fragments: list[str]) -> tuple[str, list[str]]:
"""Replace expected fragments found in prompt text."""
leaked = [fragment for fragment in fragments if fragment in prompt]
for fragment in fragments:
prompt = prompt.replace(fragment, REDACTION)
return prompt, leaked
def main() -> None:
if len(sys.argv) != 3:
raise SystemExit("Usage: python tiny_prompt_scrubber.py tests.py prompt.txt")
test_source = Path(sys.argv[1]).read_text(encoding="utf-8")
prompt = Path(sys.argv[2]).read_text(encoding="utf-8")
fragments = expected_fragments(test_source)
scrubbed, leaked = scrub(prompt, fragments)
print(f"Expected fragments found: {len(fragments)}")
print(f"Fragments present in prompt: {len(leaked)}")
print("Scrubbed prompt:n")
print(scrubbed)
if __name__ == "__main__":
main()Save the code as tiny_prompt_scrubber.py. Put the test assertions in a Python file such as tests.py, and put the draft request you intend to send in prompt.txt. Run:
python tiny_prompt_scrubber.py tests.py prompt.txtThe script prints how many simple quoted expected fragments it extracted, how many occur in the prompt, and a prompt with matches replaced by [[REDACTED_EXPECT]]. Inspect the scrubbed text before using it: the marker tells you where a match was found, not whether the remaining task is sufficient or correct.
Predict what will leak
For an assertion such as assert format_person(person) == "Mira Chen", the example extracts Mira Chen. If those characters occur in the prompt, the scrubber replaces that substring. Sorting fragments longest first helps avoid replacing a shorter fragment before a longer overlapping one.
Try asking yourself: “Predict which strings will leak.” Then consider: “What happens if an expected value is the letter a?” Replacing a one-character value everywhere could damage ordinary words throughout the prompt. The two-character minimum is a readability trade-off, not a safety rule: a short expected value can remain exposed. The sample also focuses on quoted string literals; it does not evaluate arbitrary Python expressions.
What the example demonstrates—and what it does not
Chen reports that the original prompt exposed two fixture strings and the scrubbed prompt exposed zero of those two. The same account describes a later generated draft that failed on a second input because it assumed a comma rather than a dash; delimiter handling was then fixed. These are observations from one demonstration, not a benchmark or evidence of general effectiveness. No token benchmark or timing result was reported.
A zero-match result means only that this particular script did not find its extracted fragments in the prompt. It does not establish that expected answers are absent, that the prompt is safe, or that a model will implement the intended rule. A match can also occur in harmless prose and be redacted unnecessarily.
Where the regex misses test data
The pattern is deliberately narrow. It does not reliably cover common ways Python tests express expected values, including:
pytest.mark.parametrizecases, where values are supplied separately from assertion lines.- Doctests or expected output written in examples rather than a matching assertion.
- Helper variables, such as
expected = "Mira Chen", followed byassert result == expected. - Attribute comparisons such as
assert result.name == person, where the expected value is another variable.
Those forms can produce false negatives: expected data may enter a prompt without being extracted. For that reason, do not use this teaching script as a production redaction filter or to handle secrets or sensitive student data.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Two more robust directions
Parse Python source into an AST
Python’s official ast documentation describes ast.parse as parsing source into an abstract syntax tree. Traversing that structure can make it easier to recognize assertion expressions and literal values than matching lines with a regular expression. Extraction still needs deliberate logic for parametrization, helper variables, and other test patterns. The documentation also cautions that successful parsing does not guarantee source can be compiled or executed; using an AST is not itself a leakage-prevention guarantee.
Keep expected values out of the prompt
A stronger dataflow design is to store expected values in a separate JSON fixture and avoid including that fixture in the prompt at all. This reduces reliance on detecting and deleting overlaps after they have entered prompt text. You still need to check what files, examples, and context are actually sent to the model.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
