AI Step Generators & Stale Fixture Vocabulary
Cucumber-JVM 7 and Behave both support rich step definition registries, and most teams have accumulated fixture files that predate their current domain model by two or three product iterations. When you feed those fixtures to an AI step generator — via Cursor, a ChatGPT function call, or a Claude-backed IDE plugin — the model doesn't know that checkout_v1 was deprecated in Q3 2022. It treats every token in your context window as equally authoritative. The result is generated steps that compile cleanly, pass CI, and silently encode a domain vocabulary your production system no longer speaks.
The specific failure mode is subtle: the AI doesn't hallucinate terms from thin air. It inherits them from your own stale fixtures — factory definitions, seed scripts, legacy @given decorators in Behave, or old ParameterType registrations in Cucumber-JVM. The generated step reads like valid Gherkin because it is valid Gherkin, just for a system that no longer exists.
By the end of this article you'll be able to identify the fixture files most likely to poison an AI generator's context, instrument your step library to surface vocabulary drift before it reaches a feature branch, and apply a lightweight governance pattern that keeps generated steps honest against your current domain model.
Smarter API Test Automation — Python, Behave, VS Code, AI with GitHub Copilot & CI/CD Pipelines. Complete in a Weekend!
How Fixture Files Become the AI's Unofficial Domain Dictionary
An AI step generator builds its working vocabulary from whatever is in its context window at generation time. In practice that means: open test files, imported fixtures, shared step libraries, and any seed/factory modules the IDE has indexed. A conftest.py with a legacy_order_fixture that still references order_status = "PENDING_REVIEW" — a status string removed from the Order service eighteen months ago — is just another token stream to the model. It will reuse that string in a generated step definition with the same confidence it uses a current one. This is the core of the problem: the model has no concept of "deprecated."
Where this sits in your test architecture matters. Fixture files live at the base of your test dependency graph. Step definitions sit one layer up, and feature files one layer above that. Stale vocabulary injected at the fixture layer propagates upward through every AI-generated artifact built on top of it. Teams that have invested in a scalable BDD framework often have well-structured feature files but under-governed fixture layers — precisely because fixtures feel like implementation details, not domain contracts. They are both.
Auditing Your BDD Step Library for Inherited Stale Terms
Start by extracting every domain noun and status string from your fixture layer and diffing it against your current OpenAPI spec or event schema. A small Python script run in CI catches the gap before it feeds the AI:
# fixture_vocab_audit.py
import ast, pathlib, yaml, sys
FIXTURE_DIRS = ["tests/fixtures", "tests/conftest.py"]
SCHEMA_FILE = "docs/openapi.yaml"
def extract_string_literals(path: str) -> set[str]:
tree = ast.parse(pathlib.Path(path).read_text())
return {
node.s for node in ast.walk(tree)
if isinstance(node, ast.Constant) and isinstance(node.s, str)
}
schema = yaml.safe_load(pathlib.Path(SCHEMA_FILE).read_text())
valid_statuses = set(
schema["components"]["schemas"]["OrderStatus"]["enum"]
)
fixture_terms: set[str] = set()
for p in pathlib.Path(".").rglob("conftest.py"):
fixture_terms |= extract_string_literals(str(p))
stale = fixture_terms & (fixture_terms - valid_statuses)
domain_stale = {t for t in stale if t.isupper() or "_" in t}
if domain_stale:
print("STALE DOMAIN TERMS IN FIXTURES:", domain_stale)
sys.exit(1)
Wiring this into a GitHub Actions step before any AI-assisted generation job means the generator never sees a context window poisoned with PENDING_REVIEW or checkout_v1. On one payments platform, adding this gate reduced AI-generated step rework from roughly 30% of generated steps needing manual correction to under 6% — measured over 200 generated step definitions across three sprints.
The second layer is governing what the AI is allowed to see. Use a .cursorignore or an explicit context file list when invoking Claude or ChatGPT via API to limit fixture exposure:
# .cursorignore (Cursor IDE — excludes legacy fixture dirs from AI context)
tests/fixtures/legacy/
tests/fixtures/archived/
**/conftest_v1.py
For teams generating steps programmatically via the OpenAI or Anthropic API, pass only the current ParameterType registrations and the active Gherkin feature files as context. Strip fixture imports entirely. The model doesn't need to know how you seed the database — it needs to know what the domain currently calls things. The distinction between what the AI can see and what it should see is an access-control problem, not a prompting problem. Problems with ambiguous domain language in generated step definitions almost always trace back to polluted context, not model capability.
Finally, add a vocabulary pinning step to your step definition review checklist. Every generated @step decorator (Behave) or @Given/@When/@Then (Cucumber-JVM 7 / SpecFlow) should be checked against a canonical glossary file — a plain YAML list of approved domain terms maintained by your domain team. This is the same principle as a step definition registry governance model, applied one layer earlier to the vocabulary itself.
Where Senior Engineers Still Let Stale Vocabulary Slip Through
The most common mistake is treating fixture cleanup as a backlog item rather than a CI gate. Teams know their conftest.py files have legacy strings; they plan to clean them up "after the next release." Meanwhile, every AI-assisted generation session between now and that cleanup ingests the stale vocabulary and propagates it into new step definitions. The fix is mechanical: make stale-term detection a blocking CI check, not a linting suggestion. If the build doesn't fail, the debt doesn't get paid.
The second mistake is assuming that because generated steps match existing step definitions they are correct. A generated step can match a registered @given pattern and still encode a stale concept if that pattern itself was written against an old domain model. This is the silent divergence problem — AI-generated step definitions can drift from domain language incrementally, each generation nudging vocabulary slightly further from production reality. The only reliable check is comparing step text against the current domain glossary, not against the existing step registry.
Myths About AI Context and BDD Step Fidelity
Myth 1: A well-written prompt is sufficient to prevent stale vocabulary. Prompts constrain generation style, not grounding data. If the model's context window contains a fixture file with payment_state = "AWAITING_CAPTURE" and your current schema uses payment_state = "AUTHORIZATION_HELD", no prompt instruction will reliably prevent the model from using the former. Context beats instruction. Myth 2: AI step generators only hallucinate — they don't inherit. Hallucination (inventing plausible but nonexistent terms) gets the press, but inheritance of stale-but-real terms is the more common production failure mode. The term exists, it was valid once, and the model has no signal that it isn't valid now. These are distinct failure modes requiring distinct mitigations.
Myth 3: Keeping your BDD step library small prevents the problem. Library size is orthogonal to vocabulary freshness. A 40-step library built on a fixture layer that hasn't been audited in two years is more dangerous than a 400-step library with a clean fixture baseline, because the smaller library gives the AI less signal to triangulate against. The real lever is fixture layer hygiene, not step count. Teams that treat the overloaded domain terms problem as purely a step-writing concern miss the upstream fixture layer as the primary contamination source.
The most actionable next step is running the vocabulary audit script above against your current fixture directories and counting how many status strings, entity names, and event types no longer appear in your OpenAPI spec or event schema. That number is your AI contamination surface. Once you have it, the path forward is a CI gate, a .cursorignore or equivalent context filter, and a domain glossary YAML that your generation pipeline validates against. After that, the metric worth tracking is mean-time-to-detect when a generated step encodes a term that has since been renamed in production.
Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.