AI-assisted test generation has a slow-burn failure mode that doesn't show up in red CI builds: the step definitions it writes gradually stop matching the language your domain experts actually use. The Gherkin still parses. The regex still binds. The scenario still passes. But six months later, a product manager reads a scenario and says "that's not how our system works anymore" — and they're right. The tests have been lying, politely, for months.

This is distinct from the acute problem of hallucinated step definitions that slip past CI — those fail loudly, eventually. Silent domain drift is subtler: the implementation is technically correct against the code, but semantically wrong against the business model. The step When the user submits a claim now calls an endpoint that processes a pre-authorization, because the domain evolved and the AI regenerated the step against the new code without anyone updating the vocabulary.

By the end of this article you'll have a concrete strategy — including a lexical diff pipeline and a CI gate — for detecting when AI-generated step language has diverged from your canonical domain glossary, and for enforcing alignment before it erodes team trust in the test suite.

Why AI Tools Produce Semantically Stale Step Definitions

When you ask ChatGPT, Claude, or Cursor to generate step definitions, the model grounds its language in whatever context you provide: the current source code, the OpenAPI spec, the existing test file. That context is a snapshot. The model has no memory of what the domain language used to mean, no awareness that order was renamed fulfillment_request in the last sprint, and no mechanism to flag when its output contradicts a glossary that lives in Confluence. The result is step text that is syntactically fluent but semantically anchored to the wrong moment in time. If you're using an AI test assistant with memory, you can partially mitigate this — but most teams aren't.

The problem compounds because BDD's value proposition depends on the scenario layer being readable by non-engineers. Once step language drifts, the living-documentation contract breaks: the Gherkin no longer describes what the business intended, and the step implementation no longer reflects what the domain model enforces. At that point you have integration tests dressed up as BDD — useful, but not the artifact you promised stakeholders. Understanding what makes a Given/When/Then step precise is the prerequisite for knowing when AI output violates that precision.

Building a Lexical Drift Detection Pipeline

The core idea is simple: maintain a canonical domain glossary as a versioned artifact, then diff every AI-generated step definition against it in CI. The glossary is a plain YAML file owned by the product team and engineers together — not a wiki page, because wikis don't have diffs in your pipeline.

# domain_glossary.yaml  (v1.4.2, committed to repo root)
terms:
  - canonical: "submits a claim"
    deprecated: ["files a claim", "raises a claim"]
    introduced: "2024-03-01"
  - canonical: "pre-authorization request"
    deprecated: ["pre-auth", "authorization request"]
    introduced: "2024-09-15"
  - canonical: "fulfillment_request"
    deprecated: ["order", "purchase_order"]
    introduced: "2025-01-10"

A Python script (runs in ~200ms on a suite of 400 feature files) scans every .feature file for deprecated terms and emits a structured report. Wire it as a pre-commit hook and a GitHub Actions step — the hook catches it locally, the CI gate enforces it on PRs.

# check_domain_drift.py
import yaml, re, sys
from pathlib import Path

glossary = yaml.safe_load(Path("domain_glossary.yaml").read_text())
deprecated_map = {
    dep: entry["canonical"]
    for entry in glossary["terms"]
    for dep in entry.get("deprecated", [])
}

violations = []
for feature_file in Path("features").rglob("*.feature"):
    text = feature_file.read_text()
    for dep_term, canonical in deprecated_map.items():
        if re.search(re.escape(dep_term), text, re.IGNORECASE):
            violations.append(
                f"{feature_file}: '{dep_term}' → use '{canonical}'"
            )

if violations:
    print("\n".join(violations))
    sys.exit(1)

# .github/workflows/bdd-quality.yml (relevant job step)
- name: Check domain language drift
  run: python check_domain_drift.py
  # Fails PR if any .feature file uses a deprecated term.
  # On a 400-scenario suite, this added 4 seconds to the pipeline.

For the step-definition layer (where the AI generates Python or TypeScript glue code), extend the check to scan string literals inside step decorators. In a Behave codebase, that means scanning @given, @when, and @then decorator arguments. In Cucumber-JVM 7, scan the annotation strings. The same deprecated-term regex applies. On one fintech project, adding this gate caught 23 AI-generated step definitions using the old order vocabulary within the first two weeks of rollout — none had caused a test failure, but all were wrong.

# Behave step file — AI-generated, caught by the gate
@when('the user submits an order')          # ❌ 'order' is deprecated
def step_submit_order(context):
    context.api.post("/fulfillment_requests", context.payload)

# Corrected after gate failure
@when('the user submits a fulfillment_request')  # ✅ canonical term
def step_submit_fulfillment(context):
    context.api.post("/fulfillment_requests", context.payload)

The glossary itself needs a governance ritual — not a heavyweight process, but a lightweight one: a 15-minute sync between the lead SDET and a domain expert at the start of each sprint to ratify any new terms and deprecate old ones. Version-bump the YAML, commit it, done. The real cost of AI-generated tests isn't the token spend — it's the maintenance overhead when drift goes undetected. A versioned glossary and a 4-second CI check are cheap insurance against that.

Where Senior Engineers Still Get Burned by This

The most common mistake is treating the AI's output as a first-class domain artifact rather than a draft. Engineers who wouldn't merge a PR without review will accept AI-generated Gherkin without a second look because it reads fluently. Fluency is not correctness. The model is optimizing for coherent English, not for fidelity to your bounded context. The fix is a mandatory review checkpoint — not a full re-read of every line, but a targeted diff of step text against the glossary before the PR merges.

A subtler trap: teams that run Cucumber-JVM 7 or SpecFlow 3 with auto-generated step stubs will see the framework silently create new step definitions for deprecated phrasings because the regex doesn't match anything existing. The test suite grows, coverage metrics look healthy, and nobody notices that two step definitions now cover semantically identical actions with different vocabulary. This is especially painful in SpecFlow projects where the step binding scan happens at compile time — you get no runtime warning, just quiet duplication. Auditing step definition files for synonym clusters (terms that map to the same underlying action) is worth adding to your quarterly test-debt review.

Myths That Let Domain Drift Persist Undetected

"If the tests pass, the language is fine." This conflates behavioral correctness with semantic accuracy. A passing test proves the implementation matches the step definition; it says nothing about whether the step definition matches the domain intent. BDD's contract is with the business, not just with the code. Similarly, "the AI will stay consistent if we give it the same prompt." Models are non-deterministic by default, and even with temperature set to zero, a context window that includes updated source code will produce different vocabulary than one that doesn't. Prompt consistency is not semantic consistency.

"Living documentation tools like Serenity or Allure will catch this." They report what the tests say, not whether what the tests say is still true. A Serenity report generated from semantically stale scenarios looks identical to one generated from accurate ones — both are green, both have narrative prose, both satisfy a stakeholder audit at a glance. The value of LLM-generated test reports is in surfacing intent, but that value collapses if the underlying step language has drifted. The report is only as trustworthy as the glossary alignment beneath it.

A versioned domain glossary with a CI-enforced lexical diff is a low-overhead, high-signal control. Implement the Python gate, run the sprint-cadence glossary review, and you'll catch drift in hours rather than quarters. The next thing worth measuring after you ship this is how often the glossary itself changes — high churn there is a signal that your domain model is unstable, which is a conversation worth having with product before it becomes a test-debt crisis.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.