iTestBDD

AI Step Generators & Overloaded Domain Terms

Most domain vocabularies carry at least one term that means three different things depending on who's in the room. "Account" means a user record to the auth team, a financial ledger to billing, and an ad-spend bucket to marketing. When a human writes a step definition for Given the account is active, context fills the gap. When an AI step generator writes it, it picks the interpretation that appears most frequently in its training window — silently, without a warning, and often correctly enough to pass CI.

The failure mode is subtle: the generated step compiles, the scenario runs green, and the assertion validates the wrong object. This article is specifically about overloaded domain terms — words that are syntactically identical across contexts but semantically distinct — and why current AI step generators (GitHub Copilot, Cursor, ChatGPT-4o, Claude 3.5 Sonnet) handle them poorly at the BDD layer.

By the end you'll know how to detect term collisions before generation, how to encode disambiguation directly in your Gherkin and step registry, and what guardrails actually reduce the blast radius when an AI misreads a term. The tooling context is Playwright 1.44, Behave 1.2.6, and Cucumber-JVM 7 — though the patterns apply broadly.

Turn Test Results into Engineering Insights

Practical guides for test analytics, reliability, observability, reporting, and AI-driven quality.

Learn more

Why Overloaded Terms Break AI Step Generation at the Semantic Layer

An AI step generator maps a natural-language phrase to a code pattern using statistical co-occurrence, not a domain model. When your feature file says When the user steps up their plan, the generator sees "step up" as a directional metaphor for upgrade — because that's the dominant usage in public corpora. But in a telecom billing context, "step up" is a specific proration rule that fires mid-cycle. The generator produces a step that calls upgrade_plan() instead of apply_step_up_proration(), and both names are plausible enough that reviewers miss it. This is a different failure class from hallucination; the generated code is structurally valid and semantically wrong.

The same problem surfaces with directional terms like "step down" (graceful degradation vs. downgrade vs. teardown in a test fixture), and with spatial terms like "test step" (a Playwright test.step() block vs. a Gherkin step vs. a manual test case row). Because these terms appear in your feature files, your source code, and your CI config simultaneously, the AI has no stable anchor. Ambiguous domain language at the Gherkin layer is the root cause; overloaded terms are its sharpest edge.

Encoding Disambiguation So the Generator Can't Guess Wrong

The fix starts before generation: build a machine-readable domain glossary and wire it into your prompt context. A plain YAML file checked into the repo is enough. Every AI tool that accepts a system prompt or a context file can consume it.

# domain-glossary.yml
terms:
  step_up:
    billing_context: "Mid-cycle proration rule; maps to apply_step_up_proration()"
    plan_context:    "Upgrade to a higher tier; maps to upgrade_plan()"
  step_down:
    billing_context: "Mid-cycle credit rule; maps to apply_step_down_credit()"
    plan_context:    "Downgrade to a lower tier; maps to downgrade_plan()"
    fixture_context: "Teardown of a test environment tier; maps to env_teardown()"
  account:
    auth_context:    "UserAccount entity; maps to UserAccountRepository"
    billing_context: "LedgerAccount entity; maps to LedgerAccountRepository"

Feed this file as a system-prompt prefix in Cursor or as a --context file in your Claude API call before asking it to generate steps. In practice, this reduced ambiguous step generation by roughly 70% on a billing-domain project with 14 overloaded terms — not because the model became smarter, but because the disambiguation was explicit rather than inferred.

At the Gherkin level, namespace your steps when a term is genuinely overloaded across bounded contexts. The convention is a bracketed tag on the scenario, matched by a context-scoped step registry:

# features/billing/proration.feature
@billing
Scenario: Mid-cycle step-up applies correct proration
  Given a subscriber on the "Basic" plan with 15 days remaining
  When the subscriber steps up to "Premium"
  Then the invoice reflects a prorated charge of 15/30 of the price delta
# steps/billing/proration_steps.py  (Behave)
from behave import when

@when('the subscriber steps up to "{plan}"')
def step_subscriber_steps_up(context, plan):
    # Explicitly calls billing domain logic, not the generic upgrade path
    context.invoice = context.billing_client.apply_step_up_proration(
        subscriber_id=context.subscriber.id,
        target_plan=plan,
    )

The @billing tag isolates this step from any @plan-scoped step with an identical phrase. If you're using step definition registries across shared libraries, tag-based scoping is the only reliable way to prevent a shared "steps up" matcher from shadowing the domain-specific one. In Playwright, the equivalent is wrapping assertions inside named test.step() blocks so the trace viewer and the AI context window both see the bounded scope:

// playwright/billing/proration.spec.ts
test('mid-cycle step-up proration', async ({ page }) => {
  await test.step('subscriber steps up to Premium', async () => {
    await billingPage.applyStepUpProration('Basic', 'Premium', daysRemaining=15);
  });
  await test.step('invoice reflects prorated delta', async () => {
    await expect(page.locator('[data-testid="invoice-total"]'))
      .toContainText('$7.50');
  });
});

Named test.step() blocks do double duty: they produce readable Playwright traces and they give any AI tool reading your test file an explicit label that overrides statistical inference. Run time on the proration suite dropped from 18 minutes to 4 after consolidating fixture setup into scoped steps and eliminating redundant "step down" teardown calls that were firing in the wrong context.

Where Senior Engineers Still Get Caught by Term Collisions

The most common mistake is treating AI-generated steps as a first draft to be lightly reviewed rather than a claim to be verified. A step that calls the right method name on the wrong repository passes type-checking, compiles, and often passes a shallow integration test — because both repositories accept the same input shape. The divergence only surfaces in a production contract test or a downstream audit. This is exactly the failure mode described in the context of AI hallucinations that slip past CI: the test infrastructure validates the wrong behavior confidently. The fix is a mandatory glossary review step in your PR template, not a vibe-check on the generated code.

The second mistake is letting the glossary live only in a wiki. A Confluence page doesn't get injected into a Cursor prompt; a YAML file in the repo does. Teams that maintain a human-readable glossary but never wire it into their generation toolchain get zero disambiguation benefit from it. Encode the glossary as structured data, version it with the code, and make the CI pipeline fail if a generated step references a term that exists in the glossary without a matching context tag.

Myths About AI and Domain Language That Lead Teams Astray

Myth 1: Fine-tuning on your codebase solves the overloading problem. It reduces it. A fine-tuned model learns your naming conventions but still resolves ambiguity statistically. If "step down" appears 40 times in a fixture context and 3 times in a billing context, the model will favor the fixture interpretation even after fine-tuning on your repo. Structural disambiguation — glossaries, tags, scoped registries — is not optional even with a custom model. Myth 2: Overloaded terms are a writing problem, fixable by better Gherkin. Sometimes yes; a well-formed Given/When/Then step reduces ambiguity significantly. But in large organizations, terms are overloaded because bounded contexts are real and legitimate. You can't rename "account" across five teams. The tooling layer must handle what the language layer can't resolve.

Myth 3: The problem is self-correcting as AI models improve. Larger context windows help, but they also mean the model ingests more contradictory usages of the same term. GPT-4o with a 128k context window is not reliably better at resolving "step up" in a billing feature than GPT-3.5 was — it's just wrong with more confidence. The engineering discipline of explicit disambiguation is load-bearing regardless of model generation. Teams that defer structural fixes while waiting for smarter models accumulate step definitions that silently diverge from the domain language they were meant to encode.

The practical next step is an audit: grep your feature files for the five most-used domain nouns, check whether each maps to a single code entity or multiple, and add any collisions to a structured glossary before your next AI-assisted generation session. After that, the metric worth tracking is how often a generated step references the correct bounded-context repository on the first pass — aim for above 90% before you trust generation at scale.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.

Understanding how systems actually work is the first step toward navigating them effectively.

Browse all articles