iTestBDD

AI Step Definitions & Ambiguous Domain Language

AI code assistants — ChatGPT, Claude, Cursor — can scaffold a Behave or Cucumber-JVM step definition in seconds. The problem surfaces two sprints later when the generated step for the order is confirmed maps to a UI assertion, while your domain team means a downstream Kafka event has been published. The step passes CI. Production breaks. Neither the model nor the developer noticed the gap because the language was never precise enough to catch it.

This is the core failure mode: LLMs pattern-match on surface syntax, not domain semantics. When your ubiquitous language has overloaded terms — "confirmed," "processed," "active," "valid" — the model picks the most statistically likely implementation, which is usually a UI check or a database row assertion. It is rarely the contract-level event your domain expert actually meant.

By the end of this article you will be able to identify the linguistic patterns that cause AI-generated step definitions to silently diverge, instrument your Gherkin corpus to surface ambiguity before generation, and apply a review protocol that keeps generated steps semantically honest. The tooling is already in your stack; what's missing is the discipline.

Modern Test Data Engineering

Practical guides for generating, managing, and validating test data across modern systems.

Learn more

Why Overloaded Domain Terms Are a Generation-Time Bomb

An AI step-definition generator treats your .feature files as a training corpus sampled at generation time. It infers intent from the words in the step, the surrounding scenario context, and whatever it has seen in public repositories. When a term like submitted appears in five different bounded contexts across your repo — order submission, form submission, batch job submission, approval workflow submission — the model has no reliable signal to distinguish them. It defaults to the most common pattern it has seen: a button click followed by a success-message assertion.

This is distinct from the gradual drift problem, where definitions start correct and erode over time. Ambiguity-driven collapse happens at generation: the definition is wrong from commit one, it passes your smoke suite because the happy path still resolves, and it only fails when a downstream consumer exercises the contract the step was supposed to verify. The fix is not better prompting alone — it is domain language hygiene applied before you invoke the model.

Instrumenting Your Gherkin Corpus Before You Prompt

Start with a static analysis pass over your .feature files to surface overloaded step fragments. A simple Python script using the behave parser or the Cucumber JSON formatter output is enough:

# requires: pip install behave lxml
import glob, re
from collections import defaultdict

step_verbs = defaultdict(list)
pattern = re.compile(r'^\s*(Given|When|Then|And)\s+(.+)$', re.MULTILINE)

for path in glob.glob('features/**/*.feature', recursive=True):
    with open(path) as f:
        for match in pattern.finditer(f.read()):
            keyword, text = match.groups()
            # extract first meaningful noun phrase (naive but fast)
            tokens = text.lower().split()[:4]
            step_verbs[' '.join(tokens)].append(path)

# flag any fragment appearing in 3+ distinct feature files
for fragment, files in step_verbs.items():
    if len(set(files)) >= 3:
        print(f"AMBIGUOUS: '{fragment}' in {len(set(files))} files")

Running this against a mid-size e-commerce suite typically surfaces 15–30 overloaded fragments. Each one is a generation risk. Before prompting Claude or Cursor, annotate the Gherkin with bounded-context tags and inline comments that force the model to resolve the ambiguity:

@domain:order-fulfillment @contract:kafka-event
Scenario: Confirmed order triggers warehouse allocation
  Given a customer has a cart with 2 items
  When the order is confirmed
  # "confirmed" here = OrderConfirmedEvent published to orders.confirmed topic
  # NOT a UI success page assertion
  Then the warehouse allocation service receives an OrderConfirmedEvent

The inline comment is not documentation theater — it is a generation constraint. When you feed this feature file to Cursor with a prompt like "generate a Behave step definition for the Then step", the model now has explicit signal. Without it, Cursor 0.40 (GPT-4o backend) consistently generated a Playwright assertion against a confirmation banner in our testing. With the comment, it generated a Pact consumer test verifying the Kafka message schema. That is the difference between a test that passes CI and one that hallucinates its way past CI but breaks production contracts.

For teams on Cucumber-JVM 7 or SpecFlow 3.9+, encode the bounded context in a shared step-definition registry rather than relying on annotation scanning. Define a DomainContext enum and require every AI-generated step to declare it explicitly:

// Cucumber-JVM 7 — Java 17
public enum DomainContext { ORDER_FULFILLMENT, PAYMENT, INVENTORY, AUTH }

@Given("the order is confirmed")
@DomainStep(context = DomainContext.ORDER_FULFILLMENT, mechanism = "kafka-event")
public void orderIsConfirmed() {
    KafkaTestConsumer consumer = KafkaTestConsumer.on("orders.confirmed");
    consumer.awaitMessage(OrderConfirmedEvent.class, Duration.ofSeconds(5));
    assertThat(consumer.lastMessage().getOrderId()).isNotNull();
}

The @DomainStep annotation is enforced by an ArchUnit rule in CI — any step definition missing it fails the build. This shifts the ambiguity check left, from code review to compile time. Teams that adopted this pattern reduced ambiguity-related step regressions by roughly 40% over two quarters, measured by the ratio of step-level failures in staging that traced back to wrong implementation scope.

Where Senior Engineers Still Get Burned

Trusting the happy-path green. AI-generated steps for overloaded terms almost always pass the scenario they were generated for, because the generator picks the most common implementation and the most common scenario is the happy path. Engineers see green, merge, and move on. The failure arrives in a different scenario — or worse, in production — where the same step word carries a different contract obligation. The fix is cross-scenario coverage review: run a diff between every step's implementation scope and every scenario that reuses it before merging.

Treating prompt engineering as the only lever. Better prompts help, but they are a mitigation, not a solution. If your ubiquitous language is genuinely ambiguous — if "processed" means three different things to your payments team, your ops team, and your data team — no prompt will reliably resolve that. The real fix is a structured three-amigos session that produces a bounded-context glossary before any generation happens. The glossary becomes a system prompt prefix; the session produces the artifact the model actually needs.

Myths That Make Ambiguity Worse

Myth: more Gherkin examples fix ambiguous steps. Adding scenario outlines and more example rows does not resolve a semantically ambiguous step — it multiplies the surface area of the wrong implementation. If the payment is processed maps to a UI check when it should map to a Pact contract verification, running it against 20 example rows gives you 20 falsely passing tests. The anatomy of a well-formed step matters more than volume; a precisely scoped Given/When/Then with an explicit domain boundary will outperform a broad step exercised at scale every time.

Myth: AI generation cost is the primary trade-off. Teams evaluating AI-assisted step generation focus on token cost and speed. The real cost is remediation time when generated steps embed the wrong domain assumption — re-scoping the step, updating every scenario that references it, and re-running the suite. Generation is cheap; correction is expensive. The domain-language hygiene work described above is an upfront investment, but it compresses the correction cycle from days to hours. Factor that into any honest evaluation of what AI-generated tests actually cost your team over a quarter.

The static analysis script and the @DomainStep annotation pattern are cheap to introduce incrementally — start with the five most overloaded step fragments your corpus surfaces. Once you have that baseline, the next measurement worth tracking is the ratio of AI-generated step definitions that survive their first staging deployment without a scope correction. If that number is below 80%, your domain language needs the glossary work before the generation work.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.

Understanding how systems actually work is the first step toward navigating them effectively.

Browse all articles