AI coding assistants — ChatGPT, Claude, Cursor — are now fluent enough in Gherkin and Behave/SpecFlow/Cucumber-JVM to generate step definitions that compile, bind correctly, and produce green CI runs. The failure mode nobody is talking about is subtler: the generated step looks right, exercises the right endpoint, and asserts the right HTTP 200 — but it asserts against a response shape the production contract no longer guarantees. Your pipeline is green. Your contract is broken.

This isn't a hallucination in the "AI made up a library" sense. It's a semantic drift problem: the model was trained on an older API schema, or it inferred a response field from a variable name, and it encoded that assumption into an assertion that nothing in your current test harness contradicts. Pact, OpenAPI validators, and consumer-driven contract tests exist precisely to catch this class of error — but only if they're wired into the same execution path as your BDD layer.

By the end of this article you'll know how to structure your step definitions so AI-generated code can't introduce unverified contract assumptions, how to gate on Pact verification in GitHub Actions, and where the seam between your BDD layer and your contract layer should actually live.

The Specific Failure Mode: Plausible Assertions on Stale Schemas

When an AI assistant generates a step definition for a scenario like Then the order response contains a valid shipment ETA, it has to decide what "valid shipment ETA" means in code. If your codebase has any prior art — a DTO, a fixture file, a logged response — the model will pattern-match against it. If that prior art is six months old and the field was renamed from eta_timestamp to estimated_delivery_iso in a provider migration, the generated assertion silently passes because it's asserting on a key that still exists in your test fixture, not against a live provider verification. The test is not lying; it's just not asking the right question.

This sits at the boundary between BDD's scenario layer and contract testing's provider-verification layer — a seam most teams leave unwired. BDD tools (Cucumber-JVM 7, Behave 1.2.x, SpecFlow 3.9) validate behavior orchestration. Pact (v4 spec) validates the message contract between a consumer and a provider at the schema level. An AI-generated step definition that embeds a field-level assertion is doing contract work inside a behavior test, without the versioning, broker, or verification feedback loop that makes contract testing safe.

Wiring the BDD–Contract Seam So AI Can't Introduce Drift

The fix is architectural: ban field-level assertions from step definitions entirely, and delegate schema validation to a contract layer that runs in the same CI job. Step definitions should assert on behavior outcomes (status codes, state transitions, domain events), not on payload shape. Payload shape is the contract layer's job.

Here's the pattern in Behave + Pact Python (pact-python 2.x):

# features/steps/order_steps.py
from pact import Consumer, Provider
from pact.matchers import Like, Term
import requests

# Step definition: behavioral assertion only
@then('the order response contains a valid shipment ETA')
def step_impl(context):
    # Assert behavior: response is successful and body is present
    assert context.response.status_code == 200
    assert context.response.json() is not None
    # Field-shape validation is delegated — see contract fixture below

The contract fixture carries the schema assertion, versioned and brokered:

# tests/contracts/test_order_eta_contract.py
from pact import Consumer, Provider
from pact.matchers import Like, Term

pact = Consumer('order-ui').has_pact_with(
    Provider('fulfillment-api'),
    pact_dir='./pacts',
    publish_verification_results=True,
)

def test_eta_field_contract():
    expected = {
        'estimated_delivery_iso': Term(
            r'\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z',
            '2024-11-01T14:00:00Z'
        ),
        'carrier_code': Like('UPS'),
    }
    (pact
     .given('an order with a confirmed shipment')
     .upon_receiving('a request for order ETA')
     .with_request('GET', '/orders/123/eta')
     .will_respond_with(200, body=expected))

    with pact:
        result = requests.get('http://localhost:1234/orders/123/eta')
        assert result.status_code == 200

Now wire both layers into GitHub Actions so they run sequentially — BDD first, contract verification second — and fail the PR if either breaks:

# .github/workflows/bdd-contract.yml
jobs:
  bdd:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - run: pip install behave pact-python requests
      - run: behave --no-capture --format progress2

  contract-verify:
    needs: bdd
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - run: pip install pact-python
      - run: |
          pact-verifier \
            --provider-base-url=https://staging.fulfillment-api.internal \
            --pact-broker-url=${{ secrets.PACT_BROKER_URL }} \
            --provider-version=${{ github.sha }} \
            --publish-verification-results

A team that adopted this split on a 40-scenario Behave suite reduced contract-related production incidents from 3 per quarter to 0 over two release cycles — not because the AI stopped generating bad assertions, but because those assertions were structurally prevented from reaching the contract layer. The BDD job stayed under 4 minutes; the Pact verification added 90 seconds. That's an acceptable gate.

Where Senior Engineers Still Get Burned by AI-Generated Steps

Trusting green CI as schema proof. A Playwright 1.44 or Cypress 13 test that hits a staging endpoint and asserts expect(response.json()).toHaveProperty('eta_timestamp') will pass as long as staging hasn't been migrated yet. Staging often lags production by a sprint. Engineers see green, merge, and find out at the provider deployment boundary. The fix is to run Pact broker verification against the provider's current main branch, not staging — use --provider-version-selectors '{"mainBranch": true}' in your verifier config.

Letting AI generate fixtures as ground truth. Cursor and Claude are both capable of generating a response_fixture.json that matches the step definition they just wrote — a perfectly self-consistent but externally unvalidated pair. If that fixture seeds your WireMock or Playwright route() intercept, you've built a closed loop that validates nothing real. Fixtures should be generated from a live provider response or from the Pact broker's published interactions, not from the model's inference. A Makefile target that refreshes fixtures from the broker before each local test run costs ten minutes to write and prevents this class of drift entirely.

Myths About AI-Generated Tests That Keep Biting Teams

"The AI will flag it if the assertion is wrong." It won't, reliably. ChatGPT and Claude generate code that is locally coherent — consistent with what they've seen in context. They have no runtime feedback loop. If you paste in a scenario and a DTO and ask for a step definition, the model will produce something that satisfies both inputs. It has no way to know the DTO is stale. Treating AI output as reviewed code (rather than as a first draft requiring contract-layer validation) is the mental-model error, not a limitation of the model itself.

"Contract tests are a separate team's problem." In a platform org with a dedicated contracts team, maybe. In every other org, the SDET or platform engineer who owns the BDD suite is the person closest to the consumer-provider boundary. Pact v4's consumer-driven model puts the contract definition on the consumer side — exactly where your Behave or SpecFlow scenarios live. The overhead of publishing a pact file from your existing test run is a pact.finalize() call and a broker push. That's not a separate team's problem; it's a one-afternoon integration that closes the loop AI-generated steps leave open.

The practical next step is an audit: grep your step definitions for any assertion that references a specific response field name, then ask whether that field is validated by a versioned Pact interaction or just by a fixture the AI wrote alongside the step. That audit will surface your exposure faster than any tooling change. Once you've wired the Pact broker into your CI gate, the next metric worth tracking is mean-time-to-detect contract drift — measured from provider deployment to failing verification run, not from production incident.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.