Cucumber Dry-Run Masks Missing Steps in CI

Cucumber-JVM's --dry-run flag has one documented job: verify that every Gherkin step maps to a step definition without executing the underlying automation. Teams wire it into CI as a fast pre-flight check — often as a gate on pull requests — and trust it to catch undefined steps before the full suite runs. The trust is misplaced. Under specific but common conditions, --dry-run exits 0 while leaving entire step definitions unresolved.

The failure mode isn't a bug in Cucumber-JVM 7 or Behave 1.2.6 or SpecFlow 3.9 — it's a consequence of how the dry-run contract is defined and how teams misread it. The flag validates syntax binding, not semantic completeness. Those are different things, and the gap between them is where missing coverage hides for months.

By the end of this article you'll know exactly why --dry-run produces false-green CI runs, how to reproduce the failure in Cucumber-JVM and Behave, and what pipeline configuration actually closes the gap across the three major BDD runtimes.

What --dry-run Actually Validates (and What It Doesn't)

Dry-run mode parses feature files and attempts to match each step's text against the registered step definition registry — then stops. It never instantiates your application context, never calls @Before/@After hooks, and never executes a single line of automation code. In Cucumber-JVM 7, the JUnit 5 platform runner marks a scenario as skipped (not failed) when a step is undefined during dry-run, and JUnit's exit code for an all-skipped run is 0. GitHub Actions, Jenkins, and Argo all interpret 0 as success. Your gate never fires.

The same dynamic plays out in Behave and SpecFlow, but the mechanics differ. Behave's --dry-run prints 0 features passed, 0 failed, N skipped and exits 0 regardless of undefined step count. SpecFlow's --dry-run equivalent (running with --filter on a tag that matches nothing) silently produces an empty result set. In all three runtimes, the dry-run path is architecturally separate from the undefined-step detection path — which only activates during a real execution pass. Understanding this separation is the prerequisite for fixing it; the flag was never designed to be a completeness gate, only a binding-syntax check.

Reproducing the Gap and Closing It in Cucumber-JVM, Behave, and SpecFlow

Start with a minimal reproduction in Cucumber-JVM 7. Add a new scenario to an existing feature file but deliberately omit the step definition:

# features/checkout.feature
Feature: Checkout

  Scenario: Guest checkout with promo code
    Given a guest user has items in their cart
    When they apply promo code "SAVE20"
    Then the order total reflects a 20% discount

The step they apply promo code "SAVE20" has no matching @When in the step definition registry. Run with dry-run:

# Maven
mvn test -Dcucumber.options="--dry-run --plugin pretty" \
         -Dtest=RunCucumberTest

# Exit code: 0
# Output:   Skipping step: they apply promo code "SAVE20"

The pipeline sees green. The fix is to run a real execution pass with --strict (Cucumber-JVM ≤ 6) or rely on the default strict mode in Cucumber-JVM 7, which treats undefined steps as failures — but only during execution, not dry-run. Add a dedicated "step audit" job to your pipeline that runs with zero parallelism and no retries, using a tag that selects every scenario:

# .github/workflows/bdd-gate.yml
jobs:
  step-audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run full suite in strict mode (no dry-run)
        run: |
          mvn test -Dcucumber.options="--tags @all --plugin json:target/cucumber.json" \
                   -Dtest=RunCucumberTest
        env:
          CUCUMBER_PUBLISH_QUIET: true

For Behave, the equivalent is behave --no-skipped --strict. The --strict flag causes Behave to exit non-zero when any step is undefined, pending, or skipped — which is the behavior most teams expect from dry-run but never get. Pair it with --dry-run removed entirely from the PR gate:

# behave strict audit
behave --strict --no-skipped --format json --outfile reports/behave.json features/

In SpecFlow 3.9 with NUnit or xUnit, undefined steps throw a PendingStepException at runtime and fail the test. The gap here is different: teams sometimes bind a step with a broad regex that matches the undefined text, masking the missing implementation. This is closely related to the precision problems in step regex patterns — an overly permissive .* catch-all silently absorbs new steps. The audit job for SpecFlow should include a regex coverage report generated by the SpecFlow+ runner, not just a green build. One team reduced their undefined-step escape rate from 11 per sprint to zero by replacing three catch-all regexes and running the full suite (not dry-run) as the PR gate — build time went from 22 minutes to 6 after parallelising with SpecFlow's thread-safe context injection.

Why Senior Engineers Still Ship Broken Dry-Run Gates

The most common mistake is conflating the step definition scope with the feature file scope. When step definitions live in multiple modules — a pattern that scales well but introduces load-order sensitivity — Cucumber-JVM's dry-run only loads the glue paths explicitly passed to the runner. A step defined in module-payments is invisible to the dry-run invoked from module-checkout's runner class. The step appears undefined during dry-run, gets skipped, and the gate exits clean. This is the same structural problem described in the context of step definition scope leaks across Cucumber modules — dry-run amplifies it because it never triggers the full classpath scan that a real execution does.

The second mistake is treating --dry-run as a substitute for a fast feedback loop rather than a syntax check. Teams add it to save two minutes on the PR gate, then never validate that it actually catches anything. A simple canary test — intentionally add an undefined step to a branch, push, and verify the gate fails — would expose the problem immediately. Most teams have never run that canary. The mental model that "dry-run = completeness check" goes unchallenged until a production regression traces back to a scenario that was green in CI for three months with no implementation behind it.

Three Myths About BDD Runtime Validation Worth Correcting

Myth 1: Dry-run is framework-agnostic and behaves consistently. It doesn't. Cucumber-JVM 7, Behave 1.2.6, and SpecFlow 3.9 each implement dry-run differently, with different exit-code semantics and different definitions of "undefined." Treating them as equivalent in a polyglot pipeline (common in platform teams supporting Java, Python, and .NET services) produces inconsistent gate behavior. Document the exact flag behavior for each runtime in your platform's test standards, not just "use dry-run for speed." Relatedly, AI-generated step definitions introduce a newer variant of this problem: generated steps that silently diverge from domain language over time will pass dry-run indefinitely because the binding exists — only the semantic intent has drifted.

Myth 2: A passing dry-run means the suite is maintainable. Step definition count is a poor proxy for suite health. A suite with 400 step definitions, 60 of which are bound only to deprecated scenarios, will pass dry-run cleanly while accumulating maintenance debt that compounds with every sprint. Dry-run validates binding existence, not binding relevance. Myth 3: Behave and Cucumber-JVM are interchangeable for teams choosing a BDD runtime. They aren't. Behave's Python-native context model, lack of a formal DI container, and single-process execution make it the right choice for data-science-adjacent teams or services already in Python. Cucumber-JVM with Spring or Guice injection is the right choice when you need parallel execution with isolated state and a typed domain model. Choose on architecture fit, not on which one your team learned first.

Replace --dry-run as your PR gate with a strict-mode execution pass scoped to a fast-running tag set — @smoke or @pr-gate — and reserve dry-run for local developer tooling only. The next thing worth measuring after you make this change is the undefined-step escape rate per sprint: track how many undefined steps reach main before and after. If it's above zero after two sprints, audit your glue path configuration.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.