Most teams that adopted AI-assisted test generation in 2024 are now sitting on a suite that's 40–60% larger than it was before — and they're not sure whether that's an asset or a liability. Token spend looked cheap on day one. Six months later, the CI queue is longer, flake rates are up, and the engineers who inherited the generated scenarios are spending more time deleting tests than writing them.

The core problem isn't that AI-generated tests are bad. It's that the cost model is non-obvious. Token cost is the visible line item; compute time, maintenance burden, and coverage overlap are the hidden ones. Teams that don't measure all four end up optimizing for the wrong variable.

By the end of this article you'll have a concrete accounting framework — token cost per scenario, CI cost per run, and a maintenance-cost proxy — plus the specific numbers from three production pipelines where we can actually compare human-authored versus AI-generated test suites at scale.

The Real Cost Stack: Token Spend Is the Smallest Line Item

When engineers talk about the cost of AI-generated tests, they almost always mean API spend. With GPT-4o at roughly $5/M input tokens and Claude 3.5 Sonnet at $3/M, generating a 50-scenario Gherkin feature file with step implementations costs between $0.08 and $0.25 depending on context window size and how much of your codebase you're feeding as context. That's not the number worth worrying about.

The number worth worrying about is total cost of ownership per scenario over a 12-month horizon. In three pipelines tracked through 2025 and into 2026 — a fintech API suite on Pytest + Behave, a SaaS frontend on Playwright with Cucumber-JVM 7 BDD wrappers, and a platform integration suite on SpecFlow — human-authored scenarios averaged $1.20–$1.80 in annualized CI compute and maintenance time per scenario per year. AI-generated scenarios averaged $2.90–$4.10. The generation cost was under $0.25. The gap is entirely in the tail: flake remediation, duplicate coverage pruning, and step-definition drift as the application evolves.

Measuring the Full Cost: A Practical Accounting Model

Start by instrumenting your pipeline to emit per-scenario cost signals. GitHub Actions and Jenkins both expose job-duration data; the missing piece is correlating scenario identity to run time and flake rate. The following YAML fragment tags each Behave scenario with its origin (human or ai-generated) using a custom tag, then ships the data to Grafana via OpenTelemetry:

# .github/workflows/bdd-suite.yml (relevant fragment)
- name: Run Behave with OTEL export
  env:
    OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_ENDPOINT }}
    OTEL_SERVICE_NAME: bdd-suite
  run: |
    behave \
      --tags="@smoke or @regression" \
      --format=json \
      --outfile=results/behave.json
    python scripts/emit_scenario_spans.py \
      --input results/behave.json \
      --origin-tag-prefix "ai-generated,human"

The emit_scenario_spans.py script reads the Behave JSON output, extracts per-scenario duration and status, and emits an OTEL span per scenario with test.origin, test.duration_ms, and test.flake_count as span attributes. Once you have 30 days of data in Grafana, the cost divergence becomes impossible to ignore. In the fintech pipeline, AI-generated scenarios ran 22% slower on average (more setup steps, less targeted assertions) and had a 3.1× higher flake rate than human-authored ones in the same feature area.

On the token side, use a generation wrapper that logs cost per file rather than per session. Cursor and direct Claude API calls both support this; the key is storing the cost at generation time so you can join it against the scenario's lifetime maintenance cost later:

# scripts/generate_feature.py
import anthropic, json, pathlib, time

client = anthropic.Anthropic()

def generate_and_log(prompt: str, output_path: str) -> dict:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}]
    )
    usage = response.usage
    cost_usd = (usage.input_tokens / 1_000_000 * 3.00) + \
               (usage.output_tokens / 1_000_000 * 15.00)
    record = {
        "output_path": output_path,
        "generated_at": time.time(),
        "input_tokens": usage.input_tokens,
        "output_tokens": usage.output_tokens,
        "cost_usd": round(cost_usd, 5)
    }
    pathlib.Path("logs/generation_costs.jsonl").open("a").write(
        json.dumps(record) + "\n"
    )
    pathlib.Path(output_path).write_text(response.content[0].text)
    return record

With both signals in place — per-scenario CI cost from OTEL and per-file generation cost from the log — you can build a simple join in Grafana or a Python notebook. In the SaaS Playwright pipeline, this join revealed that 34% of AI-generated scenarios were testing paths already covered by existing human-authored tests. Pruning that overlap dropped total suite run time from 18 minutes to under 11 minutes without changing coverage of critical paths. The Playwright-specific gain came from parallelism: fewer redundant scenarios meant the worker pool stopped thrashing on shared browser contexts.

Where the Budget Leaks: Three Mistakes Senior Engineers Still Make

The first mistake is generating tests against a live application rather than a contract or schema. When you point ChatGPT or Claude at a running Playwright session and ask it to generate scenarios from observed behavior, you get tests that encode current bugs as expected behavior. In the SpecFlow pipeline, this produced 17 scenarios that were technically passing but asserting on incorrect discount calculations — the AI faithfully described what the app did, not what it should do. The fix is to generate against OpenAPI specs or Pact contracts, not against observed UI state. It's slower to set up but produces scenarios with stable, intentional assertions.

The second mistake is treating AI-generated step definitions as production code without a coverage-overlap audit. Most generation prompts produce verbose, single-use step implementations rather than composable ones. The result is a step library with 200 definitions where 40 would suffice, and Cucumber-JVM 7's step-matching overhead compounds at scale. Run a duplicate-step analysis before merging any AI-generated feature file — a simple regex pass over your step registry catches 80% of the redundancy. The third mistake is not setting a flake budget per origin tag. Without a hard rule (e.g., "AI-generated scenarios exceeding 5% flake rate in 30 days are deleted, not fixed"), the suite accumulates technical debt faster than the generation saves time.

What the "AI Writes Your Tests" Narrative Gets Wrong

The dominant narrative is that AI-generated tests reduce total testing effort. The data says they reduce authoring effort and increase maintenance effort — and maintenance is where senior engineers' time actually goes. A Gherkin scenario takes 15 minutes to write and can require 3–4 hours of investigation when it starts flaking against a real application change. AI generation shifts the cost curve: you pay less upfront and more over the life of the suite. That's not inherently bad, but it means the ROI calculation only works if you're generating tests for stable, well-specified features — not exploratory coverage of moving targets.

A related myth is that AI-generated tests improve coverage metrics in a meaningful way. They do improve line and branch coverage numbers, often significantly. They don't reliably improve defect detection rate, which is the metric that matters. In the fintech pipeline, adding 80 AI-generated scenarios increased line coverage from 74% to 81% but did not change the mean-time-to-detect for production incidents over the following quarter. The generated tests were covering code paths that weren't failing in production. Coverage is a proxy; defect detection is the signal. Measure both before concluding that AI generation is paying for itself.

The accounting model here — OTEL-tagged scenario spans joined against generation cost logs — takes about a day to instrument and pays for itself in the first pruning pass. If you implement it, the next metric worth tracking is mean-time-to-detect on the scenarios you kept after the overlap audit. That number tells you whether your AI-generated tests are finding real regressions or just burning CI budget on paths that never break. The OpenTelemetry semantic conventions for testing (still evolving as of mid-2026) are worth watching as a standardization layer for exactly this kind of cross-pipeline cost attribution.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.