Scenario Failure Rates and Suite Debt

Most teams track pass/fail at the pipeline level. Green build ships; red build blocks. That binary hides a class of structural rot that compounds quietly over months: scenarios that fail intermittently, scenarios that only fail in certain tag-selected subsets, scenarios whose step bindings have drifted so far from the domain that a single shared helper change drops thirty tests at once. The signal is there — it's just being averaged away.

Scenario-level failure rates — tracked per scenario, per tag, per step definition, over time — are one of the clearest leading indicators of suite debt. Not test flakiness in the colloquial sense, but structural fragility: the kind that surfaces when a fixture changes, a shared library gets refactored, or a new engineer adds a Scenario Outline row that nobody stress-tested.

By the end of this article you'll have a concrete measurement model, a Python-based aggregation pattern you can wire into your existing CI output, and a framework for deciding when a failure-rate spike is a product bug versus a test architecture problem.

Failure Rate as a Structural Signal, Not a Flakiness Metric

Flakiness tooling — Cypress 13's retry dashboard, Playwright's --repeat-each flag, BuildPulse, Trunk Flaky Tests — measures whether a specific scenario passes on retry. That's a test-execution concern. Scenario-level failure rate is a design concern: what percentage of runs, over a rolling window, does a given scenario fail regardless of retry outcome? A scenario that fails 40% of runs and passes 60% is flaky. A scenario that fails 5% of runs but always in the same CI stage, for the same tag group, after the same fixture setup step — that's structural debt pointing at a specific seam in your architecture.

The distinction matters because the remediation paths diverge completely. Flakiness gets fixed with better isolation, smarter waits, or quarantine policies. Structural debt gets fixed by redesigning fixture ownership, decomposing overloaded step libraries, or pruning Scenario Outline tables that have silently diverged from the domain they were written to cover. Treating structural failures as flakiness means you'll keep retrying scenarios that are trying to tell you something about your suite's load-bearing walls.

Building a Scenario-Level Failure Rate Tracker

The fastest path is parsing JUnit XML or Cucumber JSON output that your CI already produces. Most runners — Behave, Cucumber-JVM 7, SpecFlow, Pytest-BDD — emit one of these formats. Aggregate across pipeline runs into a time-series store (Postgres works fine; so does a ClickHouse table if you're at scale) and query per scenario URI, not per suite.

# Behave JSON → per-scenario failure rate aggregator (Python 3.11+)
import json, pathlib, sqlite3
from datetime import datetime

def ingest(report_path: str, run_id: str, db: sqlite3.Connection):
    data = json.loads(pathlib.Path(report_path).read_text())
    rows = []
    for feature in data:
        for element in feature.get("elements", []):
            if element["type"] != "scenario":
                continue
            uri = f"{feature['uri']}::{element['name']}"
            tags = [t["name"] for t in element.get("tags", [])]
            passed = all(s["result"]["status"] == "passed"
                         for s in element["steps"])
            rows.append((run_id, uri, ",".join(tags),
                         int(passed), datetime.utcnow().isoformat()))
    db.executemany(
        "INSERT INTO scenario_runs(run_id,uri,tags,passed,ts) VALUES(?,?,?,?,?)",
        rows
    )
    db.commit()

def failure_rate(uri: str, window: int, db: sqlite3.Connection) -> float:
    cur = db.execute(
        """SELECT 1.0 - AVG(passed) FROM scenario_runs
           WHERE uri=? ORDER BY ts DESC LIMIT ?""",
        (uri, window)
    )
    return cur.fetchone()[0] or 0.0

Wire this into your GitHub Actions post-step or Jenkins post-build stage. A 30-run rolling window is enough signal for most teams; 50 runs gives cleaner percentiles on low-frequency scenarios. Once you have per-scenario rates, the next query that earns its keep is grouping by tag — because tags frequently become an unintended test selection policy, and failure rates cluster along those same tag boundaries when a shared fixture is the root cause.

-- Top failure-rate scenarios in the last 30 runs, grouped by first tag
SELECT
    SUBSTR(tags, 1, INSTR(tags||',', ',')-1) AS primary_tag,
    uri,
    ROUND(1.0 - AVG(passed), 3)              AS failure_rate,
    COUNT(*)                                 AS sample_size
FROM scenario_runs
WHERE ts >= datetime('now', '-14 days')
GROUP BY uri
HAVING sample_size >= 10
ORDER BY failure_rate DESC
LIMIT 20;

In a real migration project (Selenium 4 grid → Playwright, ~1,400 scenarios), this query surfaced 23 scenarios with failure rates above 15% — all sharing a single @db-reset tag. The fix wasn't in the scenarios themselves; it was in the hook execution order managing the database fixture. Addressing that one fixture boundary dropped overall suite failure rate from 8.2% to 1.1%, and CI wall-clock time fell from 18 minutes to 4 because retry passes were eliminated. The scenarios hadn't changed; the structural debt underneath them had.

For teams running Playwright or Cypress 13, you can emit the same structured data from their native reporters with a custom reporter plugin, then feed it into the same SQLite or Postgres schema. The aggregation logic is reporter-agnostic; the schema is what matters. If you're already shipping OpenTelemetry traces from your test runs, add scenario URI as a span attribute and push failure rates as a gauge metric to Grafana — that gives you alerting without a separate pipeline.

Where Senior Engineers Still Get Tripped Up

The most common mistake is aggregating at the feature file level rather than the scenario level. Feature-level pass rates look stable right up until they don't, because a single reliable scenario in a file masks three structurally broken ones. The second mistake is treating the failure-rate query as a one-time audit rather than a continuous signal. Teams run the analysis after an incident, fix the immediate offenders, and stop tracking — so debt re-accumulates invisibly over the next quarter. Both mistakes share the same root: the mental model that test health is a binary state rather than a distribution.

A subtler trap is conflating high failure rate with low value. A scenario that fails 30% of runs might be your most valuable test — it's catching a genuinely unstable integration boundary. Before quarantining or deleting high-failure scenarios, cross-reference with scenario risk scores to understand whether the failure is exposing real coverage gaps or just structural noise. Deleting a high-failure scenario that covers a critical path is how teams end up shipping regressions they used to catch.

What Most Teams Misread About Failure Rate Data

Myth 1: a low overall suite failure rate means the suite is healthy. A suite with 2% overall failure rate can have 40 scenarios each failing 25% of the time, offset by 1,960 scenarios that never fail because they test nothing that changes. The aggregate hides both the structural debt and the coverage hollowness. Myth 2: AI-generated test scenarios solve the debt problem. Generated scenarios can fill coverage gaps quickly, but they inherit whatever structural problems exist in your step library and fixture layer — sometimes faster than humans do. The real costs of AI-generated tests include the maintenance burden of scenarios that fail structurally from day one because they were scaffolded against a brittle shared step.

Myth 3: retry-until-green CI policies surface the real failure rate. They don't — they suppress it. A scenario that passes on the third retry is recorded as passing in most dashboard tooling, which means your rolling failure rate is systematically understated. If your CI uses retry-then-fail policies, you need to log the first-attempt result separately, or your failure-rate time series is measuring retry luck, not structural stability. This is a data collection problem, not a test design problem, and it needs to be fixed at the pipeline level before any failure-rate analysis is trustworthy.

Scenario-level failure rates are most useful when they're boring — a flat line near zero that spikes visibly when something structural breaks. Getting there means continuous measurement, not periodic audits. If you implement the aggregation pattern above, the next metric worth instrumenting is mean-time-to-detect on first-attempt failures by step definition: that query will tell you which step bindings are load-bearing and which are quietly accumulating blast radius. That's where the next round of structural debt is hiding.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.