Most teams discover their retry budget is too generous the same way: a release is blocked by a failure that "never happens locally," and the post-mortem reveals the test has been retried successfully 300 times over the past six weeks. The retry passed, the pipeline stayed green, and no one filed a ticket. The budget did exactly what it was configured to do — and that's the problem.

Retry logic is a legitimate tool for absorbing transient infrastructure noise: a Kubernetes pod that took 400 ms too long to become ready, a DNS lookup that timed out once. But when retries are applied at the job level in GitHub Actions or Jenkins without per-test telemetry, they become a signal suppressor. The pipeline reports "passed on retry 2" and moves on. The underlying flake pattern — timing sensitivity, shared state, environment coupling — compounds silently.

By the end of this article you'll be able to distinguish the four structural types of flake, instrument your CI to surface retry signal rather than bury it, and set budget thresholds that alert rather than auto-heal. The tooling examples use GitHub Actions, Pytest 7, Playwright, and OpenTelemetry spans — but the patterns apply equally to Jenkins, Behave, and Cypress 13.

What "Retry Budget" Actually Means — and the Four Types of Flake It Hides

A retry budget is the maximum number of automatic re-executions a CI system will attempt before marking a test or job as failed. In GitHub Actions that's retry-on: error with a count; in Jenkins it's the Retry Failed Tests post-build step; in pytest it's pytest-rerunfailures with --reruns 3. The budget exists to handle genuine transient failures, but its scope is almost always too broad.

The four structural types of flake that retry budgets routinely absorb are: (1) timing flake — assertions that race against async state changes; (2) ordering flake — tests that pass only when run in a specific sequence because of shared fixture state; (3) environment flake — failures caused by resource contention, port collisions, or container startup variance; and (4) data flake — tests coupled to mutable external data or non-idempotent seed scripts. Each type has a different fix. Retrying all four with a single budget conflates them, making root-cause analysis much harder. Understanding how to triage flake as a first-class CI concern starts with knowing which category you're actually dealing with.

Instrumenting Retries So They Emit Signal Instead of Silence

The first step is separating retry telemetry from pass/fail status. A test that passes on retry 2 is not the same as a test that passes on the first attempt — but most CI dashboards report both as green. The fix is to emit a structured event on every retry attempt and route it to a time-series store (Grafana + InfluxDB, or OpenTelemetry traces into Tempo) independently of the final status.

Here's a conftest.py hook that attaches retry metadata to each test report and emits it as an OTLP span. It works with pytest-rerunfailures and the opentelemetry-sdk:

# conftest.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ci.flake.tracker")

def pytest_runtest_logreport(report):
    if report.when == "call":
        rerun_count = getattr(report, "rerun", 0)
        with tracer.start_as_current_span("test_execution") as span:
            span.set_attribute("test.name", report.nodeid)
            span.set_attribute("test.outcome", report.outcome)
            span.set_attribute("test.rerun_count", rerun_count)
            span.set_attribute("test.flake_candidate", rerun_count > 0)

With this in place, Grafana can show you a flake rate per test over time — not just a binary pass/fail. In one pipeline migration from Jenkins to GitHub Actions, adding this instrumentation revealed that 14 tests were consuming 80% of the retry budget. Run time dropped from 18 minutes to 4 once those 14 tests were quarantined and fixed, because the retry overhead was eliminated entirely.

On the CI configuration side, the goal is to make retry counts visible and bounded per test file, not per job. In GitHub Actions:

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run pytest with rerun telemetry
        run: |
          pytest --reruns 2 --reruns-delay 1 \
                 --tb=short \
                 -p no:randomly \
                 --junitxml=results.xml
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_ENDPOINT }}

      - name: Fail if retry rate exceeds budget
        run: |
          python scripts/check_retry_budget.py results.xml --max-retry-rate 0.05

The check_retry_budget.py script parses the JUnit XML, calculates the ratio of retried tests to total tests, and exits non-zero if it exceeds 5%. This turns the retry budget into an alerting threshold rather than a silent absorber. The same pattern applies to Playwright's built-in retries config — note that step-level retry logic in Playwright hooks can obscure the same patterns at a finer grain if you're not emitting per-step telemetry alongside it.

Where Senior Engineers Still Get Burned by Retry Configuration

The most common mistake is setting retry counts at the runner level rather than the test level. pytest --reruns 3 applies three retries to every test in the suite, including the ones that fail deterministically due to a real regression. A deterministic failure that gets retried three times before surfacing costs real CI minutes and delays the feedback loop by minutes. The fix is to use @pytest.mark.flaky(reruns=2) on known-flaky tests while leaving the global rerun count at zero — and to treat that mark as a temporary quarantine, not a permanent label.

The second mistake is treating retry success as equivalent to test health in SLA reporting. Engineering managers often track "pipeline pass rate" as a KPI. If that metric is computed from final outcomes without weighting for retry count, a suite with a 40% retry rate can still report 98% pass rate — and look healthy. The org-level fix is to track first-attempt pass rate as a separate metric. This is the number that actually reflects test suite confidence. When BDD scenarios are involved, integrating BDD into CI without sacrificing speed requires this distinction to be explicit in your reporting contract with stakeholders.

Myths About Flake That Retry Budgets Reinforce

Myth 1: If it passes on retry, it's not a real failure. This is the most damaging belief in CI culture. A test that fails once and passes twice is exhibiting non-determinism — which means it is not reliably verifying the behavior it claims to verify. Timing flake in async APIs is particularly dangerous here: the test may pass because the retry happened to land after the eventual-consistent state resolved, not because the system was correct. The same logic applies to polling assertions that mask timing failures in async API tests — retrying at the job level just adds another layer of concealment.

Myth 2: A low overall flake rate means the suite is healthy. A 2% flake rate sounds acceptable until you realize it's concentrated in five tests that touch the payment service, all of which have ordering dependencies on a shared database fixture. Aggregate metrics hide distribution. Myth 3: Quarantining flaky tests is a permanent solution. Quarantine is a triage tool with a TTL, not a disposal bin. Tests in quarantine should have a linked ticket, an owner, and an expiry date after which they are either fixed or deleted. Suites where quarantine is permanent tend to drift toward testing theater — the scenarios exist, they run, and they mean nothing.

The immediate next step is to add per-test retry telemetry to one pipeline this sprint and compute first-attempt pass rate as a separate metric from final pass rate. The gap between those two numbers is your actual flake debt. Once you have that baseline, the next thing worth measuring is mean-time-to-detect: how long does a genuinely broken test take to surface through the retry noise before someone acts on it? That number will tell you whether your budget is absorbing noise or absorbing signal.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.