Retry-Then-Fail CI Hides Flake Root Causes

Most CI pipelines in production today will silently retry a failing job, mark it green on the second attempt, and move on. GitHub Actions retry-on-error, Jenkins' Retry build step, and Argo Workflows' retryStrategy all ship this capability out of the box — and nearly every platform team enables it within a month of feeling flake pain. The policy feels like a pragmatic tradeoff. It is, in fact, a data-destruction decision.

When a test fails on attempt one and passes on attempt two, the pipeline records a green build. What it does not record is the failure signal, the timing delta, the log diff between attempts, or any structured metadata that would let you later ask: was this the same flake that bit us on Tuesday? That first-attempt failure is gone. Multiply that across 50 pipelines running 40 times a day and you have a flake signal-to-noise problem that no dashboard can fix retroactively.

This article covers why retry-then-fail policies are structurally hostile to root-cause analysis, how to instrument retries so you preserve the signal instead of discarding it, and what a triage-ready CI contract looks like in practice. By the end you will have concrete YAML, a Python logging pattern, and a mental model for when retry budgets are acceptable versus when they are actively harmful.

What "Retry-Then-Fail" Actually Does to Your Failure Signal

A retry-then-fail policy is any CI configuration that re-executes a failed step or job up to N times before reporting a final failure. The intent is to absorb transient infrastructure noise — a flapping container registry, a slow DNS response, a race in a shared Selenium Grid. The mechanism is simple: if exit code ≠ 0, try again. What makes it structurally dangerous is that the policy operates at the job level, not the test level, so it cannot distinguish between an infrastructure hiccup and a genuine, reproducible race condition in your application under test.

In a modern test architecture — where Playwright 1.44 runs headed in a Docker sidecar, or Behave drives a Kafka consumer via a test harness — a "transient" failure is often the only observable symptom of a deeper state problem: leaked fixtures, non-idempotent setup, or a timing assumption baked into a wait_for_selector call. Retrying at the job level discards the first failure's stdout, the screenshot artifact, and the wall-clock duration of the failed attempt. Those three data points are precisely what a root-cause investigation needs. When you discard them automatically, you are not absorbing noise — you are building retry budgets that hide systemic flake behind a green badge.

Instrumenting Retries to Preserve the Failure Record

The fix is not to remove retries — it is to make every retry attempt a first-class CI artifact. That means structured logging on attempt one, conditional artifact upload regardless of final outcome, and a failure metadata schema that a downstream aggregator (Grafana, Datadog, a Postgres table) can ingest. Here is what that looks like in GitHub Actions:

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - name: Run Playwright suite (attempt ${{ github.run_attempt }})
        id: playwright
        run: |
          npx playwright test --reporter=json 2>&1 | tee results/attempt-${{ github.run_attempt }}.json
        continue-on-error: true

      - name: Upload attempt artifact (always)
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-attempt-${{ github.run_attempt }}
          path: results/

      - name: Emit flake metadata to collector
        if: steps.playwright.outcome == 'failure'
        run: |
          curl -s -X POST "$FLAKE_COLLECTOR_URL" \
            -H "Content-Type: application/json" \
            -d "{\"run_id\":\"${{ github.run_id }}\",\"attempt\":${{ github.run_attempt }},\"sha\":\"${{ github.sha }}\"}"

      - name: Fail the step if tests failed
        if: steps.playwright.outcome == 'failure'
        run: exit 1

continue-on-error: true on the test step — combined with the explicit exit 1 at the end — lets you upload artifacts and emit metadata before the job fails, without swallowing the failure. github.run_attempt is the key: it increments on each retry, so attempt-1.json and attempt-2.json are both preserved and independently queryable. A team that adopted this pattern on a 600-test Playwright suite reduced mean-time-to-triage on recurring flakes from 3 days to 4 hours, because engineers could diff two attempt artifacts instead of reconstructing the failure from memory.

On the test-runner side, Pytest's pytest-rerunfailures plugin and Playwright's built-in retries config both support per-test retry counts. The trap is enabling these in addition to CI-level retries, which creates a retry stack: a single flaky test can silently consume up to test_retries × job_retries attempts before surfacing. That multiplicative effect is exactly why step retry logic masks real flakiness in ways that job-level metrics never reveal. Pick one retry layer and instrument it; don't stack them.

# pytest.ini — single retry layer, with structured logging
[pytest]
reruns = 1
reruns_delay = 2

# conftest.py — emit attempt metadata on each rerun
import pytest, json, os, time

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call" and report.outcome == "failed":
        attempt = getattr(item, "execution_count", 1)
        payload = {
            "test": item.nodeid,
            "attempt": attempt,
            "duration": report.duration,
            "timestamp": time.time(),
        }
        with open(f"flake_log_{attempt}.jsonl", "a") as f:
            f.write(json.dumps(payload) + "\n")

This hook writes a JSONL record on every failed attempt — not just the final one. Feed those records into any time-series store and you can query: which tests have a first-attempt failure rate above 5% in the last 14 days? That query is the foundation of a flake triage process that treats CI as a first-class citizen rather than a black box. Without the per-attempt record, that query is unanswerable.

Where Senior Engineers Still Get Burned on Retry Configuration

The most common mistake is treating retry-on-error as an infrastructure concern and delegating it to the platform team, while test authors independently configure retries: 2 in playwright.config.ts or --reruns 3 in Pytest. Neither team knows the other has done it. The result is a retry stack where a single flaky scenario can pass on attempt 6 of a possible 9 — and the build stays green with zero visibility into what happened. The fix is a single, documented retry policy enforced at one layer, with the other layer disabled or set to zero.

The second mistake is scoping artifact uploads to if: failure() instead of if: always(). When a job retries and the second attempt passes, GitHub Actions evaluates the final job status as success — so if: failure() never triggers, and the first-attempt artifacts are never uploaded. You lose exactly the evidence you need. The third mistake is using wall-clock job duration as a flake proxy. A job that retried and passed in 12 minutes looks identical to a clean 12-minute run in most dashboards. Duration is not a flake signal; per-attempt exit codes are.

Myths That Keep Retry Policies in Place Longer Than They Should Be

Myth 1: Retries are a reasonable cost of operating at scale. At scale, the opposite is true. A 2% flake rate across 1,000 daily test runs means 20 silent failures per day — each one a data point that could identify a root cause. Retrying them away means you need 10× the flake rate before the pattern becomes visible in aggregate metrics. The teams that run the most tests are the ones who can least afford to discard failure data. Myth 2: Removing retries will break the build and block deploys. This conflates two problems: infrastructure instability (which retries legitimately address) and test-level flakiness (which retries mask). The correct response to infrastructure instability is infrastructure observability — OpenTelemetry spans on your test runner, not a blanket retry count of 3.

Myth 3: Flake is a test-quality problem, not a CI-policy problem. Flake has multiple root causes: timing assumptions, shared state, non-idempotent fixtures, and genuine application races. A retry policy treats all of them identically, which means it cannot inform any of them. The teams that reduce flake fastest are the ones who instrument retries as described above, build a flake leaderboard from the JSONL data, and fix the top-five offenders every sprint. That is a process decision, not a test-quality decision. Blaming test authors while the CI policy destroys the evidence is a org-level failure mode, not an engineering one.

If you implement per-attempt artifact uploads and a JSONL flake log this sprint, the next metric worth tracking is first-attempt pass rate by test file — not overall suite pass rate. A file sitting at 70% first-attempt pass rate with a 100% final pass rate is your highest-priority flake target, and it is invisible without this instrumentation. From there, cross-reference with recent fixture changes or background step modifications; non-idempotent setup is the most common culprit once the retry noise is removed.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.