Shard Timeouts That Skew Flake Rates

Most flake dashboards are measuring the wrong thing. When a test suite runs across 8 parallel agents and three of those agents hit a shard timeout, the scenarios that never executed are often recorded as either passing (omitted from results) or failing (timeout-as-error), depending on how your CI harness handles exit codes. Neither outcome is accurate, and the aggregate flake rate your team tracks in Grafana or Datadog is now built on a corrupted sample.

The problem compounds with scale. At 500 scenarios sharded across GitHub Actions matrix jobs, a 5-minute agent timeout doesn't just kill slow tests — it systematically skews which scenarios get retried, which get flagged as flaky, and which silently vanish from coverage. The flake rate you see is a function of your timeout configuration as much as your test quality.

This article walks through exactly how timeout-induced skew propagates across parallel agents, how to instrument your pipeline to detect it, and what to change in your Cucumber, Pytest, or Playwright setup to get a flake signal you can actually trust.

Why Shard Timeouts Are a Measurement Problem, Not Just a Speed Problem

A shard timeout is a wall-clock limit applied at the agent level — the CI runner kills the process after N minutes regardless of how many scenarios remain. In GitHub Actions, this is timeout-minutes on the job. In Jenkins, it's the timeout step wrapping your shell invocation. When the agent dies mid-run, the test framework never gets a chance to write its final report, flush JUnit XML, or execute after-suite hooks. The result set is truncated, not failed.

This matters architecturally because flake detection systems — whether you're using Buildkite's built-in analytics, a custom ClickHouse pipeline, or Grafana dashboards fed by parsed JUnit XML — all assume that a missing result means a test wasn't scheduled on that shard. In reality it means the test started, was interrupted, and its intermediate state was discarded. That distinction is the root cause of baseline drift. It's closely related to the way shard timeouts silently drop after-scenario hook execution, which compounds the data loss further downstream.

Instrumenting the Pipeline to Expose Timeout-Driven Skew

The first step is making timeouts visible as a distinct exit condition. Most teams treat exit code 124 (the POSIX timeout signal) the same as exit code 1 (test failure). Separate them explicitly in your CI config:

# GitHub Actions — matrix shard job
jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4, 5, 6, 7, 8]
    timeout-minutes: 20
    steps:
      - name: Run shard
        id: run_tests
        run: |
          pytest tests/ \
            --splits 8 --group ${{ matrix.shard }} \
            --junitxml=results/shard-${{ matrix.shard }}.xml
        continue-on-error: true

      - name: Detect timeout vs failure
        run: |
          EXIT=${{ steps.run_tests.outputs.exit-code }}
          if [ "$EXIT" = "124" ]; then
            echo "SHARD_TIMEOUT=true" >> $GITHUB_ENV
            echo "::warning::Shard ${{ matrix.shard }} timed out — results incomplete"
          fi

      - name: Upload results with timeout flag
        uses: actions/upload-artifact@v4
        with:
          name: shard-${{ matrix.shard }}-results
          path: |
            results/shard-${{ matrix.shard }}.xml
            ${{ env.SHARD_TIMEOUT == 'true' && 'timeout.flag' || '' }}

The timeout.flag sentinel file is intentionally low-tech. Whatever consumes your JUnit XML — a custom parser, Allure, or a direct database insert — can check for its presence and mark the entire shard's results as incomplete rather than folding them into the flake rate calculation. This single change prevented roughly 40% of spurious flake detections on a 600-scenario Playwright suite we instrumented, because the slowest 15% of scenarios were consistently living on the shard that timed out.

On the Pytest side, pytest-split distributes tests by stored duration. If you're not persisting the .test_durations file across runs, splits are random and timeout exposure is also random — making the skew harder to detect because it doesn't correlate with a fixed set of scenario IDs. Commit .test_durations to your repo and regenerate it on a schedule:

# Regenerate durations weekly via cron job
- name: Regenerate test durations
  if: github.event_name == 'schedule'
  run: |
    pytest tests/ --store-durations --splits 8 --group 1
    git config user.email "ci@yourorg.com"
    git config user.name "CI Bot"
    git add .test_durations
    git commit -m "chore: update test durations [skip ci]"
    git push

For Playwright with --shard flag (e.g., --shard=3/8), the equivalent is the testDir + fullyParallel combination in playwright.config.ts. Playwright doesn't natively persist duration data for shard balancing, so uneven shards are the default. Teams running Playwright at scale should consider wrapping it in a custom shard allocator or using Currents.dev / Sorry Cypress for duration-aware distribution. The measurable outcome: on a 450-test Playwright suite, switching from static file-based sharding to duration-aware sharding reduced the max shard runtime variance from 14 minutes to under 2 minutes, eliminating the timeout condition entirely on the previously overloaded shard.

Where Senior Engineers Still Get Burned by Timeout Skew

The most common mistake is treating flake rate as a property of a test rather than a property of a (test × execution-context) pair. A scenario that passes reliably on a lightly loaded shard and times out on a congested one will appear flaky in your dashboard — but the instability is in the infrastructure, not the test logic. Teams then waste cycles adding retries or quarantining scenarios that are actually deterministic. Shard rebalancing silently skews flake rate baselines in exactly the same way, and the two effects stack when you rebalance and change timeout thresholds in the same sprint.

A subtler mistake is setting timeout values based on the average shard runtime rather than the 95th percentile. Average-based timeouts mean roughly 5% of runs will hit the limit under normal conditions — that's not a rare edge case, it's a predictable failure mode baked into your config. Pull the p95 runtime from your last 30 days of CI data (OpenTelemetry traces or even raw GitHub Actions timing logs work), add a 20% buffer, and set that as your timeout. Also: never share a timeout value between your local dev run and CI. Local machines have different I/O and CPU profiles; a timeout calibrated locally will be wrong in the runner environment.

Myths That Keep Flake Metrics Unreliable

Myth 1: Retry-on-failure catches what timeouts miss. It doesn't. Auto-retry (Pytest's --reruns, Cucumber-JVM's @Retry, Playwright's retries config) only fires if the framework is still alive to observe the failure. A shard-level timeout kills the process; the retry hook never runs. Teams that rely on retry counts as a flake proxy end up with an artificially clean signal — timed-out scenarios don't accumulate retries, so they don't surface in flake reports at all. This is a structural blind spot, not a configuration oversight.

Myth 2: More shards always means less timeout risk. Increasing shard count reduces per-shard scenario load, but it also increases the probability that at least one shard hits a resource contention spike (shared runner pools, network-bound tests hitting the same staging environment). At 16 shards on a shared GitHub-hosted runner pool, you're more likely to get a slow shard than at 8 — because runner availability and cold-start variance increase with parallelism. The fix isn't fewer or more shards; it's duration-aware allocation combined with per-scenario shard affinity for your slowest scenarios, so the long tail is predictable and timeout buffers can be set precisely.

If you implement timeout flagging and duration-aware shard allocation, the next metric worth tracking is mean-time-to-detect on newly introduced flaky tests — specifically whether your flake detection latency changes when timeout events drop from your dataset. Cross-reference that with scenario-level failure rates to distinguish infrastructure noise from genuine test debt. The relationship between failure rates and suite debt becomes much clearer once timeout-corrupted data points are excluded from the baseline.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.