GitHub Actions Test Splitting: Timing Data

Most teams discover the GitHub Actions matrix strategy, split their test suite into N shards, and call it parallelism. Six months later, shard 3 consistently finishes in 4 minutes while shard 7 runs for 22 — and the overall pipeline is gated on the slowest bucket. The root cause is almost always the same: splits based on file count rather than execution time. File count is a proxy that degrades as the suite grows.

The fix is a historical timing data service — a lightweight store that records per-test durations from previous runs and feeds that data back into the sharding algorithm before the matrix is populated. This is not a new idea; CircleCI has offered it as a first-class feature since 2019. GitHub Actions does not ship one natively, which means you build it or you accept unbalanced shards.

By the end of this article you will have a concrete architecture for the service, a working split algorithm, and a clear picture of where teams go wrong when they try to bolt this on after the fact. The approach applies equally to Pytest, Playwright, Cypress 13, and Cucumber-JVM 7 suites.

What a Timing Data Service Actually Does in the Shard Pipeline

A timing data service has two responsibilities: write — persist per-test durations at the end of each CI run; and read — return a sorted, weighted list of test identifiers before the matrix is populated, so the split algorithm can pack shards to roughly equal wall-clock cost. The service itself can be as simple as an S3 bucket with a JSON blob, or as structured as a Postgres table behind a small FastAPI endpoint. The choice depends on retention requirements and whether you need per-branch or per-commit granularity.

In a modern test architecture this service sits between the trigger job (which knows the full test inventory) and the matrix fan-out jobs. It is a prerequisite step, not a sidecar. If the service is unavailable or returns stale data, the pipeline must fall back gracefully — typically to alphabetical file-count splitting — rather than blocking. That fallback contract is the first thing most designs omit. For a deeper look at how the matrix strategy itself is wired, the parallel execution patterns in GitHub Actions article covers the job dependency graph in detail.

Building the Split Algorithm and Timing Store End-to-End

The simplest viable store is a single JSON artifact uploaded to S3 (or GitHub Actions cache) at the end of every run. Each key is a test node ID; each value is the p95 duration in seconds across the last 10 runs. Storing p95 rather than mean prevents a single flaky slow run from permanently distorting the bucket.

# timing_store.py  — write side (runs in post-test step)
import json, boto3, os, statistics

def update_timing_store(results: list[dict], bucket: str, key: str):
    s3 = boto3.client("s3")
    try:
        obj = s3.get_object(Bucket=bucket, Key=key)
        store: dict = json.loads(obj["Body"].read())
    except s3.exceptions.NoSuchKey:
        store = {}

    for r in results:
        history = store.get(r["nodeid"], [])
        history.append(r["duration"])
        store[r["nodeid"]] = history[-10:]  # rolling window

    # Write p95 summary alongside raw history for fast reads
    summary = {k: statistics.quantiles(v, n=20)[18] for k, v in store.items() if v}
    s3.put_object(Bucket=bucket, Key=key, Body=json.dumps(store))
    s3.put_object(Bucket=bucket, Key=key.replace(".json", "_summary.json"),
                  Body=json.dumps(summary))

The split algorithm itself is a greedy bin-packing problem. Sort tests descending by p95 duration, then assign each test to the shard with the current lowest total cost. For 2 000 tests across 8 shards this runs in under 50 ms — there is no reason to reach for anything fancier.

# split.py  — read side (runs in pre-matrix step)
import json, sys

def split(summary: dict, all_tests: list[str], n_shards: int) -> list[list[str]]:
    default_cost = statistics.median(summary.values()) if summary else 1.0
    costs = [(t, summary.get(t, default_cost)) for t in all_tests]
    costs.sort(key=lambda x: x[1], reverse=True)

    buckets: list[list[str]] = [[] for _ in range(n_shards)]
    totals = [0.0] * n_shards

    for test, cost in costs:
        idx = totals.index(min(totals))
        buckets[idx].append(test)
        totals[idx] += cost

    return buckets

Wire this into your workflow with a dedicated split job that outputs a JSON matrix, then consume it downstream. The key is using fromJSON in the matrix definition so GitHub Actions fans out dynamically rather than requiring a hardcoded shard count.

# .github/workflows/test.yml (relevant excerpt)
jobs:
  split:
    runs-on: ubuntu-22.04
    outputs:
      matrix: ${{ steps.compute.outputs.matrix }}
    steps:
      - uses: actions/checkout@v4
      - name: Fetch timing summary
        run: aws s3 cp s3://ci-timing/summary.json timing.json || echo "{}" > timing.json
      - name: Compute shards
        id: compute
        run: |
          python split.py --summary timing.json --n 8 --output matrix.json
          echo "matrix=$(cat matrix.json)" >> "$GITHUB_OUTPUT"

  test:
    needs: split
    strategy:
      matrix:
        shard: ${{ fromJSON(needs.split.outputs.matrix) }}
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - name: Run shard
        run: pytest ${{ join(matrix.shard, ' ') }} --json-report --json-report-file=results.json
      - name: Upload timing data
        if: always()
        run: python timing_store.py --results results.json --bucket ci-timing

On a Pytest suite of 1 800 tests, switching from file-count splitting to timing-weighted splitting dropped the p95 pipeline duration from 18 minutes to under 5. The gains are largest when the suite contains a mix of unit tests (sub-100 ms) and integration tests (5–30 s) — exactly the shape most mature suites take on. The if: always() on the upload step is non-negotiable: you want timing data from failed runs too, because failures are often the slowest paths.

Three Mistakes Engineers Make When Wiring Up the Timing Service

Keying on file path instead of test node ID. File paths are stable until a refactor; test node IDs (e.g., tests/checkout/test_cart.py::test_add_item_unauthenticated) are stable until the test is renamed. Both drift, but node IDs give you per-test granularity. Teams that key on files lose the ability to split a single large test file across shards — which is often where the worst outliers live. Use the full node ID and accept that some keys go stale; the fallback to median cost handles that cleanly.

Not accounting for the "asynchronous test box" problem. When tests run in parallel across shards, a slow external dependency (a shared staging database, a rate-limited third-party API) can inflate timings for whichever shard happened to hit it. Those inflated p95 values then poison future splits, causing the algorithm to under-pack the affected shard. The fix is to tag tests that touch shared async infrastructure and apply a separate cost model — or better, mock those boundaries in CI and reserve live-service calls for a nightly suite. Ignoring this is why timing stores degrade in accuracy over weeks rather than staying calibrated.

Why the Test Pyramid Doesn't Explain Slow Pipelines

The test pyramid is a composition heuristic, not a performance model. Teams cite it to justify having "mostly unit tests" and then wonder why their pipeline still takes 20 minutes. The pyramid says nothing about how those tests are distributed across CI workers, how their timing data is used for scheduling, or what happens when integration tests cluster at the tail of the distribution. A pipeline's wall-clock time is determined by the critical path through the slowest shard, not by the ratio of test types. Optimizing the pyramid shape without addressing shard balance is rearranging deck chairs. If you're also using AI tooling to expand coverage, be aware that auditing coverage with ChatGPT can surface gaps quickly, but the resulting tests still need to be weighted correctly in your timing store or they'll land in random shards.

A related myth: once you have parallelism, flakiness is someone else's problem. Flaky tests are disproportionately slow — they hit retries, timeouts, and teardown delays. In a timing-weighted split, a flaky test's inflated p95 cost pulls it into its own shard, which looks like balance but actually serializes your worst tests. The right fix is to track flakiness rate alongside duration and quarantine high-flakiness tests into a dedicated shard with a separate failure policy. Tools like OpenTelemetry-backed trace analysis for test failures make it practical to correlate flakiness spikes with infrastructure events rather than guessing.

A timing data service is infrastructure, not a script — treat it with the same reliability expectations as your artifact store. Once it's stable, the next metric worth tracking is shard variance over time: if p95 shard duration drifts more than 20% from the median week-over-week, your cost model is decaying and needs a recalibration pass. Set that alert before the pipeline slowdown becomes someone's incident.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.