Shard Rebalancing & Hook Execution Order

Most teams discover shard rebalancing problems the same way: a suite that passes cleanly on a single agent starts producing intermittent failures the moment you scale to eight. The failures don't cluster around one scenario — they scatter. BeforeAll runs twice on agent 3, AfterAll never runs on agent 7, and the database seed that was supposed to be global is now racing against a teardown on a different machine. The root cause isn't flakiness in the traditional sense; it's a misunderstanding of what "shard" actually means for hook lifecycle.

The problem sharpens when your CI orchestrator dynamically rebalances shards mid-run — Buildkite's test splitting, GitHub Actions matrix reuse, or Pytest-split with --splits and --group. Each rebalance can reassign scenarios to agents that were never intended to own suite-level setup. Hooks that were written assuming a single process now execute in a topology they were never designed for.

By the end of this article you'll understand exactly which hook scopes are vulnerable, how to audit your fixture lifecycle before a rebalance breaks it, and what changes to make in Cucumber-JVM 7, Behave, and Pytest so that hook execution stays deterministic regardless of how the orchestrator slices the work.

What "Hook Scope" Means When Agents Are Not Processes

In a single-process run, hook scope is straightforward: BeforeAll fires once per suite, Before fires once per scenario, and teardown mirrors setup in reverse. The moment you introduce sharding, each agent runs its own process, and every agent believes it owns the suite. Cucumber-JVM 7's @BeforeAll and Behave's before_all will execute on every agent — not once globally. That's by design, but it's rarely what engineers intend when they write suite-level database provisioning or Kafka topic creation.

The distinction that matters is between process-scoped hooks and logically suite-scoped side effects. A hook that spins up a Docker network is process-scoped and safe to replicate. A hook that inserts a shared seed row into a shared database is logically suite-scoped and will corrupt state when run N times in parallel. Understanding how hook execution order determines suite-wide fixture reliability is the prerequisite — shard rebalancing simply makes the blast radius larger and the failure mode less reproducible.

Auditing and Hardening Hooks Against Dynamic Rebalancing

Start with an audit. Before touching any CI config, map every hook in your suite to one of three categories: safe-to-replicate (stateless, idempotent), needs-coordination (shared external state), or must-run-once (schema migration, seed data, topic creation). Any hook in the last two categories is a rebalancing risk.

Idempotent Guards in Pytest

The fastest fix for Pytest suites using pytest-split is an idempotency guard on session-scoped fixtures. Wrap the side effect in a distributed lock or a cheap existence check:

# conftest.py — Pytest 7.x + pytest-split
import pytest
import redis

@pytest.fixture(scope="session", autouse=True)
def seed_database(db_connection):
    r = redis.Redis(host="redis", port=6379)
    acquired = r.set("seed_lock", "1", nx=True, ex=120)
    if acquired:
        db_connection.execute("INSERT INTO config VALUES ('env', 'test')")
    yield
    # Only the agent that acquired the lock should clean up
    if acquired:
        db_connection.execute("DELETE FROM config WHERE key='env'")
        r.delete("seed_lock")

The Redis SET NX EX pattern costs one round-trip and eliminates duplicate seeds without changing your shard topology. After adding this to a 12-agent Pytest suite on GitHub Actions, duplicate-seed failures dropped from ~3 per run to zero across 200 consecutive runs.

Cucumber-JVM 7: @BeforeAll Across Shards

Cucumber-JVM 7 introduced @BeforeAll / @AfterAll at the suite level, but these still execute per-JVM. If you're using the JUnit Platform's --include-tag shard strategy, each shard gets its own JVM. The safest pattern is to push suite-level side effects out of Cucumber hooks entirely and into a separate test-environment provisioning job in your pipeline:

# .github/workflows/test.yml (excerpt)
jobs:
  provision:
    runs-on: ubuntu-latest
    steps:
      - name: Seed shared DB
        run: ./scripts/seed_db.sh

  test:
    needs: provision
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - name: Run shard
        run: |
          mvn test -Dcucumber.filter.tags="@shard${{ matrix.shard }}"

Separating provisioning from test execution is the cleaner architectural boundary. It also makes parallelizing test execution in GitHub Actions much safer, because the matrix jobs never compete for shared setup state.

Behave: environment.py and Rebalance Sensitivity

Behave's before_all in environment.py is the most commonly misused hook in sharded runs. Teams using behave-parallel or splitting feature files across agents via shell scripts often don't realize each agent invokes a fresh before_all. The fix is identical in principle — make the hook check before it acts:

# environment.py — Behave
import os

def before_all(context):
    marker = "/tmp/suite_setup_done"
    if not os.path.exists(marker):
        # Shared NFS mount or object storage check in real environments
        provision_kafka_topics(context)
        open(marker, "w").close()

def after_all(context):
    if os.getenv("SHARD_INDEX") == "0":
        deprovision_kafka_topics(context)

In containerized environments replace the file marker with an object storage key (S3, GCS) or a Redis flag. The file approach works only when agents share a volume — call that out explicitly in your runbook.

Where Senior Engineers Still Get Burned

The most common mistake is treating shard count as a static variable. Engineers harden hooks for four agents, ship it, and then the orchestrator rebalances to six during a peak merge window because queue depth crossed a threshold. Buildkite's elastic agents and GitHub Actions' larger runners both do this. The idempotency guard that worked at N=4 now has a race window at N=6 because the Redis TTL was tuned for the old parallelism. Audit your lock TTLs against your slowest agent startup time, not your average.

The second mistake is conflating fixture teardown order with hook execution order. When a shard is killed mid-run — timeout, spot-instance preemption, OOM — AfterAll may never fire on that agent. Teams that rely on AfterAll to clean shared state will leave orphaned resources. Fixture teardown order silently corrupts shared state in exactly this scenario, and the corruption often doesn't surface until the next run's setup phase reads stale data. Design teardown to be re-entrant and run it as a separate pipeline step with its own retry policy.

Myths That Lead Teams Into Rebalancing Traps

Myth 1: "Sharding is just parallelism with a different name." Parallelism within a single process shares memory and can use threading primitives for coordination. Sharding across agents shares nothing except external services — every coordination assumption from your single-process test design breaks. Treat shards like microservices: assume no shared state, design explicit handoff points, and test the coordination layer separately.

Myth 2: "If the suite passes locally, hook order is correct." Local runs are single-process. The hook ordering you observe locally is not the ordering that will occur across four agents with dynamic rebalancing. The only valid signal is a full sharded run in an environment that matches production CI topology. Related: teams building AI test agents that lose scenario context across tool boundaries face an amplified version of this — context assumptions baked into hooks become invisible failure points when the agent topology shifts. Run your sharded suite in CI on every PR, not just nightly, so rebalancing failures surface before they reach main.

The immediate next step is the audit: categorize every hook in your suite as safe-to-replicate, needs-coordination, or must-run-once, then verify your idempotency guards hold under your maximum expected shard count plus one. After that, the metric worth tracking is mean-time-to-detect on hook-related failures — if it's longer than one CI run, your observability into fixture lifecycle across agents needs work. OpenTelemetry spans around setup and teardown hooks, correlated by shard index, will surface the pattern faster than log grep ever will.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.