Broker Partition Reassignment & Contract State

Pact contract tests pass green in your CI pipeline, your Kafka consumer group is healthy, and then a broker partition reassignment fires during a rolling restart — and your consumer silently starts reading from the wrong offset. The contract didn't change. The schema didn't change. The test fixture didn't change. But the state the consumer expects no longer matches what the broker delivers. This is not a Pact problem; it's a distributed-state problem that Pact was never designed to absorb.

The core issue is that consumer contract tests model a point-in-time agreement between a producer and a consumer. Partition reassignment introduces a topology change that shifts which broker node owns a partition, potentially reordering message delivery or replaying already-committed offsets — neither of which is visible to the contract layer. Your Pact provider verification still passes because the message payload schema is valid; the contract says nothing about delivery order or offset continuity.

By the end of this article you'll know exactly where contract state breaks during reassignment, how to instrument your BDD scenarios to catch it before it reaches production, and which parts of the Kafka and Pact toolchain to wire together in a GitHub Actions or Argo pipeline to make the failure deterministic rather than intermittent.

Why Partition Reassignment Is Invisible to Contract State

A Pact consumer contract captures the shape of an interaction: the message key, payload schema, and any metadata fields the consumer reads. What it does not capture is the partition assignment topology at the time the interaction was recorded. When a broker reassigns partitions — triggered by a broker failure, a kafka-reassign-partitions.sh execution, or Confluent's auto-balancer — the consumer group coordinator issues a rebalance. During that rebalance window, in-flight messages may be re-delivered, and the consumer's committed offset on the old partition leader may not yet have propagated to the new leader.

In a BDD microservices contract testing pipeline, this surfaces as a scenario that passes during the provider verification phase (because the message payload is structurally valid) but fails at the consumer integration layer (because the consumer processes a duplicate or out-of-order message and enters an unexpected state). The contract fixture was built against a stable partition assignment; the live environment has a different one. This is the same class of problem as schema registry drift breaking contract test suites — the artifact the test trusts has diverged from runtime reality, and no static assertion catches it.

Instrumenting BDD Scenarios to Surface Reassignment-Driven State Breaks

The first step is making partition assignment an explicit fixture concern rather than an ambient assumption. In Behave or Pytest-BDD, add a step that asserts partition ownership before the consumer under test processes any messages. If the assignment changes between fixture setup and the scenario's When step, the test should fail fast with a clear signal — not silently corrupt state downstream.

# features/consumer_contract.feature
Feature: Order service consumes payment events

  Background:
    Given the payment topic has a stable partition assignment
    And the consumer group "order-svc" is not in rebalance

  Scenario: Consumer processes a confirmed payment
    When the broker delivers a "payment.confirmed" event on partition 2
    Then the order service transitions the order to "PAID"
    And no duplicate state transitions are recorded

The Background steps here are doing real work — they call the Kafka AdminClient to assert partition leadership before the scenario runs. If you skip this guard, you're trusting the environment. The step implementation below uses confluent-kafka-python 2.3 and fails immediately if a rebalance is in progress, which keeps the BDD pipeline signal clean rather than producing a flaky intermittent failure. This is also the layer where idempotent replay can expose hidden state in contract fixtures — a reassignment that triggers offset replay hits the same guard.

# features/steps/partition_steps.py
from confluent_kafka.admin import AdminClient
from confluent_kafka import Consumer, KafkaException

BROKER = "kafka:9092"
TOPIC = "payment-events"

@given("the payment topic has a stable partition assignment")
def assert_stable_partition_assignment(context):
    admin = AdminClient({"bootstrap.servers": BROKER})
    metadata = admin.list_topics(topic=TOPIC, timeout=10)
    topic_meta = metadata.topics[TOPIC]
    for pid, part in topic_meta.partitions.items():
        assert part.leader >= 0, (
            f"Partition {pid} has no leader — reassignment in progress"
        )

@given('the consumer group "order-svc" is not in rebalance')
def assert_no_rebalance(context):
    admin = AdminClient({"bootstrap.servers": BROKER})
    groups = admin.list_consumer_groups()
    for g in groups.result().valid:
        if g.group_id == "order-svc":
            assert g.state.name != "PreparingRebalance", (
                "Consumer group is rebalancing — abort scenario"
            )
            return
    raise AssertionError("Consumer group 'order-svc' not found")

On the Pact side, add a provider state handler that snapshots the current partition assignment into the provider state metadata. This gives you a correlation point: when a Pact verification fails in CI, you can diff the recorded partition assignment against the live one. In a GitHub Actions pipeline, emit this snapshot as a job artifact so post-mortem analysis doesn't require reproducing the environment.

# .github/workflows/contract-verify.yml (excerpt)
- name: Run Pact provider verification
  run: pytest tests/pact_provider_verify.py -v
  env:
    KAFKA_BROKER: ${{ secrets.KAFKA_BROKER }}
    PACT_BROKER_URL: ${{ secrets.PACT_BROKER_URL }}

- name: Upload partition snapshot
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: partition-assignment-snapshot
    path: artifacts/partition_snapshot.json

A team running Confluent Platform 7.5 with 12 partitions across 3 brokers reduced their contract-related flake rate from roughly one failure per 40 pipeline runs to zero over a two-week window by adding the partition-leader assertion in the Background step. The fix cost one afternoon; the diagnosis had taken three sprints of intermittent investigation.

Where Senior Engineers Still Get Burned by Reassignment Timing

The most common mistake is placing the partition-stability assertion in a @pytest.fixture(scope="session") or a Cucumber @BeforeAll hook that runs once at suite startup. A reassignment triggered mid-suite — by a broker rolling restart in the same cluster — invalidates that assertion without retriggering it. The fix is scenario-scoped guards, not session-scoped ones. This mirrors the broader problem of fixture teardown order silently corrupting shared state: the assumption baked into setup doesn't survive the full lifecycle of a long-running suite.

A second pattern that causes grief is using auto.offset.reset=earliest in test consumers without pairing it with an idempotency check. During a reassignment, a consumer may reset to earliest and replay messages the contract fixture already processed. The scenario passes because the final state is correct — but it got there via two state transitions instead of one, and any assertion on transition count or event emission will fail non-deterministically. Set auto.offset.reset=latest in contract test consumers and explicitly seed the offset in fixture setup using seek() so the starting position is deterministic regardless of reassignment history.

Myths About Contract Tests and Broker Topology Isolation

The most persistent myth is that running Pact tests against an embedded Kafka (via Testcontainers or kafka-junit) fully isolates contract verification from broker topology concerns. It does isolate schema and payload shape — but embedded brokers still perform rebalances when consumers connect and disconnect, and a test suite that spins up and tears down consumers rapidly will trigger rebalances that affect offset state. The embedded broker is not a freeze-frame; it's a live system with the same rebalance mechanics. Treat it accordingly: assert partition state in embedded environments with the same rigor you apply in staging.

A second myth is that contract testing and integration testing are substitutes in a BDD pipeline for microservices. They're not. Pact verifies the agreement; an integration test verifies the behavior under realistic conditions, including topology changes. The right model is layered: Pact contract tests in CI on every PR, integration tests against a shared Kafka cluster with realistic partition counts on merge to main. Teams that collapse these layers into one suite either over-trust Pact (and miss topology bugs) or under-trust it (and write redundant integration tests that duplicate contract assertions). Neither is efficient. If you're generating BDD steps with AI tooling, note that AI step generators frequently misclassify shared broker state as test preconditions, which compounds this layering problem.

Partition reassignment is a first-class distributed systems event, not an edge case — treat it as a named test condition in your BDD scenarios, not background noise. The next measurement worth adding once you have scenario-scoped partition guards in place is mean-time-to-detect on rebalance-induced failures: instrument your Kafka consumer with OpenTelemetry spans covering the rebalance lifecycle and correlate them against your Pact verification run timestamps in Grafana. That correlation will tell you whether your CI pipeline catches reassignment failures before they reach staging, or after.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.