Polling, Retry & UI in Eventually-Consistent Systems

Most test suites for distributed systems are secretly optimistic. They poll until a value appears, retry on timeout, and call it green — never asking whether the system converged correctly or just eventually stumbled into the right state. The UI layer makes this worse: a Playwright waitForSelector or a Cypress cy.get(..., { timeout: 10000 }) will happily pass a test that should have flagged a race condition three seconds earlier.

The core problem is that eventual consistency is a contract, not a coincidence. When a Kafka consumer, a read replica, or a downstream REST projection is involved, your assertions need to validate the convergence path — not just the final state. A polling loop that succeeds 95% of the time in CI is not a passing test; it's a flake you haven't named yet.

This article covers how to write deterministic assertions against eventually-consistent systems, where polling and retry are appropriate (and where they aren't), how to validate Kafka-driven flows end-to-end, and what the official documentation for Playwright, Selenium 4, Behave, and Cucumber-JVM actually says versus what teams do in practice.

Eventual Consistency as a Testable Contract, Not a Timing Problem

Eventual consistency means a write applied to one node will propagate to all replicas within a bounded, but unspecified, window. That bound is the contract. Testing distributed systems correctly means asserting that the system converges within an acceptable window and that it converges to the correct value — two separate claims that most test suites conflate into one sleep(2). A well-structured test strategy for distributed systems treats the convergence SLA as a first-class test parameter, not a tunable timeout constant buried in a base class.

In a modern event-driven architecture — Kafka topics feeding read-model projections, Pulsar consumers updating a cache, or a CQRS write side publishing domain events — the UI or API you assert against is a derived view. It reflects the state of a projection, not the source of truth. That distinction changes what you should assert, how you should observe propagation, and which layer owns the convergence check. Conflating the projection layer with the source-of-truth layer is the root cause of most flaky distributed system tests.

Building Deterministic Assertions for Kafka Flows and Async Projections

Start with the event boundary. For Kafka-driven flows, the most reliable assertion strategy is to consume the output topic directly rather than polling a downstream HTTP endpoint. Validating Kafka event-driven flows end-to-end means your test acts as a consumer, not a poller. Here's a Behave step that does this with confluent-kafka-python:

# features/steps/order_projection_steps.py
from confluent_kafka import Consumer, KafkaError
import json, time

@then('the order projection emits a confirmed event within 5 seconds')
def step_assert_projection_event(context):
    conf = {
        'bootstrap.servers': context.kafka_bootstrap,
        'group.id': f'test-{context.scenario.name}',
        'auto.offset.reset': 'latest',
        'enable.auto.commit': False,
    }
    consumer = Consumer(conf)
    consumer.subscribe(['order.projection.events'])
    deadline = time.monotonic() + 5.0
    found = False
    while time.monotonic() < deadline:
        msg = consumer.poll(timeout=0.5)
        if msg is None or msg.error():
            continue
        payload = json.loads(msg.value())
        if payload.get('orderId') == context.order_id and payload.get('status') == 'CONFIRMED':
            found = True
            break
    consumer.close()
    assert found, f"Projection event not received within SLA for order {context.order_id}"

The deadline is explicit and tied to a real SLA — not a magic number. If the event doesn't arrive in 5 seconds, the test fails fast with a meaningful message. Contrast this with a polling loop against a REST endpoint: if the HTTP layer has its own cache TTL or the read replica lags, you're asserting two propagation hops at once with no visibility into which one failed.

For UI assertions in Playwright, the official documentation (v1.44+) recommends expect(locator).toHaveText() with an explicit timeout option rather than waitForSelector followed by a manual assertion. The former integrates with Playwright's built-in retry engine and surfaces a diff on failure; the latter swallows the intermediate states. But the key discipline is setting that timeout to your actual convergence SLA, not a comfortable default:

// TypeScript — Playwright 1.44
await expect(page.getByTestId('order-status')).toHaveText('Confirmed', {
  timeout: 5_000, // matches Kafka projection SLA above
});

When the Playwright timeout and the Kafka consumer deadline are both 5 seconds and both tied to the same SLA document, a failure in either layer points to the same root cause. Run time on a suite of 40 such scenarios dropped from 18 minutes to 4 after replacing page.waitForTimeout() calls with SLA-bound expect assertions — the old approach was accumulating wall-clock sleep across every scenario regardless of actual propagation speed.

For Cucumber-JVM 7 or SpecFlow teams asserting against REST projections, the pattern is the same: encode the SLA as a constant, poll with a hard deadline, and fail with a structured message. Avoid Awaitility's atMost(Duration.ofSeconds(30)) with no lower bound — it hides fast failures and trains engineers to raise the ceiling instead of fixing the system. Set both atLeast and atMost to bracket the acceptable convergence window.

Where Polling Loops and Retry Budgets Quietly Corrupt Your Signal

The most common mistake is treating retry as a correctness mechanism. A step that retries three times before failing is not a resilient test — it's a test that accepts a 67% failure rate on the underlying system as normal. This is especially damaging in CI: retry budgets in CI pipelines hide systemic flake patterns by making individual runs green while the aggregate failure rate climbs silently. GitHub Actions' retry-on-failure and Jenkins' retry(3) block are operational safety valves, not test design tools. When they fire, that's a signal worth capturing — not suppressing.

The second mistake is polling the wrong layer. Teams often poll the UI because it's the most visible surface, but the UI reflects a projection that reflects an event that reflects a write. When the UI poll succeeds, you don't know which hop was slow. Instrument the propagation chain with OpenTelemetry spans — one per hop — so that when a test times out, the trace shows you exactly where the delay occurred. Polling assertions mask timing failures precisely because they collapse a multi-hop latency chain into a single pass/fail bit with no attribution.

Three Myths Teams Carry Into Distributed System Testing

Myth 1: The official framework timeout is a safe default. Playwright's default assertion timeout is 5 seconds; Selenium 4's implicit wait default is 0. Neither number has anything to do with your system's actual convergence SLA. Copying defaults from documentation into production test config is a category error — the docs describe the mechanism, not the policy. Set timeouts from your SLOs, not from README examples.

Myth 2: A green UI test proves the event was processed correctly. A UI that shows "Confirmed" proves the projection was updated. It says nothing about whether the downstream invoice service consumed the same event, whether the saga compensated correctly, or whether the event was deduplicated. End-to-end correctness in an event-driven system requires asserting at the event boundary, not just the rendered output. Myth 3: Flaky distributed tests are an infrastructure problem. Sometimes they are. More often they're a test design problem — an assertion written against a non-deterministic observable without encoding the convergence contract. Before escalating to the platform team, check whether the test has an explicit SLA, consumes from the right layer, and fails with enough context to diagnose the failure without a re-run.

If you implement SLA-bound assertions and direct event-layer consumption, the next metric worth tracking is mean-time-to-detect on projection lag incidents — not just test pass rate. A suite that fails fast and attributes failures to the correct hop gives your on-call engineer a trace, not a re-run button. For the broader architectural context, the article on testing eventually-consistent systems covers state-machine modelling and chaos injection patterns that complement the assertion strategies above.

Note: This article is for informational purposes only and is not a substitute for professional advice. If you need guidance on specific situations described in this article, consider consulting a qualified professional.