Skip to content

15. E2E Testing — Verifying the Whole System with a Client

Guide Home | Previous: 14. Picking a Sample — Start with the Example Closest to Your Problem | Next: 16. Options — Setting List And Defaults

View in another language — C#/.NET · C++ · Java · Kotlin · Node/TypeScript

This chapter has no spec document that owns its contract. That's because it covers how to build tests in your own system. What each sample verifies is defined by the common sample document. The connector's formal API surface is owned by the per-language Stream Connector public contract. This chapter covers how to build E2E tests in your own system.

0. Where E2E Testing Is Needed

No matter how tightly you write handler unit tests, some things stay unverified: whether registration actually took effect, whether routing between two nodes is correct, whether a push reaches other participants in the room. These can only be verified by starting real processes and checking over a real connection.

At this point, teams usually implement a separate test-only client, rewriting the code to open a socket, assemble frames, and wait for a response for every scenario. ZLink doesn't need that work. The client library your real users use is itself the verification tool. An E2E test comes down to just this much code.

// A real connection
co_await client.connect ().async ();
// A real request
auto auth = co_await client.request (authenticate_req_t{actor_id})
              .async<authenticate_res_t> ();
// Confirms a real push arrived
auto push = co_await other.wait_for<player_joined_notify_t> ().async ();
ensure (push.payload.actor_id == auth.player.actor_id);

Because the connector itself provides the wait functions verification needs, like wait_for, you don't implement a separate test harness. Every sample in this repository is verified this way.

Distinguish what E2E does and doesn't cover. E2E confirms things like registration, routing, push, and lifecycle — items that only surface when multiple processes run together. Branches or calculations inside a handler are far faster and more precise to verify with a unit test, so they don't belong in E2E.

1. The Libraries Used for Verification

The two libraries used for verification don't overlap in role.

Zlink.HttpClient Zlink.Stream.Connector
What it verifies The management/gateway HTTP API A STREAM server node
When to use it Things that finish with a single request-response exchange, like creating a room, querying, or admin commands Things that require a live connection, including confirmation of server-initiated pushes
Representative call Post(...).Body(...).Fetch<T>() connect · request · wait_for · ExpectNone

Most scenarios chain the two together — create a target over HTTP, then connect to STREAM using the endpoint returned in that response.

// Step 1 -- create a room through the gateway API.
auto api = zlink::http_client::client_builder_t (options.api_url)
             .timeout (options.http_timeout)
             .build ();
// fetch returns the deserialized body as-is.
auto room = api.post ("/games")
              .body (create_game_http_req_t{options.game_name})
              .fetch<create_game_http_res_t> ();

// Step 2 -- open a real-time connection to the endpoint the response gave us.
zlink::stream_connector::connector_options_t connector_options;
connector_options.endpoint = room.play_endpoints[0];
connector_options.connect_timeout = options.stream_timeout;
connector_options.request_timeout = options.stream_timeout;
// Console scenarios use the automatic pump.
connector_options.dispatch_mode = zlink::stream_connector::dispatch_mode_t::immediate;
auto client = zlink::stream_connector::connector_factory_t::create (connector_options);

When DispatchMode is Immediate, the connector handles receiving on its own, so the scenario code never runs a separate pump. Environments that must pump manually to match a frame loop, like a game engine, are covered by the Stream Connector guide.

Each library's guide covers its full usage.

  • The HTTP Client guide — request construction, body, auth/TLS, retry, and error handling, across 13 chapters
  • The Stream Connector guide — per-runtime integration (Unity, Godot). Server-side STREAM registration is covered by 09-stream.

2. Verification Functions and Usage

Most scenarios are expressed with the verification functions the connector provides.

What's verified Function used
Send a request and check the response Request(req) — the response type is specified on the terminal
Confirm a push the server sends first arrives WaitFor<TNotify>()
Confirm a push does not arrive ExpectNone<TNotify>().Within(window)
Confirm pushes arrive in a fixed order WaitForSequence<TNotify>().Expect(...).Expect(...)
Confirm a request fails ExpectFailure(...)

The terminal call follows the language — .NET uses Async, C++ uses async, Java/Node use submit, and Kotlin uses await (Async Execution Policy).

Value comparison uses Ensure(condition, message). The message is required, and on failure the scenario ends with an exception carrying that message.

Confirming a Push Arrives

Specify a condition with Where(...) to wait for the first message matching that condition. Other, nonmatching pushes may arrive without affecting the scenario.

auto joined = co_await client1.wait_for<player_joined_notify_t> ()
                .where (&player_joined_notify_t::actor_id, options.o_actor_id)
                .async ();
ensure (joined.payload.mark == tictactoe_marks_t::o);

Confirming a Push Doesn't Arrive

You can't confirm something never arrives without an observation window, so Within(...) must be specified. Omitting it is an error.

// The player who just joined shouldn't receive their own join notification.
co_await client2.expect_none<player_joined_notify_t> ()
  .within (std::chrono::milliseconds (250))
  .async ();

Confirming Push Order

In a flow where state changes in stages, the contract isn't whether something arrives but its order.

auto status_sequence =
  co_await customer.wait_for_sequence<delivery_status_notify_t> ()
    .expect ([&] (const auto &m) { return m.delivery_id == delivery_id
                                          && m.status == delivery_status_t::assigned; })
    .expect ([&] (const auto &m) { return m.delivery_id == delivery_id
                                          && m.status == delivery_status_t::accepted; })
    .expect ([&] (const auto &m) { return m.delivery_id == delivery_id
                                          && m.status == delivery_status_t::picked_up; })
    .expect ([&] (const auto &m) { return m.delivery_id == delivery_id
                                          && m.status == delivery_status_t::delivered; })
    .timeout (customer.options ().wait_timeout)
    .async ();

Confirming a Request Fails

Whether a request with no permission or an out-of-order request gets rejected is also part of the contract. Verifying only the success path leaves this path unverified.

// Can't open a conversation before authenticating.
bool failed = false;
try {
    co_await agent.request (open_conversation_req_t{"unauthenticated"}).async<open_conversation_res_t> ();
} catch (const zlink::stream_connector::stream_error_t &error) {
    failed = error.code == zlink::stream_connector::error_code_t::remote_error;
}
ensure (failed);

3. How to Handle Waiting for a Message

Most E2E flakiness has the same cause. You act first, then start waiting, and miss a push that arrived in between.

Reverse the order. Register the wait first, then run the action that triggers that push.

// Register the wait first -- don't co_await it yet.
auto status_sequence_task =
  customer.wait_for_sequence<delivery_status_notify_t> ()
    .expect ([&] (const auto &m) { return m.delivery_id == delivery_id
                                          && m.status == delivery_status_t::assigned; })
    .timeout (customer.options ().wait_timeout)
    .async ();

// Then run the action that triggers the push.
auto created = http.post ("/deliveries")
                 .body (create_delivery_req_t{
                   delivery_id, "customer-1", "Kitchen 12", "Customer Lobby"})
                 .fetch<create_delivery_res_t> ();

// Receive the result last.
auto status_sequence = co_await std::move (status_sequence_task);

If multiple clients need to confirm the same event, register a wait for each and receive them together with Task.WhenAll.

// Bingo -- once both players have joined the room starts, and both clients get the same push.
auto client1_started = client1.wait_for<bingo_game_started_notify_t> ().async ();
auto client2_started = client2.wait_for<bingo_game_started_notify_t> ().async ();

co_await std::move (client1_started);
co_await std::move (client2_started);

Don't use Sleep to line up timing. Express every wait through the timeout on wait_for/ExpectNone/WaitForSequence. Sleep fails on slow hardware and wastes time on fast hardware.

4. A Complete Scenario Example

The TicTacToe sample is the shortest. Create a room over HTTP → both players connect and authenticate → confirm the join push → make a move → confirm the opponent observes that move, in that order.

task_t<void> run (const tictactoe_client_options_t &options)
{
    // 1. Create a room through the gateway API and get the endpoint to connect to.
    auto api = zlink::http_client::client_builder_t (options.api_url)
                 .timeout (options.http_timeout)
                 .build ();
    auto room = api.post ("/games")
                  .body (create_game_http_req_t{options.game_name})
                  .fetch<create_game_http_res_t> ();
    ensure (room.play_endpoints.size () >= 2);

    // 2. Connect the two players to different Play nodes -- this verifies routing between nodes.
    auto client1 = create_stream_client (room.play_endpoints[0], options);
    auto client2 = create_stream_client (room.play_endpoints[1], options);

    // 3. Whoever connects first authenticates and enters the empty room.
    co_await client1.connect ().async ();
    co_await client1.request (authenticate_req_t{options.x_actor_id}).async<authenticate_res_t> ();
    // Register wait -> send -> receive (see §3)
    auto join1 = co_await join_game (client1, room.room_id);
    ensure (join1.state.status == tictactoe_status_t::waiting_for_players);

    // Being alone in the room, their own join notification shouldn't come back to them.
    co_await client1.expect_none<player_joined_notify_t> ()
      .within (std::chrono::milliseconds (250))
      .async ();

    // 4. Once the second player joins, the room starts and a push reaches the first player.
    co_await client2.connect ().async ();
    co_await client2.request (authenticate_req_t{options.o_actor_id}).async<authenticate_res_t> ();
    auto join2 = co_await join_game (client2, room.room_id);
    ensure (join2.state.status == tictactoe_status_t::in_progress);

    // 5. Making a move -- the response and the push delivered to the opponent should point to the same state.
    auto move = co_await client1.request (place_mark_req_t{0}).async<place_mark_res_t> ();
    auto saw_move = co_await client2.wait_for<game_state_notify_t> ()
                      .where ([] (const auto &m) { return m.state.last_move_cell == 0; })
                      .async ();
    ensure (saw_move.payload.state.board == move.state.board);
}

Choose verification points by this rule. Don't just check a request's own response — also check whether another client observes the same fact. Making the result that actually reaches the user, not server-internal state, the contract, is the point of E2E.

5. Verifying with Multiple Clients

A single scenario can create several clients. Splitting roles verifies contracts a single client can't confirm.

  • Two players — whether one's action reaches the other, and conversely, that it doesn't reach themselves
  • A spectator — whether a notification reaches a non-participant connection, and conversely, that a participant-only notification doesn't
  • Two connected to different nodes — whether routing and location resolution between nodes actually work
// The join completion arrives as a client push -- register the wait before the one-way send.
task_t<join_game_notify_t> join_game (auto &connector, const std::string &room_id)
{
    auto completion = connector.wait_for<join_game_notify_t> ().async ();
    co_await connector.send (join_game_msg_t{room_id}).async ();
    co_return (co_await std::move (completion)).payload;
}

The Bingo sample uses this composition as-is — it brings together two players and one spectator, and even confirms the win notification is delivered only to the spectator.

6. Run Scripts and Success Criteria

The run script is responsible for starting the server, running the client, and cleaning up afterward.

start_server play-a "$PLAY_BIN" --config="$CONFIG_DIR/play-a.json"
start_server play-b "$PLAY_BIN" --config="$CONFIG_DIR/play-b.json"
start_server api-a  "$API_BIN"  --config="$CONFIG_DIR/api-a.json"

# Wait until the port opens. Doesn't use sleep.
wait_port play-a "$PLAY_A_ROUTE_ENDPOINT"

"$CLIENT_BIN" --config="$CONFIG_DIR/client.json" >"$LOG_DIR/client.log" 2>&1

RUN_SUCCEEDED=1

The script follows these rules.

  • The client's exit code is the success criterion. Under set -e, if the client exits with an exception, the script fails at that point too. No separate judgment logic is implemented.
  • Wait on a condition, not sleep. Server startup is confirmed by whether the port is open; async post-processing, by whether a specific line appeared in the log.
  • Clean up with trap so that a scenario failing partway through doesn't leave started processes, temp directories, or containers behind to affect the next run.

Even if the client passes, also check that the server logs have no errors. Sometimes the client observes normal behavior while the server records a dispatch error.

if grep -R -q "dispatch-error" "${LOG_DIR}"; then
  echo "Unexpected dispatch-error in sample logs." >&2
  exit 1
fi

7. Common Problems

  • A push isn't received, causing intermittent failure → check that the wait was registered before the action (§3). Starting the wait afterward misses a push that arrived in between.
  • ExpectNone ends in an error → Within(...) wasn't specified. You can't confirm something never arrives without an observation window, so the window is required explicitly.
  • wait_for returns a different message → you waited on type alone, with no condition. Narrow it with Where(...) to the event this scenario is actually waiting for.
  • It passes locally but fails only in CI → check for remaining timing dependencies implemented with sleep. Express every wait through a wait function with an explicit timeout.
  • The client passes but the server log has an error → the script doesn't check server logs for errors (§6).
  • It connects but the push never arrives → in an environment that needs manual pumping, like engine integration, dispatch wasn't run (see the Stream Connector guide).