Messaging Local Bench Specification¶
This document is the standard for comparing the relative cost of gRPC, the ZLink raw binding, and
the ZLink framework in the same format on a local development machine. The target languages are
the five dotnet, node, java, kotlin, and cpp, and a reference bench written in C is used
alongside them as the floor value. It doesn't represent a production environment's mesh, TLS, L7
load balancer, multi-node distribution, or network latency. The thing measured is messaging
from server process A to server process B (server-to-server), and the load is generated inside A
after it receives an HTTP trigger (§10). The HTTP call is the start signal, not a measured
operation.
This bench is a service-level comparison. It measures the cost actually paid when the same task is implemented with each stack. It doesn't ask whether the two stacks use the same mechanism internally. A difference that arises because a capability exists in only one stack isn't hidden; it's left in the result as it is. The question this document answers is "implementing this task with gRPC costs this much, implementing it with ZLink costs this much." "Which side is faster when the two stacks transmit the same way" isn't this bench's question.
1. Comparison Targets¶
1.1 Implementation Names¶
Three implementations are compared per language. <lang> is one of dotnet, node, java,
kotlin, cpp. The report table and the RESULT line use the same names.
| Implementation Name | Meaning |
|---|---|
grpc-<lang> |
That language's gRPC unary RPC |
zlink-<lang> |
The raw binding's ROUTER↔ROUTER TCP path, bypassing the framework |
zlink-framework-<lang> |
Framework RouteMesh request/send addressed directly to a node RID |
The default execution order is grpc-<lang>, zlink-<lang>, zlink-framework-<lang>. Since the
three implementations run the same pattern at the same payload size and the same active duration,
the table shows the three implementations side by side under one pattern.
1.2 The C Reference Bench¶
grpc-c and zlink-c from framework/bench/grpc/c are kept alongside as the floor
reference. There's no framework layer in C, so only these two implementations exist there. The
zlink-c request-backpressure value is the reference value for judging where each language's raw
binding stands (§7.2). The C reference bench stays client-driven; it isn't converted to the
server-driven model of §10, because what's borrowed from it is the per-request cost, not the place
where load is generated. That difference is noted next to the denominator in §7.2.
1.3 The ZLink Socket Axis¶
Both ZLink rows use ROUTER↔ROUTER and address the peer node directly by RID. This condition isn't optional; it's the contract that makes the judgement formula hold. Every cell has one server (B).
| Row | Socket Configuration |
|---|---|
zlink-framework-<lang> |
RouteMesh's direct node calls requestToNode / sendToNode, naming the server node's RID. RouteMesh connects as ROUTER↔ROUTER |
zlink-<lang> |
The raw binding's ROUTER↔ROUTER. The client side also creates a ROUTER and requests/sends by specifying the peer ROUTER's routing id |
zlink-c |
The same ROUTER↔ROUTER configuration as the raw binding row above |
%%{init: {'flowchart': {'nodeSpacing': 32, 'rankSpacing': 40, 'padding': 8, 'wrappingWidth': 180}, 'themeVariables': {'fontSize': '18px'}}}%%
flowchart LR
FC[framework RouteMesh ToNode client] <--> FS[framework node handler server]
RC[raw binding ROUTER] <--> RS[raw binding ROUTER]
This contract is needed because of the judgement formula in §7.2.
zlink-framework-<lang> / zlink-<lang> is the ratio for looking at the cost the framework layer
additionally requires. Only when both rows use the same socket pattern and the same way of naming
the target does this ratio isolate the framework layer. If the raw row is DEALER→ROUTER, or the
framework row sends by channel name through requestToChannel/sendToChannel (which includes
node selection), the division result contains the framework layer cost together with the socket
pattern and target selection difference, and that value isn't the framework layer cost. The gRPC
row is one channel attached 1:1 to one server, and the configuration above has the same shape.
Comparisons between ZLink models — DEALER→ROUTER versus ROUTER↔ROUTER, ToChannel (with node
selection) versus ToNode, ClientServer versus RouteMesh — are not items of this bench; they
belong to a separate ZLink model-comparison bench. This bench compares only the 1:1 request path
that has the same shape as gRPC.
The current implementation state is recorded alongside. Java and the .NET, C++, and Node framework
rows have moved to this configuration (Issue #13). Moving the raw paths and the C reference bench
client is separate work. The table above prescribes the configuration to be used for measurement,
and a value obtained before that change completes isn't a value that satisfies this specification.
2. Measurement Patterns¶
The initial scope uses only two payload sizes and three patterns. The default payload sizes are
1024 and 4096 bytes. The payload size is the full size of the protobuf bytes body or the raw
ZLink message body. The first 29 bytes are used as the measurement header, and the rest is filled
as the business payload area. The gRPC HTTP/2 frame, protobuf field overhead, ZLink envelope, and
ZMP header aren't included in this size.
| Pattern Name | gRPC | ZLink raw binding | ZLink framework | Interpretation |
|---|---|---|---|---|
request-serial |
unary Echo RPC |
raw request send and reply receive | node request call | Sends one request and sends the next only after the reply completes |
request-backpressure |
unary Echo RPC |
raw request send and reply receive | node request call | Puts no ceiling on incomplete requests. Submits continuously until admission backpressure |
send-saturation |
unary Command RPC, replying Empty |
raw one-way send submission | node send submission | Compares the command path, which has no reply payload |
The API names the three implementations use differ per language, but the contract is the same. In
Java, the framework node request is requestToNode(mesh, rid, message) and the node send is
sendToNode(mesh, rid, message) (§1.3). The per-language modules are in §8.
request-serial's throughput is decided by a single request's round-trip latency. This value is
meant to show the cost of a "process one at a time" usage pattern.
request-backpressure puts no application ceiling on the number of incomplete requests. Without
waiting for replies, the client submits requests continuously until it meets admission
backpressure, and resumes submitting when it is signalled to. The gRPC side has the same shape: it
submits until its own flow control stops it. Neither side gets an application ceiling.
In this pattern the incomplete depth is a measurement, not a setting. The depth a stack reaches is an outcome its design decides, so it is recorded as part of the result (§5.2).
This loop shape is not new; it is ported from an implementation already in the repository. The
reference implementation is bindings/node/perf/multi/perf_multi_socket_reqrep.ts, which submits
async requests continuously without waiting for a reply and then hands the turn to the completion
pump. All six rows use the same shape, and the gRPC side is treated the same way: unary calls are
submitted continuously rather than awaited one at a time.
This repository's perf convention already requires that inflight depth not be fixed artificially
and that requests be submitted continuously until admission backpressure (the request/reply client
items in doc/perf/PERF_MULTI_TEST_POLICY.md, and the fixed application window item in
doc/perf/PERF_POLICY.md). The fixed-window pattern (request-window) diverges from that
convention, so it is excluded from this bench. Behaviour at a fixed depth is an item of the ZLink
model-comparison bench.
send-saturation isn't averaged together with request/reply. This pattern exists to look at the
command family's relative cost separately.
2.1 The gRPC Counterpart for send-saturation¶
The gRPC side of send-saturation uses the current unary
Command(BenchPayload) returns (Empty) as it is. No client-streaming RPC is added, and no RPC is
added to the proto. The reasons this is the correct comparison for this bench are below.
- In a real service, a call that needs no response is implemented as a unary call with an
Emptyreply. Client-streaming is a form meant for uploads and bulk ingestion, and isn't how a service sends a single command that needs no response. - gRPC has no one-way call primitive. So it pays one round trip when handling this task. In a service-level comparison that cost is a value that belongs in the result.
- Inserting a different usage form on the gRPC side to even out the comparison would measure an implementation that real services don't use.
The reading rule for this cell is fixed alongside. This cell isn't described as a transport-speed difference. The correct sentence form is below.
When a command needs no response, gRPC pays a unary round trip while ZLink completes with a
one-way send. Under this condition the difference was N times.
3. Execution Conditions¶
- For each implementation (
grpc-<lang>,zlink-<lang>,zlink-framework-<lang>) run one source process A and one target process B. A has the HTTP trigger listener, a stats endpoint, and the client toward B (gRPC stub, raw ROUTER, framework node client). B has the echo (request) or receive counter (send) and a stats endpoint. §10 defines the roles, the trigger contract, and the cell order. - The local runner starts B then A for each cell and announces the phase start to A's trigger endpoint over HTTP. The runner itself generates no load.
- Only a loopback address (
127.0.0.1) is used. Ports use the per-language bands in §9. - Runs as a Release build.
- Runs a fixed-duration measured active window after warmup. The warmup length is set per language and the value used is recorded in the result (§8.2).
- The default payload size is
1024,4096bytes. - The
request-backpressurepattern has no incomplete-request ceiling setting. In that pattern depth is not a condition that is set but a result that is measured and recorded (§5.2). This bench has norequest_windowsetting. - The default send concurrency is
8. - gRPC and ZLink framework use the same protobuf DTO. The ZLink raw binding skips the framework but
puts the same shape on the wire: two parts, an envelope header part and a protobuf-encoded
BenchPayloadpart, with the 29-byte measurement header inside that protobufbytes body. This shape is not a choice; it is a precondition of the judgement. §7.2 formula 1 divideszlink-<lang>byzlink-c, so two rows with different wire shapes would divide two different experiments. The reference implementation isframework/bench/grpc/c,bench_zlink_client.cpp:14-16and:130-140. - ZLink uses a manual endpoint connection with no location store.
- Both ZLink rows use the ROUTER↔ROUTER configuration in §1.3.
- The ZLink raw binding's request echo endpoint and command receive endpoint are separated. In the command measurement, if request echo replies mix into the same socket, the one-way receive volume with no reply can't be observed correctly.
- TLS, compression, service mesh, gateway, and broker are not used.
- Languages aren't measured at the same time. Only one language is measured at a time.
The two ZLink rows differ in how many endpoints they use. The raw binding row separates its
request echo endpoint from its command endpoint and so uses two, while the framework row uses one
RouteMesh connection that carries both a request handler and a send handler. This difference is
intentional and contaminates no cell, because patterns are measured one at a time. While the
send-saturation cell is measured, only send traffic occurs on either row; while a request-family
cell is measured, only request traffic occurs on either row.
This reasoning depends on the premise that only one pattern is measured at a time. If a scenario that runs two patterns concurrently is added, the premise no longer holds and this endpoint configuration must be revisited.
The settle between cells follows this contract. After a cell completes, do not wait a fixed time. Poll the server's received count until it stops advancing, with an upper bound on that wait. If the count has not stopped advancing within the bound, record that fact and the observed drain time in the results, and mark the next cell that uses the same server as contaminated, excluding it from the tables and from every judgement. A contaminated cell is not measured and published. The observed drain time is recorded per cell in the results. This bound is not a formality; it is load-bearing. A bound set too small loses cells even on a healthy run, so it must be large enough that a healthy run never reaches it. The reference value is 30 seconds.
4. Output Format¶
The report table is grouped by pattern, with the implementation name shown on each row. The example follows the format below.
> Benchmarking current for request-backpressure...
Testing local:
| Implementation | Size | Throughput | Bandwidth | Lat.Mean(ms) | Lat.P95(ms) | Lat.P99(ms) | Client CPU | Client Mem | Server CPU | Server Mem |
|-------------------------|----------|------------------|--------------|--------------|--------------|--------------|------------|------------|------------|------------|
| grpc-dotnet | 1024B | 10.00 KOPS | 10.24 MB/s | 1.000 ms | 2.000 ms | 3.000 ms | 10.0% | 100.0 MB | 10.0% | 100.0 MB |
| zlink-dotnet | 1024B | 30.00 KOPS | 30.72 MB/s | 0.300 ms | 0.600 ms | 0.900 ms | 10.0% | 100.0 MB | 10.0% | 100.0 MB |
| zlink-framework-dotnet | 1024B | 20.00 KOPS | 20.48 MB/s | 0.500 ms | 0.900 ms | 1.200 ms | 10.0% | 100.0 MB | 10.0% | 100.0 MB |
The report and console output also leave a per-metric RESULT,current,... format alongside, for
easier handling by the perf runner. The fields of one row are in the following order.
An example follows.
RESULT,current,grpc-dotnet-request-backpressure,local,1024,throughput,10000.000
RESULT,current,zlink-dotnet-request-backpressure,local,1024,latency,0.300
RESULT,current,zlink-framework-dotnet-request-backpressure,local,1024,latency_p99,1.200
<scenario> is <implementation name>-<pattern name>, so the language is part of the name. For
example, Node's equivalent cell is zlink-framework-node-request-backpressure.
The metric value uses throughput, bandwidth, latency, latency_p95, latency_p99,
client_cpu_percent, client_memory_mb, server_cpu_percent, server_memory_mb. The raw value
of throughput is completions per second, and the table displays it divided into KOPS or
KMSG/s.
In the server-driven model (§10) client_* is the value of source process A and server_* is
the value of target process B. The metric names are kept for the aggregator and for older raw
files; the table header shows them as Source CPU and Target CPU. The per-cell raw JSON carries
the following in addition.
| Field | Meaning |
|---|---|
role |
source or target. A and B each write their own raw file and the runner merges them into one cell |
trigger |
The trigger request A received, kept as is. Fields: runId, cellId, pattern, payloadBytes, durationMs, warmup (the warmup length, in the unit the language harness records per §8.2), endpoint (A's trigger URL, the companion information of §7.1) and receivedAtUnixMs (the wall-clock time A received the trigger). All eight are required; the aggregator accepts no alias or default |
streams |
A's logical stream count and the per-stream in-flight ceiling (§10.3) |
target_stats |
Receive count, error count, and drain time the runner read from B's stats endpoint after settle |
5. Metrics¶
The required output is the metrics below. The throughput unit is separated to match the pattern's character.
| Metric | Meaning |
|---|---|
Throughput |
Throughput during the measured window. The table shows the value and unit together in one cell, e.g. 10.000 KOPS, 183.618 KMSG/s |
Bandwidth |
The transfer volume calculated from payload size and throughput. Shown as MB/s |
Lat.Mean(ms) |
Mean latency. For request/reply this runs from just before A's outbound call until the reply completes; for send it's the receive latency B calculates from the header |
Lat.P95(ms) |
p95 latency |
Lat.P99(ms) |
p99 latency |
Source CPU (client_cpu_percent) |
The CPU percentage source process A used during the active window |
Source Mem (client_memory_mb) |
Source process A's working set |
Target CPU (server_cpu_percent) |
The CPU percentage target process B used during the active window |
Target Mem (server_memory_mb) |
Target process B's working set |
request-serial and request-backpressure calculate KOPS based on the number of completions where an
echo reply came back. Here, 1 KOPS means 1,000 request/reply completions per second.
send-saturation calculates KMSG/s based on the number of messages the server received during the
active phase. Here, 1 KMSG/s means 1,000 messages per second. Since a ZLink send doesn't wait for
a reply, throughput isn't calculated from the client's submission call count alone.
5.1 The Source Saturation Rule¶
"Client" in this section means the process that generates the load, which in the server-driven
model is source process A. Source CPU is recorded for every cell. It isn't optional. A percentage alone cannot decide
saturation, so the number of cores used is recorded alongside the percentage. The percentage is
taken against all of the machine's logical cores. On a machine with 20 logical cores, a
single-threaded client that fully occupies one core still reads 5%, and no fixed percentage
threshold can catch that saturation.
Saturation is judged against the instrument and the ceiling that each language's harness declares. A cell whose declared instrument reaches 0.95 of the declared ceiling is marked as a saturated cell and is excluded from throughput ranking. In that cell the limit was set by the client runtime, not by the transport. A saturated cell's throughput is recorded only as "the value observed with this client configuration."
A harness declares what it measures, not only the ceiling, because the right instrument differs per language. The purpose of the rule is to catch "the client runtime, not the transport, set this limit", so the instrument has to measure the execution resource on which user code runs.
| Language | Declared instrument | Ceiling |
|---|---|---|
dotnet |
process cores used | the parallelism the harness declares |
node |
performance.eventLoopUtilization() from perf_hooks |
1.0 |
java, kotlin |
CPU of the submit threads (ThreadMXBean, jvm_thread_cores) |
the submit parallelism the harness declares |
cpp |
CPU of the application thread that runs submit and the completion drain (CLOCK_THREAD_CPUTIME_ID, submit_thread_cores) |
1 |
dotnet is the only language that declares process cores used.
Measurement shows why process cores are the wrong instrument for Node. The ZLink binding runs native I/O threads, so process CPU divided by elapsed time reads 1.3-1.4 cores on that client, and those threads run no user code. Against a ceiling of 1 every cell would be marked saturated even with an idle JS thread, and the mark would carry no information. What actually limits this client is the one JS thread on which user code runs, and event loop utilization measures exactly that.
Process CPU is not a usable saturation instrument for a ZLink client. That is why this section
makes the instrument a per-language declaration. Three languages reached the same conclusion for
three different reasons: in Node process CPU also counted the binding's native I/O threads, in Java
it counted GC and JIT threads, and in C++ it again counted the binding's I/O threads. Four of the
five languages declare something other than process CPU. The problem is not the threshold but
whether the two rows a judgement divides are commensurate. Measured in C++,
zlink-cpp request-backpressure reads 0.950 on its declared instrument but 1.90 in process cores, while
grpc-cpp in the same run reads 0.700 and 0.70 — effectively the same value. Judged on process
cores, that difference reports which row links Core rather than which row was client-bound.
The declared instrument and ceiling differ per language, so both are recorded in the cell record and in the report. A result that declares no instrument or no ceiling cannot have its saturation judged, and the fact that it could not be judged is recorded in the result. An older result that declares no instrument is read as having declared process cores used.
5.2 Incomplete Depth Is Recorded Per Cell¶
Request-family cells record the three values below for every cell. The three are not optional.
| Metric | Meaning |
|---|---|
peak_in_flight |
The maximum number of incomplete requests observed during the active window |
| Depth | The mean incomplete count calculated as throughput x mean latency (Little's law) |
abandoned |
The number of requests that did not complete by the drain bound |
In request-backpressure these three are the result. Since no depth is configured, the depth a
stack reaches is an outcome its design decides, and it goes into the table with the same standing
as throughput. Quoting throughput without the depth reached makes values obtained at different
depths read as values from the same condition.
Cells with a nonzero abandoned and cells with nonzero errors are not used for throughput
comparison. What such a cell reports is not a speed but the fact that some requests did not
complete. That fact is recorded as a correctness item, not a performance item.
6. The Measurement Payload Header¶
A 29-byte header with the same meaning as bindings/c/perf's metric header is placed at the front
of the measured payload. This header is part of the payload body, going at the front of the
protobuf bytes body or the raw ZLink message body.
| Offset | Size | Value |
|---|---|---|
| 0 | 4 | magic 0x5A4C4E4B (ZLNK) |
| 4 | 4 | run id |
| 8 | 1 | phase (0 warmup, 1 active) |
| 9 | 4 | payload size |
| 13 | 8 | sequence |
| 21 | 8 | send timestamp ns |
For request-serial and request-backpressure, the client validates the header that came back in the
reply payload, and calculates KOPS from the active-phase reply count. send-saturation has the
server read the header to calculate the active-phase message count and server-side receive
latency. This approach puts the measurement value inside the payload so a one-way throughput isn't
overstated by a client stopwatch alone.
This header layout is identical in every language. If a language used a different layout, its cells couldn't be placed side by side with the others.
7. Interpreting Results¶
This bench is a small local comparison tool.
7.1 Information Kept Alongside the Result¶
To put a performance-superiority statement into a result document or guide, keep the following information alongside it.
- CPU, OS
- That language's runtime version and gRPC library version
- Commit hash
- Payload size
- Warmup and active duration configuration
- gRPC and ZLink endpoint
- The depth reached (
request-backpressurepattern, §5.2) - A's logical stream count, per-stream in-flight ceiling, and trigger endpoint (§10)
- Per-cell
peak_in_flight, depth, andabandoned(§5.2) - The send concurrency value
- Source (A) CPU and whether the cell was saturated (§5.1), target (B) CPU
- The original result JSON
7.2 Judgement Between Layers¶
Performance judgment compares within the same ZLink layer, not against gRPC. The raw binding is
judged against the zlink-c request-backpressure result under the same conditions. If the raw binding
reaches 80% or more of the C result, the binding layer's baseline performance is judged as
passing. The framework is judged against the same language's raw binding result. If the framework
reaches 80% or more of the raw binding result, the framework's added cost is judged as passing.
at each of payload sizes 1024 and 4096:
zlink-<lang> / zlink-c >= 0.80 binding layer passes
zlink-framework-<lang> / zlink-<lang> >= 0.80 framework added cost passes
Both formulas are calculated separately per payload size. A language passes only when it satisfies
the criterion at both 1024 and 4096. A result that satisfies only one size isn't a pass, and
the per-payload values are always recorded as they are. A stack that holds the criterion at 1024
but degrades at 4096 has a real problem, and since the report shows both sizes anyway, a
per-payload gate exposes that problem at no added cost.
This criterion is applied to patterns where ZLink can send the next request without waiting for the
reply. Such a pattern is the sound judgement cell because gRPC unary Echo and a ZLink request
give the same guarantee, namely confirmation that the server processed the message.
request-serial is kept as a supplementary metric for looking at the round-trip latency of a
process-one-at-a-time usage pattern.
The reference pattern for judgement is request-backpressure. Whether a language passes is
decided by that pattern satisfying the criterion at both payload sizes.
The grounds for choosing the reference pattern this way are below.
- Both formulas are ratios of two rows measured under one condition, so that condition has to be one the numerator and the denominator can actually reach. A ratio computed under a condition that is not reached reports that condition rather than the layer cost.
- A fixed window is a condition no service imposes on itself. A low ratio under it can mean "the per-request cost is high" or "that depth was never reached", and because both causes appear as the same number, the ratio alone does not separate them.
request-backpressurecomputes the ratio with each stack at the depth its own design allows, so it compares what a caller submitting normally actually gets. That is what "the binding layer's baseline performance" has to mean.
What this choice does not solve is recorded with it. request-backpressure removes the fixing
of depth, not the difference in depth. A stack that stays at a low depth through its own
backpressure still produces a low ratio in this pattern, and the ratio alone still fails to
separate per-request cost from depth reached. So for either pattern, a ratio published without
the three depth values of §5.2 beside it is not published at all.
There is no fixed-window pattern in this bench. How a stack behaves at an externally chosen depth is an item of the ZLink model-comparison bench, and loss and non-completion observed there are recorded separately as correctness items.
The second formula represents the framework layer cost only when the socket configuration in §1.3 is observed.
The denominator of the first formula, zlink-c, comes from the client-driven bench (§1.2) while
the numerator comes from the server-driven model (§10). The two models generate load in different
places, so read the ratio as "the same per-request cost measured under a different model" and say
so in the report. The centre of the public comparison report isn't this ratio but the direct
grpc · zlink · zlink-framework comparison within one language; the ratio goes in an appendix.
7.3 Cross-Language Reading Rules¶
Absolute throughput isn't compared across languages. Placing grpc-node and grpc-java side by
side is a runtime comparison, not a ZLink comparison. If two languages' throughput differs in the
table, the cause is mostly the runtime and the gRPC library, and this bench doesn't separate that
cause out.
What can be read across languages is the ratio. zlink-framework-<lang> / zlink-<lang> is a value
each language calculated against itself, so it can answer "which language's framework layer is the
most expensive." A table holding a cross-language comparison places this ratio as its main entry.
7.4 Unit Alignment¶
Section 4 fixes the throughput unit as completions per second. Even when a runner records the value
at a different scale, the shared aggregator (framework/bench/grpc/tools/) recovers that scale from
bandwidth (which section 5 fixes as MB/s) and normalizes it, so comparison and judgement are
always made from the aggregator's output. The table a language client prints for itself is a
convenience for checking a single run directly, not a basis for judgement. Ratios are not calculated
by reading a runner's report directly.
Results are interpreted only as "faster/slower under this condition." No claim of general production performance or superiority across every payload is made.
8. Per-Language Configuration¶
8.1 Modules Used Per Language¶
| Language | gRPC library and server | Framework module | Raw binding | protobuf codec |
|---|---|---|---|---|
dotnet |
ASP.NET Core gRPC | framework/languages/dotnet/src/Zlink.Framework |
bindings/dotnet |
Zlink.Framework.Codecs.Protobuf |
node |
@grpc/grpc-js |
framework/languages/node/packages/framework |
bindings/node |
packages/framework-codec-protobuf |
java |
grpc-java | zlink-framework-core |
bindings/java |
zlink-framework-codec-protobuf |
kotlin |
grpc-kotlin coroutine stub | zlink-framework-kotlin |
bindings/kotlin |
Uses the same codec as Java |
cpp |
system libgrpc++ and grpc_cpp_plugin |
framework/languages/cpp/framework |
bindings/cpp |
zlink::framework_codec_protobuf |
A's HTTP trigger listener uses each language's standard HTTP server (ASP.NET Core minimal API, Node
http, the JDK HttpServer, C++ framework HTTP hosting). The listener sits outside the measured
path and enters no cell's cost.
Kotlin is excluded from the full matrix and only supplementary cells are measured (§10.5). Kotlin uses a suspend interface on the ZLink side, so the gRPC side uses the grpc-kotlin coroutine stub as well. Only when the coroutine stub can't be used is the grpc-java blocking stub used, and that reason is recorded in the result.
C++ uses the libgrpc++ installed on the system. gRPC isn't built through vcpkg. This machine's
version is 1.51.1, and since it's an old version it must be recorded in the result.
8.1.1 Per-message Payload Work¶
For every request/send submission, each supported client creates a body containing the 29-byte
measurement header and a message object matching the BenchPayload schema, then serializes it
with that language's protobuf runtime. Raw uses constant
wire envelope header parts. Received payloads are parsed with protobuf; request replies serialize
a typed payload and preserve the request's measurement header and timestamp. Sends have no
payload reply; gRPC Command serializes its existing Empty response.
| Language | Raw request/reply serialization and receive parsing | gRPC | Framework |
|---|---|---|---|
| Java | A fresh BenchPayload: toByteArray / parseFrom |
grpc-java serializes/parses the same generated type | ZLinkProtobufCodec serializes/parses the same generated type |
| .NET | A fresh BenchPayload: WriteTo / Parser.ParseFrom |
Google.Protobuf based gRPC serializes/parses the same generated type | Protobuf codec serializes/parses the same generated type |
| C++ | A fresh BenchPayload: protobuf serialization / ParseFromArray |
libgrpc++ serializes/parses the same generated type | Protobuf codec serializes/parses the same generated type |
| Node | A fresh BenchPayload DTO through the existing proto-loader serializer / deserializer |
Uses the same proto-loader protobuf serializer / deserializer | Connects the same proto-loader serializer / deserializer to the schema envelope codec |
| C binding | C++ driver creates a generated BenchPayload per message and uses existing libprotobuf |
Same generated type and libprotobuf | No C framework row |
The C binding bench uses .cpp drivers and already depends on protobuf. Raw shares that dependency;
there is no manual protobuf framing exception. This table compares payload work, not transport calls,
envelope processing, or buffer copy counts. Java raw and gRPC use ByteString.copyFrom, while
Framework retains its existing UnsafeByteOperations.unsafeWrap path.
Run tools/test_raw_wire.sh to compare the frozen pre-change 29-byte payload dump with protobuf
output, check length varint boundaries, and verify request reply body preservation.
Results including raw serialization cost form a new baseline. Do not compare performance trends
with framework/raw ratios from raw drivers that bypassed serialization. A lower raw throughput
can raise this ratio; that reflects corrected measurement conditions, not improved framework performance.
8.2 Values Set Per Language That Must Be Recorded¶
The three values below are set differently per language. The goal isn't to make them identical but to leave the values used in the result.
| Item | Reason |
|---|---|
| Warmup length | The JVM reaches steady state only after JIT warmup finishes. Forcing the same warmup as .NET would measure an unwarmed runtime |
| gRPC server configuration | The default server implementation differs per language. The gRPC side is left at each language's default configuration, and that configuration is recorded in the result |
| Runtime and gRPC library version | The SDK version, runtime version, and gRPC library version are recorded in the cell's raw output and in the report |
9. Port Bands¶
One implementation needs four ports (A trigger, A stats, B endpoint, B stats; the raw binding needs five because B's request and command endpoints are separate) and each language has three implementations, so every language gets a band of 20. The meaning of an offset within a band is the same in every language.
| Language | Band | grpc A trigger/stats, B endpoint/stats | zlink raw A trigger/stats, B request/command/stats | framework A trigger/stats, B endpoint/stats |
|---|---|---|---|---|
dotnet |
5200-5219 | 5200/5201, 5202/5203 | 5205/5206, 5207/5208/5209 | 5212/5213, 5214/5215 |
node |
5220-5239 | 5220/5221, 5222/5223 | 5225/5226, 5227/5228/5229 | 5232/5233, 5234/5235 |
java |
5240-5259 | 5240/5241, 5242/5243 | 5245/5246, 5247/5248/5249 | 5252/5253, 5254/5255 |
kotlin (supplementary, §10.5) |
5260-5279 | 5260/5261, B uses the java band 5242/5243 | none | 5272/5273, B uses the java band 5254/5255 |
cpp |
5280-5299 | 5280/5281, 5282/5283 | 5285/5286, 5287/5288/5289 | 5292/5293, 5294/5295 |
| C reference (client-driven) | 6200-6219 | 6200/6201, 6202/6203 | 6205/6206, 6207/6208/6209 | none |
+16 to +19 of each band are reserved. The runner checks that its own band's ports are free
before starting a measurement. If one is in use it stops instead of moving to another port. Moving a
port would make the endpoint recorded in the result disagree with the endpoint actually used. The
Kotlin supplementary cells reuse Java's B, so they never run at the same time as a Java measurement
(one language at a time).
10. The Server-Driven Execution Model¶
10.1 Roles¶
| Role | Processes | What it does |
|---|---|---|
| trigger client | 1 (the runner) | Sends HTTP POST /bench/start to A's trigger endpoint. Generates no load and counts no result |
| source A | 1 per implementation | On trigger, repeats request or send toward B over the logical streams of §10.3, aggregates completions, latency, and errors itself, and writes the cell's raw JSON. Exposes progress on its stats endpoint |
| target B | 1 per implementation | For request, echoes the same payload; for send, reads the header and counts received messages and receive latency. Exposes the values on its stats endpoint |
The A·B pairs of the three implementations are independent process pairs. Even within one language only one pair is measured at a time.
10.2 Trigger Contract¶
The trigger and admin contract of the common perf specification
(framework/doc/framework/common/perf/README.en.md §4.2, §5.1, §16) is used as is. Request and
response are JSON and the five languages use the same fields.
POST http://127.0.0.1:<A trigger>/bench/start
{ "runId": "...", "cellId": "...", "pattern": "request-backpressure",
"payloadBytes": 1024, "phase": "warmup" | "active",
"durationMs": 5000, "requestWindow": 100, "sendConcurrency": 8 }
→ 200 { "accepted": true, "runId": "...", "cellId": "...", "phase": "active", "startedAt": <monotonic ns> }
- A phase starts once. A duplicate trigger with the same
runId,cellId, andphasereturns the same start acknowledgement and generates no second load. - The trigger only carries pattern, payload, duration, window, and concurrency; it doesn't change the specified values. The defaults are the values §3 sets.
GET http://127.0.0.1:<A stats>/bench/statsandGET http://127.0.0.1:<B stats>/bench/statsreturn the phase, submitted/completed/error counts, received count, and current in-flight. They are what settle (§3) polls.- The HTTP round trips of trigger and stats are not measured operations and don't enter
throughputorlatency.
10.3 Logical Streams and the Pattern Mapping¶
A runs independent load flows toward B as logical streams. A pattern is expressed as a stream count and a per-stream in-flight ceiling.
| Pattern | Streams | In-flight per stream | Meaning |
|---|---|---|---|
request-serial |
1 | 1 | Send one request, then the next after the reply |
request-backpressure |
1 | No ceiling | Submit until admission backpressure is met |
send-saturation |
send_concurrency (default 8) |
1 (until send completion notification) | The command path with no reply |
The stream count and per-stream in-flight are recorded in the cell's raw file (§4). How a language harness implements a stream (thread, task, coroutine, event loop) differs per language and is left in the result as in §8.2.
10.4 Cell Order¶
- The runner checks that its language band (§9) is free.
- Start the implementation's B and wait until its stats endpoint answers (30-second limit).
- Start the implementation's A and wait until A connects to B and reports route ready (30-second
limit). If A's stats say
ready=false, the cell isn't started and that fact is recorded. phase=warmuptrigger → wait until A finishes warmup and reportsphase=idleon stats.phase=activetrigger → A closes the measured window afterdurationMs.- Settle: poll B's (and A's) stats until the received and completed counts stop growing (30-second limit, the contamination rule of §3 unchanged).
- A writes the cell's raw JSON under
log/<lang>/<stamp>/and emits theRESULTlines. The runner merges B's stats into the same JSON astarget_stats. - Stop A and B. The next cell starts with a fresh process pair (a process is never reused across cells, so a previous cell's residual state can't enter the next one).
The gRPC implementation's A has the same trigger listener and runs as many unary stubs toward B as there are streams. The gRPC server configuration stays at the language default and is recorded (§8.2).
10.5 Kotlin Supplementary Cells¶
Kotlin shares the binding, server, and codec with Java, so it's excluded from the full matrix.
Instead, two supplementary cells that show the cost of the Kotlin call layer — grpc-kotlin
(coroutine stub) and zlink-framework-kotlin (suspend calls) at request-backpressure @1024 — are placed
next to the Java rows. Only A is Kotlin; B is the Java binary on the Java band of §9, as is.