Observability¶
Spec table of contents · Next: 01. Runtime Status and Operational Diagnostics
1. What Can Be Observed¶
The application and the operator observe what one process is doing right now, at four layers, using this topic's four documents. The complete state at this moment (whether the Host can accept new work, and whether RouteMesh — the scope in which multiple nodes exchange Channel messages — as well as ClientServer and automatic fanout are each ready) is confirmed by query, and the moment that state changes is observed via a change stream. Throughput, wait, failure, and current counts that accumulate over time are collected as metrics, and how far one message progressed through processing and which other message shares its originating cause are traced using traces and the two identifiers.
The four documents each own a different observation unit and don't overlap — the ownership boundary per observation unit is defined by §2. These records don't change message delivery/completion guarantees, routing, handler distribution, or lifecycle decisions. The application, operational tooling, and the provider each intervene differently.
| Party | What it does in this topic |
|---|---|
| Application | Configures the record level (diagnostics level), sampling ratio, and whether to record message size, and queries or observes status by registered name. |
| Framework | Builds status/metric/trace records at each processing boundary and delivers them through standard paths. Creates, propagates, and restores correlation/flow identifiers. |
| Provider | Configures the structured logger, metric exporter, and telemetry provider. Delaying processing or changing the result is forbidden. |
| Operational tooling | Doesn't intervene inside an unregistered process, and judges process state using only public query/subscription and structured log. |
2. Documents in This Topic¶
| Document in this topic | Ownership scope |
|---|---|
| 01. Runtime Status and Operational Diagnostics | The complete status at a specific point in time, the status-change stream, and structured-log identifiers |
| 02. Runtime Metrics | The name/kind/unit/label of metrics accumulated and collected over time |
| 03. Message Flow Tracing | The progress record (trace) of one message and its attributes and record level |
| 04. Request Correlation | The creation, format, propagation, ownership, and lifetime of correlation_id, flow_id, flow_origin |
The results of individual host operations (relocation, shutdown) aren't owned by this topic but by Host Relocation and Shutdown.
3. Find by Question¶
| Question | Section with the answer |
|---|---|
| How does an operator confirm at once whether the whole process can accept new work right now | 01. Runtime Status and Operational Diagnostics "3. Host State — Values Read at Once" |
| What confirms the readiness of each of RouteMesh, ClientServer, and automatic fanout | 01. Runtime Status and Operational Diagnostics "5. Topology State — RouteMesh, ClientServer, Automatic Fanout" |
| What should be observed to avoid missing the moment state changes, and what happens if the observer is slow | 01. Runtime Status and Operational Diagnostics "6. Observing State Changes — Sequence and the Complete Status" · "7. When the Observer Is Slow — Source, Coalescing, and the Lost-Update Count" |
| How to query the current location of this Actor/Spot using an operational tool | 01. Runtime Status and Operational Diagnostics "8. Querying an Object's Current Location" |
| Where to find why a state changed | 01. Runtime Status and Operational Diagnostics "9. Structured Log" |
| Which metrics to collect to see throughput, wait, failure, and current counts on a dashboard | 02. Runtime Metrics "2. Naming and Aggregation Rules" |
| What guarantees the same instrument can be seen on the same dashboard/alert in every language | 02. Runtime Metrics "2. Naming and Aggregation Rules" · "10. Label Cardinality" |
| What numbers show how long one relocation took and where it stalled | 02. Runtime Metrics "8. Host Relocation and Shutdown" |
| How to trace how far one message got in processing and where it failed | 03. Message Flow Tracing "2. Which Processing Stages Are Recorded" |
| How much it costs to turn this flow record on and off, and whether it's really zero cost when off | 03. Message Flow Tracing "5. Changing the Record Level at Runtime and the Cost Rule" |
| What links a request and its reply | 04. Request Correlation "2. The Role of the Two Identifiers" |
| How to determine whether several messages started from the same cause | 04. Request Correlation "5. Propagation Rule" |
| Whether personal information or payload goes into these identifiers | 04. Request Correlation "8. Observability and Privacy" |
| What an operator turns on and reads first when investigating an intermittent failure | §4 The Order for Chasing an Intermittent Failure |
4. The Order for Chasing an Intermittent Failure¶
When chasing an intermittent failure, first turn on and read the existing message tracking and file log. Adding a new temporary log and repeating the reproduction is forbidden. That approach forces you to rerun the whole reproduction cycle just to see one exception, and it misses the cause even when it's already stamped in the existing log.
4.1 What to Turn On First¶
| Target | How to turn it on |
|---|---|
Message flow (full-span tracking including flow, corr) |
The runtime diagnostics' message flow mode |
| C++ / .NET spot discovery trace | ZLINK_DEBUG_FRAMEWORK_SPOT_DISCOVERY |
| Java / Kotlin stream trace | ZLINK_JAVA_STREAM_TRACE=1 |
| Preserving sample server logs | .NET ZLINK_SAMPLE_EVIDENCE_DIR, JVM ZLINK_SAMPLE_KEEP_RUN_DIR=1, Node is automatic on failure |
If a sample fails intermittently, keep the server log from the first reproduction run. A reproduction run without a log leaves only the fact of failure, not the cause, so that cycle is wasted.
4.2 How to Read It¶
First, line up the normal and failed cases side by side using flow and
find where the transition breaks. flow is the only value that
joins one message across process boundaries. Treating trace kinds as
noise and filtering them out of grep skips right past the line that
contains the cause.
4.3 A Failure Is Always Left in the Flow¶
Don't create a termination that only returns an error kind to the
application and discards the cause. A failure that leaves no cause can
only be tracked by reproduction, and the reproduction cycle itself
becomes the cost of investigation. Record a termination such as a failure,
rejection, or abort with outcome=failed and reason, as defined by
03. Message Flow Tracing, carrying a
cause-description string within the implementation-defined length
limit, under the same flow as the message that produced that
failure.
The cost rule for turning this record on and off is defined by 03. Message Flow Tracing "5. Changing the Record Level at Runtime and the Cost Rule".
5. What This Topic Does Not Define¶
| Content | Owning document |
|---|---|
| The progress and result of individual host operations (relocation, shutdown) | Host Relocation and Shutdown |
| The ownership and size of metadata the application sends with a message | Message Model |
| Transport connection liveness and peer deadline | Transport Connection Liveness |
The definition of the STREAM connection close reason (close_reason) |
Session "STREAM Server Session" |
Spec table of contents · Next: 01. Runtime Status and Operational Diagnostics