Complete Host Relocation Flow¶
Location·Relocation Topic Table Of Contents · Spec Table Of Contents · Previous: 04. Complete Actor And Spot Relocation Flow · Next: 06. Failure Handling And Failover Scope
What this document defines — the complete sequence and results in which Host
Relocatefixes the stateful workload as relocation units, selects a target, moves the units, returnsRelocated, and removes source resources through Message Follow andShutdown. The single source for one unit's owner transition and message processing order is Complete Actor And Spot Relocation Flow; this document adds only how the host operation enumerates those units, the modes, and the completion conditions.
1. Questions This Document Answers¶
This document defines which operations the application calls when moving a host's stateful workload to another node or shutting the host down, the order in which the Framework processes them, and what result each call ends with.
The unit that runs message handlers and Actors is called a Spot. In this document, stateful workload means the Actors/Spots a host is processing and the not-yet-finished messages and timers.
To inspect or reboot a node while keeping the application version, call Relocate with
PlannedMaintenance. To switch to a prepared new application version, call it with
RollingUpdate. The way the target version is chosen is called a
relocation mode.
On success, both modes only detach the stateful workload from the source host. The host and infrastructure connections are kept. The application or deployment orchestrator confirms this result and then separately calls Shutdown, which stops the runtime and refuses new operations.
To shut down a host without guaranteeing stateful workload continuity, call Shutdown
alone, without Relocate.
1.1 Failure-Handling Scope¶
Relocate only supports a graceful handoff that runs until the source runtime, the
chosen target runtime, and the
Location Store — the store where
multiple nodes together confirm each Spot's current owner and location — finish the
operation. A transient Store or
transport error within the same process can be retried within the deadline. But once the
source or target process terminates, a different runtime doesn't take over the
relocation, and there's no automatic recovery by picking a different target. Recovery
after the source or target process terminates is outside this contract.
The rule against creating two owners at once still applies here too. If a Location Store change's result isn't received, success or failure isn't guessed — the same record is re-read. Source admission isn't reopened, and target application message processing isn't started, before confirming the actual owner.
A connection group in which several nodes exchange messages is called a RouteMesh. The name that identifies the group of nodes connected under the same name is called a MeshName. The name selecting a target among nodes participating in the same Channel is ChannelName. The application doesn't directly assemble a shutdown sequence by picking only some components via MeshName, ChannelName, or node RID.
A RouteMesh, a runtime node participating in that group (MeshNode), a ClientServer server, and a Classic fanout publisher are all coordinated together at the host level.
This document owns the host lifecycle the application observes, the method for fixing Actor/Spot relocation units, the complete order for removing source resources after cutover, and the handoff result. The node currently processing an Actor/Spot is called the owner. The record by which multiple nodes together judge an Actor/Spot's owner and location is called authority. The order in which the Framework uses authority and the two Stores is defined by Location Runtime. The provider contract for the Location Store, which stores the current owner and location, is defined by Location Store (Redis). The application state and unexecuted queue/timer payload being moved doesn't pass through a store — it's sent directly from source memory to the target, and that transfer contract is owned by Complete Actor And Spot Relocation Flow. The provider contract for the responsibilities remaining in the Relocation Store — the store that keeps the record made when an Instance Spot is newly created by its first message, and the terminal result of a pending request completing after relocation — is defined by Relocation Store (Redis). This document doesn't repeat the transfer format — it only defines the public order of host operations.
2. Operations the Application Chooses¶
2.1 Choosing a Relocate Mode¶
The caller must specify a mode when calling Relocate. The only difference between the
two modes is the target application version. The subsequent queue seal, state restore,
authority transition, and session handoff rules are the same.
| Value | Mode | Value the caller specifies | Target the Framework selects |
|---|---|---|---|
| 0 | PlannedMaintenance |
Doesn't specify TargetApplicationVersion. |
Selects only a node whose application version exactly matches the source. |
| 1 | RollingUpdate |
Specifies a TargetApplicationVersion greater than the source. |
Selects only a node whose application version exactly matches what the caller specified. |
PlannedMaintenance's effective target version is the source's ApplicationVersion.
Specifying a target version alongside it is an argument error. RollingUpdate is an
argument error if there's no target version or it's at or below the source version. The
Framework rejects these invalid combinations before changing runtime state or admission.
The final time an operation must finish is called a
deadline. Both modes use 30 seconds if Deadline is
omitted. A specified value must be greater than 0.
2.2 Public Operation¶
The following .NET declaration is one expression of the common contract. The name and signature in other languages are defined by that language's interface document.
public enum ZLinkFrameworkRelocationMode
{
PlannedMaintenance = 0,
RollingUpdate = 1
}
public sealed record ZLinkFrameworkRelocationOptions
{
// choose whether this is maintenance at the same version or a new version rollout.
public required ZLinkFrameworkRelocationMode Mode { get; init; }
// only for RollingUpdate — specify one version, greater than the source.
public long? TargetApplicationVersion { get; init; }
// uses 30 seconds if omitted.
public TimeSpan? Deadline { get; init; }
}
public readonly record struct ZLinkFrameworkRelocationResult(
ZLinkFrameworkRelocationMode Mode,
long TargetApplicationVersion,
ZLinkFrameworkRelocationOutcome Outcome,
ZLinkFrameworkRelocationReason Reason);
public interface IZLinkFrameworkRuntime
{
// provides the current host lifecycle state and last result.
ZLinkFrameworkRuntimeStatus Status { get; }
// observes host state and terminal result changes in order.
IAsyncEnumerable<ZLinkObservedStatus<ZLinkFrameworkRuntimeStatus>> ObserveAsync(
CancellationToken cancellationToken = default);
// moves stateful workload; stays in the Relocated state on success.
ValueTask<ZLinkFrameworkRelocationResult> RelocateAsync(
ZLinkFrameworkRelocationOptions options,
CancellationToken cancellationToken = default);
// cleans up accepted work and host resources without a new relocation.
ValueTask<ZLinkFrameworkTerminationResult> ShutdownAsync(
TimeSpan? deadline = null,
CancellationToken cancellationToken = default);
}
A typical rolling-update call sequence looks like this.
var relocation = await runtime.RelocateAsync(
new ZLinkFrameworkRelocationOptions
{
// only use nodes prepared as N+1 as target candidates.
Mode = ZLinkFrameworkRelocationMode.RollingUpdate,
TargetApplicationVersion = currentVersion + 1,
Deadline = TimeSpan.FromSeconds(30)
},
cancellationToken);
if (relocation.Outcome == ZLinkFrameworkRelocationOutcome.Relocated)
{
// once relocation success is confirmed, clean up host resources separately.
// shutting down immediately also removes the Message Follow routes and
// the cutover retransmission copies (§14).
await runtime.ShutdownAsync(cancellationToken: cancellationToken);
}
Calling Shutdown right after Relocated, as in the example above, is allowed. A
deployment that wants to use both old-route message forwarding and cutover
retransmission to the end waits until
SafeToShutdown — the observation value
indicating the source runtime can shut down safely — is published in the runtime status
before calling Shutdown; the publication condition is explained by
§14.
A Relocate result always includes the mode and effective target version. Even when no
target is found, the same values are preserved so the caller can confirm what condition
it was waiting on. If Relocate ends Blocked, the caller can retry, or Shutdown
without continuity.
3. Host State and Completion Results¶
The host lifecycle is owned by a single FrameworkRuntimeState.
Information a host publishes to the Store — its endpoint, run generation, and capabilities — is called a descriptor. The state where it's ready to accept new application work is called ready.
| Value | State | Meaning |
|---|---|---|
| 0 | Preparing |
Proceeds with registration, bind, descriptor verification, and recovery; doesn't accept application messages. |
| 1 | Serving |
The host is ready and accepts new application work. |
| 2 | Relocating |
Excluded from new placement and selection, but local units not yet sealed keep processing messages and timers. |
| 3 | Relocated |
Every stateful object has been detached from source dispatch. Host and infrastructure connections are kept. |
| 4 | Draining |
Shutdown has closed new admission and is cleaning up already-accepted work and resources. |
| 5 | Stopped |
Application resource, infrastructure resource, and listener cleanup are finished. |
| 6 | Error |
Can't serve due to a startup or runtime error. |
IsReady is true only in Serving. A component lifecycle snapshot provides information
about each component's state and doesn't substitute for host state. Per-component
Drain, AwaitDrained, Stop, or a public operation targeting only some Meshes isn't
provided.
%%{init: {'state': {'nodeSpacing': 32, 'rankSpacing': 40, 'padding': 8}, 'themeVariables': {'fontSize': '18px'}}}%%
stateDiagram-v2
[*] --> Preparing
Preparing --> Serving: required component ready
Preparing --> Error: startup error
Serving --> Relocating: Relocate intent published
Serving --> Draining: Shutdown seals admission
Serving --> Error: runtime error
Relocating --> Serving: source processing restored after Blocked
Relocating --> Relocated: every relocation unit detached
Relocating --> Draining: Shutdown requested
Relocated --> Draining: Shutdown requested
Error --> Draining: bounded cleanup starts
Draining --> Stopped: resource cleanup complete
For each unit, only an explicit failure before its relay-ready reply reaches the accepted
state may clean tentative work and restore source processing. Even when another unit has
crossed that boundary, the host may restore source workload that hasn't crossed it and
return to Serving. A unit across the boundary never returns to source regardless of its
cutover-submit result; it continues through target cutover receipt (including a
retransmission after connection re-establishment) or the cutover-wait fallback.
Returning to Serving doesn't mean every unit rolled back to the source.
Relocation outcome is fixed to the following values. Among the reasons in the table,
DeadlineExceeded means the call ended
because it did not finish within its allotted time; it does not mean the remaining work was
cancelled.
| Value | Outcome | Allowed reason | Meaning |
|---|---|---|---|
| 0 | Relocated |
None |
Every stateful object has been detached from source dispatch. |
| 1 | Blocked |
TargetUnavailable, StoreUnavailable, RelocationDisabled, StateIncompatible, DeadlineExceeded, RelocationFailed, RuntimeNotReady, ManualTopologyUnsupported, ShutdownRequested, OperationInProgress |
Relocation couldn't start, or the whole workload move didn't finish. |
The wire values are Relocated=0, Blocked=1. Reason is None=0,
TargetUnavailable=1, StoreUnavailable=2, RelocationDisabled=3,
StateIncompatible=4, DeadlineExceeded=5, RelocationFailed=6, RuntimeNotReady=7,
ManualTopologyUnsupported=8, ShutdownRequested=9, OperationInProgress=10. An
undefined outcome/reason combination is a protocol error.
Shutdown outcome is Stopped=0, ForceStopped=1, and reason is None=0,
DeadlineExceeded=1, TeardownFailed=2. ForceStopped isn't a separate host state —
it's the result of finishing cleanup via bounded teardown, with host state Stopped.
Relocation failure is owned by the relocation result and isn't mixed into the
termination reason.
4. Conditions Checked Before Selecting a Target¶
Relocate started from Serving checks the whole host at once before changing host
state and application admission. At this point it doesn't block new work for the moving
targets or pre-secure target capacity.
The way of saving application state as bytes and restoring it on the target is the Preserve-state relocation policy. The period during which a host keeps current owner eligibility is called an owner lease.
| Check item | Passing condition |
|---|---|
| Concurrently in-progress work | If there's new object creation, join, Instance placement, session binding, or inbound relocation, the work to finish first is confirmed. |
| Local workload | Checks every MeshNode's Actors, Spots, timers, sessions, and in-progress infrastructure operations. |
| Store | The Location Store's current location record and the target descriptor's owner lease must be usable. Because the payload being moved is sent directly from source memory, Relocation Store readiness isn't a handoff condition — it's checked only in deployments that need the remaining responsibilities: the record made when an Instance Spot is newly created by its first message, and the terminal record of a pending request completing after relocation. |
| Unit compatibility | The relocation policy and state adapter used by an explicitly application-created User Spot, an Instance Spot created on demand by its first message, and the Actors within them, are all compatible with the target factory and capacity. |
| Topology | Every service topology the host uses finds remote endpoints via Store descriptors. This method is called automatic discovery. |
If even one manual RouteMesh peer, ClientServer client endpoint, fanout subscriber
endpoint, or manual fanout publisher not publishing a descriptor is registered, it's
Blocked/ManualTopologyUnsupported. Registration is checked, not current connection
status. A setting where an automatic component explicitly binds a listener address isn't
manual topology.
The scope the runtime checks is local registration. It doesn't check connections another process made outside the Framework, so the condition that the whole set of participating processes uses only automatic discovery is guaranteed by the deployment.
5. Selecting a Target Matching the Mode¶
The Framework narrows down targets in this order.
| Order | Condition |
|---|---|
| 1 | PlannedMaintenance keeps only the same application version as the source. RollingUpdate keeps only the version the caller specified, excluding both lower and higher versions. |
| 2 | Keeps only an Object Server that isn't the source and is in Serving state. |
| 3 | Keeps only nodes compatible in factory — the function the application registered to create objects — stable type — the language-independent object type name — relocation policy, and state adapter. |
| 4 | Keeps only nodes with sufficient remaining capacity, and, if the source has a maintenance wave set, only nodes with a different value. Maintenance wave is an application setting distinguishing hosts in the same maintenance operation. |
| 5 | Keeps only nodes whose RID and lifecycle generation both match in the same-point-in-time descriptor list and the Core peer table. Lifecycle generation distinguishes different process runs using the same RID. That peer must be Admitted and Ready. |
Version is checked before capacity and placement weight. So a different-version node's remaining space or higher weight doesn't affect the selection result. The relative share by which new work is assigned among several candidates is called weight. This value only applies when there are multiple targets satisfying the conditions.
A published descriptor or a created connect intent alone doesn't establish that a target is ready. The result of copying state at a specific point in time into a read-only value is called a snapshot. If the descriptor snapshot is empty, contains only the source itself, or every remote peer is draining, there's no target.
5.1 When There's No Target Yet¶
Before searching for a target, the Framework first checks whether there's a unit to move. If
the source owns no Actor, User Spot, or Instance Spot, there's nothing to move, so a target
isn't searched for and it ends with Relocated/None. Relocation's purpose is workload
migration — a host with no workload to move isn't blocked just because it has no target.
Even in this case, the host state transition and admission closing are the same as any
other relocation.
If there's a unit to move but no target of the requested version exists, source state
and admission are kept while waiting up to the deadline for the descriptor and Core peer
table to converge. A process with multiple Meshes must satisfy the condition on every
Mesh. If no target is secured by the deadline, tentative coordination is cleaned up and
Blocked/TargetUnavailable is returned.
%%{init: {'sequence': {'actorFontSize': '18px', 'messageFontSize': '18px', 'noteFontSize': '18px', 'boxMargin': 8, 'width': 140}, 'themeVariables': {'fontSize': '18px'}}}%%
sequenceDiagram
participant Target as Replacement node
participant Source as Source host
participant Store as Location Store
participant App as Deployment orchestrator
Target->>Store: [request] publish replacement Serving descriptor
Store-->>Target: [reply] descriptor version fixed
App->>Source: [request] relocate host workload to requested-version target
Source->>Source: [local] check requested-version peer is Ready
Source->>Store: [request] transition source host to Relocating
Store-->>Source: [reply] Relocating state fixed
Source->>Target: [request] prepare temporary queue, Restore, and relay for every unit
Target-->>Source: [reply] relay reception ready for every unit
Source->>Target: [send] ingress-hold relay and cutover for every unit
Source->>Store: [request] transition source host to Relocated
Store-->>Source: [reply] Relocated state fixed
Source-->>App: [reply] host relocation result Relocated
App->>Source: [request] clean accepted work and shut down host
Source->>Source: [local] clean up accepted work and infrastructure
Source-->>App: [reply] shutdown result Stopped or ForceStopped
Relocated is the source-side result that every unit's cutover submit attempt reached a
success or failure terminal; it doesn't mean the source awaited a target-CAS completion
reply. Relocated keeps descriptor, connection, listener, and infrastructure resources.
This diagram shows the normal flow when a target is ready. Without a target, the
deadline rule from the previous section returns Blocked/TargetUnavailable.
Automatic ClientServer clients and fanout subscribers build new connections using the replacement descriptor and reflect the source's state in selection. An existing connection with accepted work and a remaining barrier isn't closed immediately just because the descriptor changed.
6. Concurrent Calls and Cancellation¶
| Situation | Result |
|---|---|
Concurrent Relocate with the same mode and effective target version |
Shares the deadline of the first operation. A later call doesn't change the deadline. |
Concurrent Relocate with a different mode or effective target version |
Blocked/OperationInProgress without waiting. |
Relocate again after Blocked |
Blocked isn't stored, so host conditions are re-checked from the start. |
Relocate again in Relocated |
Returns the first Relocated/None's mode and effective target version. |
Concurrent Shutdown |
Shares the same operation and stores the terminal result. |
Shutdown again in Stopped |
Returns the stored result. If none, Stopped/None with no new work. |
Relocate in Preparing, Error, or Stopped |
Blocked/RuntimeNotReady without changing admission. |
Caller cancellation ends only that waiter — it doesn't cancel the shared operation.
Shutdown interrupts startup in Preparing and starts bounded cleanup in Error.
7. Relocation Units and Batch Order¶
The bundle of Actors or a Spot the Framework can move independently is called a relocation unit. The relationship of which Entry Spot or User Spot an Actor currently belongs to is Actor membership.
| Relocation unit | Boundary |
|---|---|
SpotWide User Spot aggregate |
The User Spot and every member Actor at the moment new work is blocked |
| Actor | One Actor belonging to an Entry Spot or PerActor User Spot |
PerActor Spot authority transition |
The target's stateless Spot shell and Spot-level queue authority |
| Instance Spot | One Spot with no Actor membership |
The Entry Spot itself doesn't move. A source Entry Spot's Actor moves as an Actor unit to
the target node's Entry Spot. A PerActor User Spot also moves its Actors as the same
unit without moving Spot application state.
The source runtime reads active state once, immediately before the host enters
Relocating, and fixes the inventory. An Actor in a User Spot membership appears exactly
once, either as a PerActor Actor unit or inside a SpotWide aggregate, according to
the Spot's execution mode. An Entry Spot Actor and a standalone Actor not included in a
User Spot membership each become an Actor unit. The same Actor never appears in both a
Spot unit and a standalone unit. No new stateful placement is admitted after the
inventory is fixed.
Host relocation runs the following batch order. A batch means units with the same dependency; it isn't a fixed-size chunk based on a numeric unit count.
| Batch | Units | Start condition |
|---|---|---|
| 1 | Stateless Spot shell and Spot-level queue authority for each PerActor User Spot |
Inventory and target preflight are complete and the current Spot turn has ended. |
| 2 | Entry Spot Actors, PerActor member Actors, and standalone Actors |
Each Actor's current turn has ended; a PerActor member also waits for its Spot shell transition. |
| 3 | SpotWide User Spot aggregates and Instance Spots |
The entire aggregate's current turn and application safe point have ended. |
The next batch starts after the previous batch reaches a terminal state. Units without a
dependency in the same batch may run concurrently. A PerActor member Actor never starts
before its Spot shell, and a SpotWide member Actor isn't split into a separate Actor
unit. When a unit fails, no later batch starts. Units that have already started continue
to a safe terminal state.
%%{init: {'flowchart': {'nodeSpacing': 32, 'rankSpacing': 40, 'padding': 8, 'wrappingWidth': 180}, 'themeVariables': {'fontSize': '18px'}}}%%
flowchart LR
I[Fix active inventory] --> S[Batch 1: PerActor Spot shells]
S --> A[Batch 2: Entry, PerActor, and standalone Actors]
A --> G[Batch 3: SpotWide and Instance aggregates]
G --> C[Finish every cutover submit attempt]
C --> R[Publish Host Relocated]
For relocation correctness, the Framework sets no separate cap on concurrent unit count, participant count, or relay record count. How much moves concurrently per unit is paced by the in-flight payload budget, which limits the bytes occupied by in-progress relocation payloads on a peer connection at the same time; the budget's calculation and waiting rules are owned by Complete Actor And Spot Relocation Flow §5.3. When the budget is full, the next unit waits before its source admission seal is applied, and a waiting Actor/Spot keeps processing messages normally in the meantime — the coordinator doesn't set a separate cap on concurrent unit count. Existing runtime and Store provider memory, frame, and page-size limits still apply, and if a resource isn't immediately available, it waits before blocking source application dispatch. Reaching a limit doesn't fail a relocation that has already started.
If the target is already saturated, chunk intake slows down, so the source's budget releases come later and the next unit's start is also delayed — a move meant to shed load is slowed by that load. This is the intended result of the backpressure contract that never bypasses saturation; no bypass path is created for relocation. If the move doesn't finish within the host operation deadline, the existing §8 rule against starting any new unit relocation applies, so the operator first decides between extending the deadline and relieving target load. Deadline sizing uses the transfer throughput observed in that deployment as input, not a formula.
This batch order isn't an Application Job Queue capacity chunk. Each target's pre-dispatch temporary queue and saved work form an ordered durable backlog owned by a retained-byte owner; ordinary staging ingress uses a shared reservation for receive and returns it at durable handoff. After target-only CAS and required lifecycle work make dispatch runnable, backlog handler turns acquire live queued-job permits one at a time in order. So a compatible target whose job limit is smaller than an aggregate backlog executes it progressively instead of failing the aggregate as a capacity blocker or leaving members at the source.
SpotWide moves the Spot and every member Actor at the end of the current turn as one
unit. PerActor and Entry Spot move each Actor once that Actor's current turn ends. A
PerActor Spot lane is only briefly blocked during authority transition and doesn't
wait for every member Actor to become ready at the same time.
A SpotWide User Spot using ApplicationSignaled readiness uses the turn boundary the
application registers via RelocationReady().Defer() after target preparation. If no
relocation is prepared, a Continued completion callback runs on the source in the next
application turn. Cancellation before relay-ready is accepted restores the source queue
and then calls Continued. After that boundary, cancellation or cutover-submit failure
doesn't restore source. After owner commit, Relocated runs as the first application
turn on the target queue.
8. Interruption Budget Target¶
| Target | Measurement unit |
|---|---|
| Entry Spot Actor | One Actor |
PerActor User Spot |
One Spot direct admission — targeting by Global Spot ID — and each Actor |
SpotWide User Spot |
One aggregate including the Spot and member Actors |
| Instance Spot | One Spot |
Measurement starts when the source admission seal is applied. Wait time for target selection, target preparation including temporary-queue registration, and waiting for the current turn or an application safe point are excluded. This preparation must finish before source admission is blocked. Capture, encoding, chunk transfer to the target, checksum verification, authority change, target Restore, and queue/timer restore run after the seal and are all included in measurement. Measurement ends when the source's ordered-relay-then-one-way-cutover submit reaches a terminal result, whether success or failure. What this metric measures is the time the source is stopped — not the full interruption the application observes until the target resumes processing.
Each relocation unit records the following points in time. Each point is recorded, on its own clock, by the node where the event happens.
| Point | Meaning | Recorded by |
|---|---|---|
| S0 | Source admission seal applied | Source |
| S1 | Terminal result of the one-way cutover submit | Source |
| S2 | Target's Location Store CAS confirmed | Target |
| S3 | Target application dispatch opened | Target |
| S4 | The moment the Message Follow route — which keeps relaying messages that arrive at the old address to the target — may be removed. Based on MessageFollowDuration expiry; the source owns the Message Follow route, so this is a source-side point. |
Source |
| Metric | Interval | Meaning |
|---|---|---|
| Source stop time | S0→S1 | Same as the measurement interval defined above. |
| Target resume time | S2→S3 | Target-local interval from owner confirmation to opening application dispatch. |
| Route convergence time | S1→S4 | Source-local interval that grounds how long the source must keep the Message Follow route. |
No metric subtracts timestamps of different nodes directly. A whole interval crossing nodes, like S0→S3, is observed only through same-flow correlation in message flow tracing. S3 is the dispatch-open moment, so because of backlog and permit ordering, the moment the application observes its first handler result can be later. The formal definition of metric names and instruments is owned by Runtime monitoring and Runtime metrics; this section fixes only the point definitions and who measures them.
Each unit has a default target of under 1 second. This value is neither a timeout nor a
correctness condition — it shares the same number as
Complete Actor And Spot Relocation Flow's RelocationCutoverWaitTimeout (default
1,000 ms) but is a
different value: this 1 second is an observation-only target used solely as a warning
threshold, while RelocationCutoverWaitTimeout is the protocol fallback deadline after
which, once the cutover wait ends, the target moves on to CAS and queue opening.
Exceeding it doesn't cancel the relocation or roll back to the source. The Framework
keeps the same operation going until the one-way cutover submit reaches a terminal result, and records
a warning and the zlink.relocation.interruption histogram. Source application close
runs after the cutover submit's success or failure terminal. The target sends no
processing-start ACK; target admission opening is observed through target-local status
and traces.
Once the host operation deadline ends, no new unit relocation is started. A unit already
started performs a safe abort only when target explicitly fails before sending its
relay-ready reply. If the reply result is indeterminate, target may have started the
cutover-wait fallback, so source dispatch doesn't reopen. A unit that attempted cutover
also doesn't roll back to source, and its target continues the owner transition until
the Restore validity deadline. If source doesn't attempt every unit's cutover, the host
doesn't become Relocated; success or failure of an attempted submit isn't a completion
condition.
9. The Order for Relocating One Unit — Owned by 04¶
The single source for one Actor or Spot unit's owner transition, ordered relay, temporary-queue installation, checksum verification, cutover, Location Store CAS, queue merge, and Session route transition is Complete Actor And Spot Relocation Flow "4. Normal Processing Order". Once host relocation starts, the Framework moves the source host's workload to the target node without the application calling a separate move API per Actor or Spot. Actor and Spot IDs are kept, and once moved, a target continues message processing in existing queue order.
This section adds only how many units the host divides that common order into and in what order it opens them — the batch rules dividing units are defined in §7. This section covers only the differences, per unit kind, in what the target prepares and what values it changes together, and which callbacks it calls or doesn't call, in §10.
| Participant | What it does (same as 04) |
|---|---|
| Application | Calls the host's Relocate. Only a SpotWide User Spot that chose ApplicationSignaled signals a safe move moment via RelocationReady().Defer(). |
| Source runtime | Finishes currently running work and stops application dispatch. Fixes application state and the not-yet-executed queue/timers as a payload in source memory and sends it directly to the target, and relays to the target only messages that arrive at the old address after capture. It doesn't change the Location Store. |
| Target runtime | Assembles the received chunks and verifies the checksum, then creates an Actor or Spot using the same ID and restores state and existing work. After receiving the relay cutover boundary, or reaching the cutover-wait fallback after relay-ready, it CASes the Location Store from source to target and opens the queue only if that succeeds. |
| Location Store | Records which node currently processes an Actor or Spot. When multiple values must change together, changes all or none. |
| Relocation Store | Holds no handoff payload. Its only remaining responsibilities are recording when an Instance Spot is newly created by its first message and recording the terminal result of a pending request completing after relocation. |
RelocationCutoverWaitTimeout (the wait time from the relay-ready reply until cutover
arrives, default 1,000 ms) is owned by
Framework API.
What the target factory does, per policy, is as follows.
| Policy | Handling |
|---|---|
DisableRelocation |
If that object remains, Blocked/RelocationDisabled. |
RecreateOnRelocation |
Runs the target factory with the same object ID. Application state isn't moved, but not-yet-finished Framework work is moved. |
PreserveStateWith |
Stores the bytes the adapter returned and Restores them into the target factory instance. The application manages the bytes' format, version, and migration. |
The Framework doesn't add a separate state contract ID or generic state type.
One relocation unit's temporary queue has no bound on record count or stored size, and the Framework doesn't create an additional temporary queue for the same object.
10. Differences by Unit Kind¶
10.1 An Actor Belonging to an Entry Spot¶
An Entry Spot instance belongs to the Object Server lifecycle, so it isn't moved to
another node. Only Actors belonging to a source Entry Spot are moved, each as an
independent relocation unit. The target runtime restores the Actor into the Entry Spot
the target node already created at startup. Since this move isn't a join the
application requested, it doesn't call the target Entry Spot's OnJoinedActor or the
source Entry Spot's OnLeaveActor/OnActorJoin.
10.2 PerActor User Spot¶
A PerActor User Spot changes the node handling Spot messages and the node handling each
Actor separately. So even while some Actors are still on the source, already-moved Actors and
new Spot messages can be handled on the target.
- The target runtime creates an empty Spot instance using the same SpotId and ObjectGeneration as the source. Before the Location Store's current location changes, this instance isn't returned as an external lookup result.
- The source runtime finishes the current Spot handler and in-progress Actor Create/Join. Spot messages arriving afterward are briefly held on the source.
- The target runtime registers a Spot relocation temporary queue. The source runtime keeps relaying held Spot messages and later messages arriving on the previous route to this queue.
- Once the target's Spot is ready, the Location Store changes the node handling Spot
messages from source to target. Held Spot messages move to the real Spot queue, the
temporary queue registration is removed, and new
ToSpot, Actor Create, and Join are handled via the existing dispatch path. - Each source member Actor finishes its current turn and then moves to the target as
an Actor unit.
ToActorfor a not-yet-moved Actor goes to the source; for a moved Actor it goes to the target. - Once the last Actor and all messages held on the source are delivered to the target,
OnClosing(RelocationOut)is called on the source Spot.
The Framework doesn't create a temporary SpotId during the move. Once the Location
Store records the target as the current node handling Spot messages, the source Spot no
longer handles new ToSpot, Create, or Join. The source Spot only continues handling
not-yet-moved Actors and delivering messages during the move.
The empty Spot created on the target doesn't restore the source Spot's application fields, so it doesn't call the Spot relocation adapter. Each member Actor uses its own relocation policy and Actor adapter. Applying the Session command 44 route update doesn't block processing of each Actor or the next Actor relocation.
10.3 SpotWide User Spot¶
A SpotWide User Spot moves the Spot and every member Actor, as of the moment new work
is blocked, as a single move operation. If even one Actor fails to satisfy the
relocation policy, state adapter, or target capacity condition, the Location Store isn't
changed and the whole move is aborted. The ID distinguishing this move is a non-zero
128-bit value.
The target registers the Spot and every member Actor in the same relocation temporary
queue group. Each record preserves the actual target Spot or Actor identity. Only after
every participant's Restore and the aggregate owner change does it split saved work,
pre-boundary relay, and remaining temporary work into the real Spot and Actor queues,
then switch to the regular route. It next finishes OnRelocationReadyCompleted and
opens dispatch. If one participant fails, no work in the temporary queue is run and the
whole group is discarded.
There's no 1,024 cap on the total number of Actors belonging to a User Spot. The Framework splits the relocation target list across multiple Location Store pages. One page records at most 1,024 entries, and one encoded page's size is at most 1 MiB. For example, with 2,500 Actors, at least three pages are used. The Framework confirms the total Actor count and each page's content matches the originally stored list. Only when everything matches does it change the node processing the User Spot and all Actors from source to target, all at once. If a conflict occurs mid-way, it doesn't change the locations of only some Actors. This method — changing everything or nothing, only if the first-read Store version is unchanged — is called CAS.
Member Actors' OnActorJoin, OnJoinedActor, and OnLeaveActor aren't called. Bound
Session location updates proceed per Actor after the Spot and Actors start message
processing, and one Session owner's response doesn't block processing of a different
Actor or Spot.
10.4 Instance Spot¶
An Instance Spot can't contain Actors, so one Spot is the relocation unit. Once the
source's current handler finishes, direct messages and timers are held. The target
runtime creates the Instance Spot with the same SpotId, and if it's PreserveStateWith,
restores the directly transferred application state via Restore. Once the Location
Store records the target as the current processing node, the target processes the
restored queue and timers. Since an Instance Spot has no Actor, Actor location or
Session binding isn't updated.
Host relocation only moves an Instance Spot that already exists on the source. It doesn't start cold activation, which creates an Instance Spot not on the source from its first message.
10.5 Callbacks Not Called During the Move¶
Entry Spot and PerActor User Spot Actor relocation isn't a join or leave requested by
the application. So OnActorJoin, OnJoinedActor, and OnLeaveActor aren't
called. The Framework moves Actor state, not-yet-executed queue, and timers, and
changes only the current processing node.
A SpotWide User Spot also only changes the processing node without changing which
Spot each Actor belongs to, so member Actors' join/joined/leave callbacks aren't
called. If ApplicationSignaled was used, only OnRelocationReadyCompleted(Relocated)
is called on the target after the regular-route switch and immediately before dispatch
opens.
OnClosing(RelocationOut) is called on a User Spot's or Instance Spot's source instance
after the Location Store's location change. Since an Entry Spot instance doesn't move,
the Entry Spot's closing callback isn't called. An Instance Spot has no Actor, so there's
no Actor lifecycle callback at all.
A cross-node Actor join uses the same policy and adapter, but the specific lifecycle is owned by Spot and Actor membership. A same-node join doesn't call the adapter. Instance Spot maintenance relocation doesn't newly create an Instance Spot not present on the source.
11. Which Location Is Kept on a Mid-Way Failure¶
If the target explicitly fails before the relay-ready reply is accepted, the source keeps processing messages. The Framework doesn't expose the target instance, discards the temporary queue, and restores source messages and timers to the original queue. The target doesn't create a request terminal result or run a one-way message from the temporary queue.
After the relay-ready reply is accepted, source dispatch doesn't reopen while the Location Store still points to source, even if cutover hasn't been sent yet or its submit fails. The target continues CAS after receiving cutover (including a retransmission after connection re-establishment) or through the cutover-wait fallback. If target CAS ultimately fails, the target removes its object and queue, the Session cleans up under its own seal timeout, and the source's Message Follow also ends after its defined duration.
Once the Location Store records the target as the current processing node, it isn't rolled back to the source. If the target runtime is still running, a failed stage can be retried. Location Store update retries until Restore validity expires; if target ownership isn't confirmed by then, the prepared Actor or Spot and queue are removed and the Session route isn't updated. If the source or target process terminates, a different runtime doesn't take over this relocation. If the target terminates after commit, it isn't rolled back to the source — that object is left unavailable. Automatic recovery afterward isn't part of the contract. The source sends one-way cutover, waits for no completion reply, and changes to Message Follow. After CAS and queue opening, the target sends the Session route update one-way. This choice doesn't guarantee exactly-once behavior or global ordering across a process-crash window. The specific failure result the application observes is defined by §13 Relocate Completion and Failure.
12. Moving Pending Messages, Timers, and Sessions¶
The number distinguishing whether an object under the same ID was deleted and re-created is called ObjectGeneration. The value distinguishing one operation, to avoid processing a message or request twice, is operation identity. The window during which the source temporarily holds new messages during a move is called relocation ingress hold.
| Resource | Move rule |
|---|---|
| A message arriving after new work is blocked | The source holds arriving messages with no bound on record count or stored size. If the owner change succeeds, the message is delivered to the target with its operation identity and ObjectGeneration preserved. On an explicit cancellation before the relay-ready reply is accepted, it's restored to the source queue in arrival order; afterward it isn't restored to source. |
SpotWide/Instance Spot timer |
The runtime handle and continuation aren't moved. Logical registration, next fire time, and pending tick are moved, and the target automatically restores them in queue order. The application doesn't duplicate-capture a timer or re-register it in restore. |
Entry/PerActor Actor timer |
Moves with the Actor queue to the Actor owner. Spot-level application timers aren't moved — a schedule that must be kept is managed in the application's external state. |
| A session connected to an Actor | The physical STREAM connection is kept. The specific seal/route-transition order and timeouts are summarized by 04 §7 and owned by Session and Actor Binding "8". |
Operation identity and authority generation are also kept when delivering a
late-arriving message to the target via the previous owner. Independently of Session
command 44 application, a Message Follow route only delivers packets arriving on the
previous route to the target Actor within MessageFollowDuration. Packets and replies
of a previous generation are rejected. A newly created Actor under the same ActorId
must be rebound by the application.
An Instance Spot's Close and relocation are ordered within the same authority commit.
If Closing comes first, close finishes and it isn't moved. If relocation comes first,
a late Close is a moving result and isn't automatically resubmitted.
13. Relocate Completion and Failure¶
Once every unit is detached from source dispatch and the one-way cutover submit attempt
for each target that sent a relay-ready reply reaches a success or failure terminal, the
host transitions to Relocated and returns Relocated/None. This result is not
confirmation that target Location Store CAS completed. Descriptor lease, listener, peer
connection, and raw transport resources aren't cleaned up at this point.
| Completion point | Observer | Meaning |
|---|---|---|
| Restore and relay-ready reply | Source unit | The target temporary queue and Restore are ready, and the source is still the owner. |
| One-way cutover submit terminal | Source unit | Source attempted cutover once after pre-boundary relay and obtained a success or failure terminal. Neither result confirms target CAS. |
Relocated/None reply |
Source host and caller | Every source unit dispatch has ended and every cutover submit attempt reached a terminal result. Submit success isn't a completion condition. |
| Successful Location Store CAS | Target unit | The target is the owner and may open the transferred existing queue and the relay queue in order. |
| Applied Session route update | Session owner | Owned by 04 §7 and Session and Actor Binding "8". |
The target sends neither a cutover reply nor a Session route-update reply. The source host doesn't create an acknowledgement journal or numeric high-water to wait for target CAS or Session route application.
Because cutover can be lost to a connection failure, the source keeps a copy of each
unit's pre-boundary relay batch and cutover for the same duration as the cutover wait
time (RelocationCutoverWaitTimeout) after the first cutover submit terminal. That
duration is the unit's cutover retransmission window. If the connection to the target is
re-established within the window, the source resends the batch and cutover over the new
connection, and the target discards its partially received pre-boundary relay span and
atomically replaces it with the whole retransmitted batch — a full replacement, not
per-message deduplication or partial merging, so the span's order is fixed by batch
order. Retransmission resends one batch; it's not a per-message ACK or journal. The copy
is source-memory retention that occupies no pipe, and when the window ends the source
cleans it up exactly once and never retransmits afterward. The retransmission window
changes none of the completion points in the table above — the host still transitions to
Relocated at the first cutover submit terminal, and retransmission is a recovery
action after it. Once the source process has been cleaned up or terminated,
retransmission is impossible and the target proceeds through the cutover-wait fallback.
The per-unit retransmission and replacement rules are owned by
Complete Actor And Spot Relocation Flow.
The result when an operation doesn't satisfy its completion condition by the deadline is
called DeadlineExceeded.
| Timing and cause | Result |
|---|---|
| No target candidate satisfying the requested application version and registered factory/type eligibility is ready by the deadline. | Blocked/TargetUnavailable |
| Store read, write, or owner lease check fails before relay-ready reply acceptance. | Cleans up temporary records without changing owner and returns Blocked/StoreUnavailable |
A DisableRelocation policy remains. |
Blocked/RelocationDisabled |
After target selection, the transferred state schema/type adapter is incompatible, or Capture and Restore both fail across every allowed retry. |
Blocked/StateIncompatible |
| Before relay-ready reply acceptance, the Framework cancels a callback due to the deadline or work exceeds the deadline. | Blocked/DeadlineExceeded |
| Target explicitly rejects Restore before relay-ready reply, so the source queue can be restored. | Restores source workload that hasn't attempted cutover and returns Blocked/RelocationFailed |
An explicit failure before relay-ready is accepted cleans temporary records and lets
that source authority and queue accept new work again. A unit that crossed this boundary
doesn't roll back to source regardless of cutover-submit success or failure. Only source
workload that hasn't crossed the boundary may be reprocessed before the host transitions
to Serving.
A target CAS, queue opening, or Session route-update failure after relay-ready is
accepted is not a Host result delivered synchronously to source. The target retries CAS
until the Restore validity deadline. If it cannot confirm target ownership, it removes
the prepared unit and records an Error log. It doesn't change an already returned
Relocated result or reopen source dispatch.
If the directly transferred payload's checksum differs from the assembly result, the
target doesn't restore from a partial assembly and answers with an explicit failure
before the relay-ready reply, and the source restores its queue from the payload kept in
memory — the same path as Blocked/RelocationFailed in the table above. If the
SpotWide relocation target list's content checksum differs from the originally stored
list, it's an unrecoverable DataLost even on retry. It doesn't guess a previous list or
roll back to the source.
If some MeshNodes' Relocating descriptor write result can't be confirmed, every
attempted descriptor is rolled back to Serving. Only once every rollback is confirmed
is it Blocked/StoreUnavailable. If even one can't be confirmed, no new work is
accepted, cleanup proceeds for a fixed maximum time, and it ends as
ForceStopped/TeardownFailed.
14. The Race Between Shutdown and Relocate¶
Shutdown isn't blocked by an absent target, policy, capacity, or Relocation Store. The
action that changes state to stop accepting new application work is called
admission seal. Shutdown first applies an
admission seal to the whole host. It doesn't guarantee stateful workload continuity and
completes within a fixed time, in this order.
- Changes the host to
Drainingand closes new application admission and the start of new relocation units. The mesh node follows the same seal: it neither starts nor accepts a new peer admission (sends no Hello, answers no inbound Hello with Admit) and sends already-admitted peers only theDrainingUpdate. A peer's transport loss, liveness expiry, or send failure after the seal does not move the node state back beforeDraining. - Publishes a
Drainingdescriptor, excluding it from new selection and placement. Only the publication's terminal (success or failure) is awaited; there is no timed wait for propagation after publishing. - Processes already-accepted handlers, request completions, relocation units, and session barriers up to the deadline.
- Doesn't start new object relocation. While Actor membership and local instances
remain valid, delivers a
HostShutdownclosing context to every Entry, User, and Instance Spot. Per-Actor closing callbacks aren't called. - After Spot callbacks, cleans up local Actor and Spot scope, owner record, descriptor, listener, and transport, in order.
- If finished within the deadline, ends with
Stopped/None; if not, ends withForceStopped/DeadlineExceededorForceStopped/TeardownFailedafter bounded teardown.
When a listener and transport are cleaned up, already-accepted transport callbacks and in-flight read/write operations are completed or cancelled first. In particular, TLS/WebSocket resources and a per-connection write queue aren't destroyed until cancellation completion has been observed on the owning transport execution context. A late callback accessing a destroyed resource or turning an accepted operation terminal twice isn't a valid bounded teardown.
| Operation confirmed first | Handling |
|---|---|
Shutdown's admission seal |
Returns capacity secured on the target, and ends a pending Relocate call with Blocked/ShutdownRequested. |
Relocating publication |
Only confirms the current unit to a terminal state and doesn't start the rest. Preserves published authority; the waiter gets Blocked/ShutdownRequested. |
Shutdown in Relocated only cleans up accepted work and infrastructure. Calling it
directly from Serving doesn't move objects.
While Relocated, the source retains Message Follow routes and the descriptors, peer
connections, and listeners required to forward sends and requests that still arrive at
the old address, and also keeps each unit's cutover retransmission copy until its
retransmission window (§13) ends. A deployment that needs the full configured
MessageFollowDuration calls Shutdown after that period. Calling Shutdown earlier
removes the remaining Message Follow routes and retransmission copies with the source
transport.
For a relocation operation it started, the source runtime publishes SafeToShutdown —
the observation value saying it's safe to shut down — into its own runtime status after
every unit has reached the moment its Message Follow route may be removed (S4, §8) and
every unit's retransmission window (§13) has ended. Both conditions are events happening
on the source, so no other node's clock is needed for this judgment. The value is not a
completion ACK sent by the target or anyone else — the source publishes it and other
parties observe it; the specific status surface is owned by
Runtime monitoring. A deployment
orchestrator can confirm it via status queries and observation of changes before calling
Shutdown. Calling Shutdown before publication is also allowed, but as the first
paragraph of this section says, the remaining Message Follow routes and retransmission
copies disappear with it — a request from a sender that still caches the previous route
can end with Unavailable.
Descriptor and owner lease keep renewing during Draining. To avoid losing owner
eligibility before already-accepted requests, relocation, and session route changes
finish, lease use ends only after all work finishes. The cleanup order is as follows.
Information a fanout publisher publishes to the Store — its endpoint, identity, and run generation — is called a fanout publisher descriptor.
- While keeping Actor membership and local instances, finishes the Spot closing callback and cleans up local scope.
- Only the source holding current authority changes or removes owner and relocation target records to the next state.
- Releases the MeshNode, ClientServer server, and fanout publisher descriptor and owner lease.
- Closes peer connections, listeners, executors, and binding transport.
A language supporting standard cooperative cancellation passes a cleanup cancellation
signal representing the remaining deadline to the Spot closing callback. An
already-accepted handler's token isn't reused. A callback exception is
ForceStopped/TeardownFailed; deadline expiry is ForceStopped/DeadlineExceeded.
Callback execution isn't guaranteed on hardware failure or SIGKILL. It doesn't
guarantee that a different runtime automatically takes over an interrupted relocation or
cleanup.
15. Admission per State¶
The way the Framework picks one Server candidate among several under the same ChannelName is called select-one. A call where the caller directly specifies a node RID is Node direct. A feature that sends a message to several Spots participating in the same Channel is Logical Multicast.
The path from the source runtime to the current owner is called an owner route.
| Public feature | Relocating |
Relocated |
Draining |
|---|---|---|---|
| ChannelName select-one | Excluded from new selection, but existing direct owner routes are kept. | Excluded from new selection; infrastructure connections are kept. | Closes new admission; only already-submitted operations process to a terminal state. |
| Logical Multicast | Excluded from the new target snapshot; already-accepted submissions are kept. | Excluded from the new target snapshot. | Closes new admission; only already-accepted submissions are processed. |
| Node direct application request | An existing owner's requests are accepted until unit seal. | No local stateful owner, so new requests aren't accepted. | New requests end with a shutdown result. |
| Node direct infrastructure control | Keeps accepting relocation, completion, binding, and recovery control. | Keeps accepting monitoring and shutdown control. | Only accepts the control needed for the termination barrier, up to the deadline. |
| Spot/Actor direct | Keeps processing payload and timers until unit seal. | No local stateful owner, so new payload isn't accepted. | Rejects new payload; only already-accepted turns are processed. |
| Spot/Actor create and join | Rejects new owner and membership admission. | Keeps the same rejection. | Keeps the same rejection. |
| Instance Spot placement | Excluded from new target claims; existing direct routes are kept until seal. | Excluded from new target claims. | Only activations accepted before seal are processed to a terminal state. |
| STREAM | Excluded from new binding; existing sessions are handled via unit barrier. | Excluded from new binding; infrastructure connection is kept. | Doesn't accept new sessions; only pending reply and binding barrier are processed. |
| ClientServer server | Excluded from new selection; accepted handlers and reply routes are kept. | Service connection is kept but excluded from new selection. | Closes handler admission; only accepted requests' reply routes are kept. |
| Classic fanout publisher | Doesn't create new automatic subscriber connections; processes accepted events. | Infrastructure is kept but new publish admission isn't accepted. | Closes publish admission; only accepted events are processed. |
An already-accepted request only ends once, via reply, error, timeout, or shutdown. Even while an application callback waits, infrastructure execution keeps proceeding with request completion, peer lifecycle, recovery, and session binding. An observer or monitoring callback doesn't own a claim that blocks maintenance.
16. Observability Information¶
State and relocation result changes are observed via
zlink.runtime.host.relocation_changed; shutdown result changes via
zlink.runtime.host.termination_changed. A terminal status isn't overwritten by an
intermediate status; the observer's terminal retention bound and discard rule are owned by
runtime monitoring §7.2. Relocation events and a limited set of diagnostic states include mode and
effective target version. Version isn't added as a metric label.
Host state and terminal results are checked in host status and structured logs. When aggregation is needed, host state, relocation mode/outcome/reason, and shutdown outcome/reason are recorded on the instruments defined by Runtime metrics. Object relocation instruments and host-wide operation instruments use different names.
The point definitions and measuring parties for the per-unit points S0–S4 and the three
interval metrics (source stop S0→S1, target resume S2→S3, route convergence S1→S4) are
fixed by §8. The status surface of the SafeToShutdown observation value (§14) is owned
by Runtime monitoring; the
interval-metric instruments and the cutover_timeout counter, which counts cutover-wait
fallbacks, are owned by
Runtime metrics.
The global string address for finding a Spot system-wide is called a Spot ID. Metric labels don't include Actor ID, Spot ID, node RID, endpoint, session ID, or relocation ID. Individual blockers and relocation states are checked via count-limited diagnostic queries and traces. Telemetry provider failure doesn't block operation progress. The full observability contract is owned by Runtime monitoring and Runtime metrics.
17. Implementation and Contract-Test Verification Requirements¶
The following is confirmed using only the public surface (Relocate/Shutdown's
options and return values, host status/observe stream, Location Store record lookup,
metrics/events). Verification of one unit's handoff mechanics is owned by
Complete Actor And Spot Relocation Flow
§13;
this section covers only items observable at the host level. Each item maps to one test.
Mode and target selection
- Planned maintenance selects only the same version, and rolling update selects only the requested higher version.
- Version applies before capacity and weight, and excludes the same wave.
- The Framework proceeds only once that Core peer is
Readyon every Mesh. If there's no target, it waits and blocks on manual topology.
Lifecycle and concurrency
- A blocked preflight keeps
Serving; success becomesRelocatedwith infrastructure kept. Shutdownis called separately, with a default deadline of 30 seconds.- Caller cancellation ends only the waiter, and doesn't change admission in an invalid runtime state.
- Relocations with the same option share one operation, as do concurrent shutdowns.
- A different relocation option ends with
OperationInProgress; shutdown during relocation ends withShutdownRequested; repeated calls return the same terminal result.
Inventory and batches
- Entry Spot Actors,
PerActorshells and member Actors,SpotWideaggregates, Instance Spots, and standalone Actors each appear exactly once, while the Entry Spot instance is excluded. - The
PerActorshell, Actor, and aggregate batch order and its dependencies are kept, and only independent units in the same batch start concurrently. - No relocation-specific cap on unit count, participant count, or relay record count is required. Reaching the in-flight payload budget appears only as waiting before the seal, and doesn't fail a unit that already started.
SpotWide/PerActor handoff
- A
SpotWideUser Spot aggregate commits at once and moves queue, timer, and pending tick together. - The target dispatcher registers the Spot and every member Actor in the same relocation temporary queue group while preserving each record's actual target.
- After every Restore and aggregate commit, the target moves saved work, pre-boundary relay,
and remaining temporary work into the real queues in order, switches to the regular route,
then completes
OnRelocationReadyCompletedand opens dispatch. No participant's application work runs before that switch. - A Message Follow route is removed after
MessageFollowDurationindependently of command 44 application. An Instance Spot isn't secretly re-created. - Entry Spot and
PerActorUser Spot move only Actors independently and don't call a Spot adapter or membership callback. - After the Spot authority transition,
ToSpot/Create/Join use the target, andToActoruses each Actor's current owner. - Spot and Actor relocation temporary queues are registered independently. The order of transferred existing work, temporary work, and direct work after the transition is preserved, and resending the same relocation request doesn't create the temporary queue and Restore twice.
Interruption budget and pacing
- For each Actor, Instance Spot,
SpotWideUser Spot, andPerActorSpot direct message, measure a source-local 1 second from when the source blocks new work through the one-way cutover submit's success or failure terminal. - Don't create a target processing-start ACK, and don't treat exceeding this as a failure, rollback, or retry condition.
- After the host deadline, don't start a new unit, and process an already-started unit to a safe terminal state.
- When the in-flight payload budget is full, a new unit waits before the source admission seal, and the waiting Actor/Spot keeps processing messages meanwhile. The coordinator doesn't set a separate cap on concurrent unit count.
- If the connection is re-established within the retransmission window, the source resends the pre-boundary relay batch and cutover, the target replaces partially received staging with the complete retransmitted batch, and once the window ends, the copy is cleaned up exactly once and no further retransmission occurs.
Metrics and SafeToShutdown
- Source stop time (S0→S1), target resume time (S2→S3), and route convergence time (S1→S4) are measured on the clock of the node where each point happens, and no metric directly subtracts timestamps of different nodes.
SafeToShutdownisn't published before every unit reaches S4 and every unit's retransmission window ends, and neither judgment uses another node's clock.Shutdownbefore publication is also allowed, and its result matches §14's route and copy cleanup.
Failure and cleanup
- Only on an explicit abort before relay-ready is accepted is the target temporary queue discarded without running and the source's original work restored to the queue. After that boundary, source isn't restored regardless of cutover-submit result.
- A request's terminal result isn't duplicated across two runtimes.
- If the same target runtime fails after owner commit, it doesn't roll back to the source or automatically pick a different target.
- Returns the specific
Blockedreason, completes the terminal result exactly once, and performs bounded teardown if descriptor rollback can't be confirmed. Automatic relocation resumption after process termination isn't a verification target. - Lease keeps renewing until the barrier finishes, and an accepted request completes exactly once.
- Callback failure is classified with a defined reason, and state, outcome, reason, event, and metric match the wire values.
- Topology cleanup doesn't change a different authority.
Location·Relocation Topic Table Of Contents · Spec Table Of Contents · Previous: 04. Complete Actor And Spot Relocation Flow · Next: 06. Failure Handling And Failover Scope