---
title: "Replication apply seam (IReplicationApplier)"
url: "https://nsta1.github.io/Orleans.Lattice/docs/lattice.replication/replication-apply.html"
source: "https://github.com/NSTA1/Orleans.Lattice/blob/release/9.9/docs/lattice.replication/replication-apply.md"
package: "Orleans.Lattice.Replication"
version: "9.9.0"
documents: "Orleans.Lattice 9.9.0 (release line 9.9)"
built: "2026-10-04"
all-pages: "https://nsta1.github.io/Orleans.Lattice/llms.txt"
bundle: "https://nsta1.github.io/Orleans.Lattice/docs/lattice.replication/llms-full.txt"
---
# Replication apply seam (`IReplicationApplier`)

Part of the [Replication documentation](README.md).

`IReplicationApplier` is the public, in-process inbound seam over the per-tree apply pipeline. It installs a single `WalRecord` authored on a remote cluster onto the local tree while preserving the remote cluster's origin id end-to-end and, except where [Source-HLC and origin preservation](#1-source-hlc-and-origin-preservation) notes otherwise, its `HybridLogicalClock`, and it filters re-delivery via a snapshot-pinned causal floor plus a shadow-forward identity cache and per-key last-writer-wins idempotence, so at-least-once transports become at-most-once apply.

The contract is deliberately neutral: there is no transport binding, no per-peer state, no ack envelope. It is the seam custom transports and integration tests plug into.

## API

The interface and result type live in `Orleans.Lattice.Replication`:

```text
public interface IReplicationApplier
{
    Task<ApplyResult> ApplyAsync(
        WalRecord entry,
        CancellationToken cancellationToken = default);

    Task<ApplyResult> ApplyBatchAsync(
        IReadOnlyList<WalRecord> entries,
        CancellationToken cancellationToken = default);
}

public readonly record struct ApplyResult
{
    public bool Applied { get; init; }
    public HybridLogicalClock HighWaterMark { get; init; }
    public bool Deferred { get; init; }
}
```

| `ApplyResult` member | Semantics |
|---|---|
| `Applied` | `true` when the entry was merged onto the local tree; `false` when the entry was filtered out as a re-delivery (its `Timestamp` was at or below the origin's pinned snapshot floor, or its identity tuple hit the shadow-forward cache), parked on the causal-apply buffer to await its dependencies, deferred by a restore saga's receive fence, acknowledged as a no-op because it is a tombstone-reap envelope, or rejected as inapplicable (its `OriginClusterId` matched the local cluster id and would have looped, its tree is not enrolled for replication on this receiver, its wire merge mode disagreed with the locally-resolved mode, or the tenant-isolation gate refused it). For batch calls, `true` if **any** entry in the batch was newly merged. |
| `HighWaterMark` | For point applies (`Set` / `Delete`) this is the per-origin HWM after the call - equal to `entry.Timestamp` when `entry.Timestamp` advanced the frontier, or the current HWM otherwise (including when `Applied` is `false`). For range deletes, saga terminal marks, tombstone-reap envelopes, local-origin no-op rejections, receive-fence deferrals, and receiver-side enrollment / merge-mode / tenant-isolation rejections - none of which reads or advances the HWM - this is `HybridLogicalClock.Zero`. For batch calls, the pointwise maximum HWM across every distinct origin in the batch. |
| `Deferred` | `true` only when a cross-cluster restore saga's durable receive fence has paused inbound apply for the tree, so the entry (or run) was **not** applied and must be re-shipped once the fence lifts. For batch calls, `true` if **any** entry or run in the batch was deferred. Receive paths turn a deferred result into a not-accepted, cursor-preserving ack; every other `Applied == false` outcome is terminal and lets the sender advance past the entry. |

## Apply semantics

The applier composes these concerns for every call:

### 1. Source-HLC and origin preservation

For `LwwRegister` mode point applies route through the core library's apply seam, which persists the entry's value with the supplied `Timestamp` and `OriginClusterId` **verbatim** - no fresh local HLC is stamped. This is what unlocks transitive replication (A -> B -> C with A's HLC intact) and deterministic LWW resolution against concurrent local writes.

For typed CRDT modes, a steady-state entry carries the producer's typed delta in `WalRecord.Delta` (authored via the accessor at commit time). The applier forwards that delta verbatim through the same `CrdtDelta`-recording grain seam used by both the batch path and a locally-authored CRDT write, preserving the source `OriginClusterId` and folding the delta into the visible state in one grain turn. The source `Timestamp` is preserved only when the entry is folded as part of a multi-entry batched run, where the batch path re-stamps each item with its source HLC; a per-entry apply - a single-entry batch, an `OrMap` entry, a causal-buffer drain, or a dead-letter replay - takes a fresh local HLC at the merge point. The receiver therefore records a `CrdtDelta` revision with per-member ADDED/REMOVED changes - identical history fidelity to a local write - rather than a flattened full-value `Set`. The fold is wrapped in a `LatticeOriginContext.With(originClusterId)` scope so the receiver's commit-time observer publishes the foreign origin and the producer-side ship loop filters the resulting entry out. A bootstrap committed-projection row carries the full state in `WalRecord.Value` with no delta; it has no per-delta shape, so it folds via state-based merge under optimistic concurrency and stays a full-state set, written at a fresh local HLC and without the row's `ExpiresAtTicks`, so the key is stored as a durable entry. For `OrMap` mode the receiver resolves the concrete `(TKey, TValue)` shape through `CrdtShapeRegistry` (populated by `ISiloBuilder.AddOrMapShape<TKey, TValue>`); an `OrMap`-mode apply against an unregistered tree faults with a clear configuration-error message.

Range deletes carry the producer's issue HLC: the authoring cluster pins one HLC for the whole range-delete fan-out, stamps every tombstone it writes with it, and publishes it on the `WalRecord`. The receiver walks the leaf chain locally and pins every tombstone it writes to that same HLC, so a range delete authored at `T` cannot overwrite a foreign-origin write whose HLC is strictly greater than `T`; the remote `OriginClusterId` rides through an ambient `LatticeOriginContext` scope so the receiver-side change-feed observer publishes it on every emitted `LatticeMutation`. A predicate-filtered range delete also ships the explicit set of keys it matched, and the receiver tombstones exactly those keys instead of re-deriving membership from the range bounds. Only an entry persisted by an older producer carries `HybridLogicalClock.Zero`; for such an entry the receiver falls back to stamping each tombstone with a freshly-ticked local HLC.

### 2. Snapshot-pinned causal floor (and per-origin high-water-mark)

The applier tracks two distinct per-`(TreeId, OriginClusterId)` quantities on the high-water-mark grain:

- The **per-origin high-water-mark** is the max-applied source HLC. It advances monotonically after every successful point apply and drives FIFO / causal ordering, observability, and the bootstrap handoff. It is **not** a drop criterion for steady-state point writes.
- The **pinned causal floor** is written only by `PinSnapshotAsync` (the bootstrap handoff) and records the snapshot frontier below which every mutation is already contained in the pinned snapshot. When no snapshot has been pinned the floor is `HybridLogicalClock.Zero`.

Before applying a point entry the applier reads the pinned floor; if `entry.Timestamp <= floor` the call is a no-op (`Applied = false`), because everything at or below a pinned snapshot frontier is already durably present. Otherwise the entry is admitted and its at-most-once guarantee rests on per-key last-writer-wins idempotence at the leaf (a superseded or duplicate write merges to the same state) plus the shadow-forward identity cache below. After a successful point apply the HWM advances monotonically; a laggard's lower advance becomes a no-op.

This is deliberately **not** a `entry.Timestamp <= hwm` drop. The per-origin HLC stream is not monotonic in write-ahead-log / ship order: HLCs are stamped per leaf (each leaf carries its own clock) and the write-ahead-log partitions by key hash, so many leaves interleave in one partition and a genuinely-new point write can arrive with a source HLC below the running max-applied HLC. Dropping such an entry on a scalar `hwm` comparison silently strands it - the cross-cluster data-loss regime this seam is designed to avoid. A legitimately out-of-order-but-new entry that lands below the current HWM is applied and increments `apply.fifo_violations` (observability only). Typed CRDT modes consult the floor and advance the HWM the same way; state-based merge is naturally idempotent, so the floor gate just short-circuits redundant grain calls for the below-snapshot backlog.

Range deletes bypass the floor by design. Range applies are naturally idempotent at the leaf layer: re-running a range delete on already-tombstoned keys merges to the same state, so dedupe is unnecessary.

### 3. Local-origin rejection

A `WalRecord` whose `OriginClusterId` matches the local cluster id is rejected as a no-op (`Applied = false`). This check is the receiving cluster's only enforcement that a local-origin entry is never applied back onto its authoring cluster - it is not defence-in-depth behind the sender's outbound origin filter. That filter runs on the sending cluster and decides what the sender ships, not what this receiver accepts, and a hand-built apply pipeline or a test can hand the applier such an entry directly.

### 4. Receiver-side enrollment and merge-mode gate

Before the concerns above run, the applier gates every inbound entry against this receiver's own per-tree replication configuration, re-resolving the tree's enrollment and merge mode locally instead of trusting the wire. The peer-supplied `OriginClusterId` is unverified and `WalRecord.Mode` is a peer-controlled header field, so neither is taken on faith. The gate yields two rejections:

- **Not enrolled here (dropped).** A tree that is not enrolled for replication on this receiver is dropped: the call returns `Applied = false` with `HighWaterMark = HybridLogicalClock.Zero`, records the apply-duration outcome `rejected-not-replicated`, and is **not** dead-lettered. A non-enrolled tree id is peer-controlled, so parking it in a dead-letter queue would let a hostile peer spawn unbounded dead-letter-queue activations; dropping keeps the rejection cheap and bounded. This closes the gap where a peer that holds the mesh secret could otherwise write a tree the cluster deliberately kept cluster-local by not enrolling it - the reserved-prefix core-tree guard covers only the `_lattice_` core trees, not the `sys-`-prefixed authorization and identity trees.
- **Enrolled but wire mode mismatched (dead-lettered).** A tree that *is* enrolled but whose peer-supplied wire mode disagrees with the locally resolved merge mode is dead-lettered: the call returns `Applied = false` with `HighWaterMark = HybridLogicalClock.Zero`, the entry is enqueued to the tree's dead-letter queue tagged `mode_mismatch`, and the apply-duration outcome `rejected-mode-mismatch` is recorded. The tree is enrolled and therefore a bounded id, so parking the entry cannot be abused to spawn unbounded activations. Re-resolving the mode locally rather than trusting the wire field stops a peer from overriding the local merge algebra by shipping a different mode.

The merge mode is always re-resolved locally through the receiver's per-tree resolver (`ILatticeReplicationContext.ResolveMergeMode`, falling back to the raw `LatticeReplicationOptions.ReplicatedTrees` map); the wire `Mode` field is only ever compared against that resolution, never adopted. An applier with neither an injected replication context nor a `ReplicatedTrees` map has no enrollment signal, so the gate fails closed: every inbound entry is dropped as `rejected-not-replicated` (not dead-lettered) and a one-time warning is logged. Production registers the replication context, so the gate is always evaluable there.

A run the gate rejects - either way, or for want of an enrollment source - is also never recorded as inbound contact in `ReplicationPeerStats`, on any receive path, so a peer cannot plant a tree id of its choosing in the peer statistics or the peer-status report. The inbound half of that state is additionally capped, because the origin id of an admitted run is still the peer's own claim; see [observability](observability.md#bidirectional-peerlast_contact_seconds-and-the-liveness-probe).

Two further receiver-side gates run after an entry clears enrollment, before any high-water-mark read:

- **Tenant isolation (dead-lettered).** When tenancy is on, the owning tenant is derived from the tree id alone - never from a wire field - and an entry whose tenant does not exist here, is not resident in the region serving this receiver, or has been suspended or disabled is refused. It is dead-lettered with the `foreign_tenant`, `tenant_offline`, or `tenant_suspended` reason, records the matching `rejected-foreign-tenant` / `rejected-tenant-offline` / `rejected-tenant-suspended` apply-duration outcome, and leaves the high-water-mark unchanged. The rejection is not a deferral, so the receive path still acknowledges the batch and the sender advances past the entry rather than re-shipping it; once the tenant exists, becomes resident, or is reinstated, the parked entry can be replayed from the [dead-letter queue](dead-letter-queue.md). With tenancy off the gate is inactive and costs nothing.
- **Restore receive fence (deferred).** While a cross-cluster restore saga has paused inbound apply for the tree, the entry is not applied: the call returns `Applied = false` with `Deferred = true`, and the sender keeps its cursor and re-ships once the fence lifts. A single-entry apply records the deferral under the `dedup` apply-duration outcome; the batch path defers a multi-entry run whole and records no apply-duration sample for it.

The batch path applies the same classification once per run, except that it checks the restore receive fence first: a run for a fenced tree is deferred whole before it is classified. The wire mode is part of the run key - a run is a contiguous `(TreeId, OriginClusterId, Mode)` segment, so a mode change starts a new run that is classified on its own - and the representative first entry therefore classifies the whole run. A rejected run neither merges nor advances the per-origin high-water-mark; every entry still records its matching apply-duration outcome so per-entry receiver observability is preserved, while a single warning is logged per run rather than per entry to avoid a log-flood amplification from a hostile peer.

### 5. Shadow-forward dedupe cache

A structural rewrite that shadow-forwards a user write into a different shard - a shard split, or a shard consolidation (merge), which reuses the split's shadow-write window - generates a duplicate-emit pair: one entry from the originating shard's commit, one from the destination shard's commit, both carrying identical `(originClusterId, timestamp, key, op)` identity tuples, because the forward carries the original last-writer-wins value and its HLC. An atomic-write abort is not a source of such pairs: it issues no per-key rollback writes, only an abort recorded in the tree's transaction registry and `TxAbort` terminals that discard the prepared writes. Because the pinned-floor gate does not drop above-floor point writes, both deliveries reach the apply path; the identity cache is what collapses the redundant second grain hop before it happens.

The applier holds a per-tree bounded FIFO cache of recently-applied identity tuples (`LatticeReplicationOptions.ShadowForwardDedupeCacheSize`, default `4096`, validator floor `64`). The cache is consulted *after* the pinned-floor gate so floor-deduped entries do not pollute it (which preserves operator-driven re-pin semantics where lowering the pinned floor must re-admit previously-deduped identity tuples). On cache hit the apply is suppressed with `Applied = false` and the apply-duration histogram is tagged `outcome=shadow-forward-dedup`. Range deletes bypass the cache - they are applied before it is consulted - and the leaf layer is naturally idempotent for range applies.

The cache is a fast-path optimisation, not the correctness backstop. It suppresses the duplicate-emit pair before the apply grain hop; if an entry it would have caught has been evicted under sustained churn, the duplicate still re-applies to the same state under per-key last-writer-wins idempotence at the leaf, so an eviction can never cause a divergent re-merge - it only costs one redundant grain hop.

### 6. Causal-dependency gate

Entries authored with causal-plus tracking carry a `VectorClock` frontier. Before applying such an entry the receiver fetches its local vector clock and checks that every component of the entry's frontier is dominated-or-equal locally; an entry with an unsatisfied dependency is parked in the per-tree bounded causal-apply buffer and retried each time a later apply advances the local clock. The buffer is bounded by `CausalBufferMaxEntries` (default `1024`) and `CausalBufferMaxBytes` (default 16 MiB); when a new park would exceed either bound, the oldest parked entries are evicted to the tree's dead-letter queue with reason `hlc_skew`, and each evicted entry's shadow-forward identity reservation is released, so a dead-letter replay or a peer re-delivery of it is applied rather than suppressed as a duplicate. Two frontier components are exempt from the check:

- **The entry's own origin diagonal.** The per-origin high-water-mark tracks that origin's own FIFO progression, so requiring the local clock to dominate the diagonal would deadlock the very entry being applied.
- **The receiver's own cluster id.** The receiver-side local vector clock tracks only *foreign*-applied frontiers - it never advances its own diagonal - but the receiver durably holds every write it authored itself, so any dependency on one of the receiver's own writes is trivially satisfied. Without this exemption a peer entry whose frontier references a write the receiver originated (for example, site C's post-partition write that causally follows site A's pre-partition write, once an A-C partition heals) would park forever against a perpetually-zero self-component and stall convergence.

## Validation

`ApplyAsync` throws `ArgumentException` when:

- `entry.TreeId` is null or empty.
- `entry.OriginClusterId` is null or empty.
- `entry.Op == Set` on an `LwwRegister` entry and `entry.Value` is null, or `entry.Op == Set` on a CRDT-mode entry that carries neither a typed `Delta` nor a full-state `Value`.
- `entry.Op == DeleteRange` and `entry.EndExclusiveKey` is null, or the range delete carries atomic-batch metadata (`AtomicBatchSize > 0`).
- A prepared entry (`IsPrepared == true`) carries an empty `TransactionId`, or a prepared `Set` carries a null `Value`.
- A saga terminal mark (`TxCommit` / `TxAbort`) carries no usable shard index or an empty `TransactionId`.

`InvalidOperationException` is thrown when:

- `entry.Mode` has no apply rule, or `entry.Op` is not a point-apply kind. The merge-mode gate normally intercepts an unknown wire mode first, dead-lettering it as `mode_mismatch`, because it cannot match the locally resolved mode.
- An `OrMap` tree has no registered `(TKey, TValue)` shape.
- A typed CRDT state-merge exhausts its CAS retry budget under sustained contention on the target key.

`OperationCanceledException` is thrown when the supplied `CancellationToken` is already cancelled or fires during a grain call.

## Registration

`AddLatticeReplication` registers the default `IReplicationApplier` implementation as a silo-side singleton:

```csharp verify
siloBuilder.AddLatticeReplication(o => o.ClusterId = "site-a");
```

Resolve it from inside a silo-side service (typically a transport adapter or a hosted-service inbound pipeline) via constructor injection on `IReplicationApplier`. The applier is not exposed on the cluster client - it is a silo-local seam by design, because the apply path must run inside the cluster that owns the receiving tree.

## Threading and concurrency

The applier is a silo-wide singleton. It holds no per-call state, only process-local per-tree structures - the shadow-forward dedupe cache, the causal-apply buffer, and the FIFO-diagnostic tracker - and all durable coordination flows through the per-origin high-water-mark grain (single-threaded under Orleans turn semantics) and the per-tree apply grain (`StatelessWorker`). The HWM grain serialises each individual read and advance, but it does not serialise whole `ApplyAsync` calls: two concurrent deliveries for the same `(tree, origin)` pair can both pass the floor gate before either advances the HWM, which is why the shadow-forward identity cache and per-key last-writer-wins idempotence at the leaf, not the HWM, absorb a concurrent duplicate. Calls for different pairs are independent.

## Bootstrap handoff

The pinned causal floor is the explicit handoff contract for the bootstrap protocol: a newly-bootstrapped peer calls `PinSnapshotAsync`, which atomically installs both the per-origin HWM and the pinned floor at the snapshot's authoring frontier, then resumes incremental replication from that pinned frontier with exactly-once apply guarantees across the snapshot / incremental boundary. `PinSnapshotAsync` *replaces* the floor rather than max-merging it, so a peer that rewinds to an older snapshot (a pinned value lower than the receiver's prior frontier) correctly lowers the floor and re-admits entries above the new, lower cut. The grain therefore exposes this unconditional pin alongside the monotonic HWM advance and a `GetPinnedFloorAsync` read.

## Caveats

- **Range deletes preserve the producer's issue HLC, not per-leaf HLCs.** The wire carries the single HLC the producer pinned for the whole range delete and the receiver stamps every tombstone with it, so LWW resolution against a concurrent write compares against the producer's authoring HLC rather than the receiver's local clock. An entry persisted by an older producer carries `HybridLogicalClock.Zero` and falls back to fresh local HLCs, where LWW resolution depends on the local clock at apply time. Idempotence at the leaf layer is what makes a re-applied range delete safe.
- **The HWM is per-origin, not per-shard.** A receiver applying entries from origin `X` against a tree split into N shards advances a single HWM row keyed `(tree, X)` regardless of which shard the entry targets. This is intentional: the HWM contract is the bootstrap-handoff seam, and bootstrap operates per-origin not per-shard.

## Batch apply path

Inbound transports deliver batches of `WalRecord` records, not single entries: a 256-entry gRPC push from a single producer is one network round-trip carrying 256 mutations. `ApplyBatchAsync` is the seam that lets the receiver process such a batch as one logical operation rather than 256 independent `ApplyAsync` calls - for each contiguous run of entries sharing a tree, origin, and merge mode it reads the high-water-mark and the pinned causal floor once, merges the run's plain point writes in one batched grain call instead of one apply call per entry, advances the high-water-mark once, and drains the causal-apply buffer once at the end of the run instead of after every successful apply.

The default-interface-method body provides backward-compatible semantics: it loops over `ApplyAsync` and aggregates the per-entry results - `Applied` if any entry was newly merged, the pointwise-maximum `HighWaterMark`, and `Deferred` if any entry was deferred - so any custom `IReplicationApplier` written before the batch seam existed continues to work without changes, and a restore saga's receive-fence deferral returned by its `ApplyAsync` still reaches the receive path as a deferred, cursor-preserving result. The shipped applier overrides the batch path with the optimised implementation described below.

### Run grouping

The optimised batch path walks the inbound list and identifies maximal contiguous runs of entries that share the same `(TreeId, OriginClusterId, Mode)` tuple (a well-formed batch carries one merge mode per run, so including the mode never splits a legitimate run). For a 256-entry batch shipped by a single producer the entire batch is one run; for an interleaved batch (e.g. a snapshot recovery merge that intersperses entries from two origins) the path emits one run per contiguous group, each amortised independently. Within a run:

- A single `GetAsync` reads the persisted per-origin HWM and a single `GetPinnedFloorAsync` reads the pinned causal floor at the start of the run.
- The pinned floor is constant for the run, so every entry in the run is dedup-tested against the same floor with no further round trips; there is no in-batch running-HWM accumulator (a below-max-applied-HLC entry is a genuine write under non-monotonic per-origin HLC, not a duplicate, so it must not be dropped mid-run).
- Causal-dependency entries fetch the local vector clock lazily on first use and reuse it until an apply has occurred, at which point a `localVcDirty` flag forces a re-fetch on the next causal-dep check.
- A single `TryAdvanceAsync` advances the persisted HWM to the highest applied HLC at the end of the run.
- The causal-apply buffer is drained once, if the run advanced the persisted HWM.
- Non-prepared `LwwRegister` `Set` / `Delete` entries that pass classification are deferred and merged in one batched grain call per run, and non-prepared typed-CRDT `Set` entries that carry a delta (every CRDT mode except `OrMap`) fold in one batched delta call. Range deletes, saga terminal marks, prepared entries, CRDT-mode deletes, and `OrMap` or delta-less CRDT entries flush the pending batch first and take their own per-entry apply hop.

For a 256-entry single-origin `LwwRegister` batch this collapses roughly 4 x 256 = 1,024 grain round-trips on the per-entry path (a high-water-mark read, a pinned-floor read, a point apply, and a high-water-mark advance per entry) to about four (one of each, with the 256 point applies folded into one batched merge) - the dominant receiver-side cost on every inbound push.

### Preserved per-entry semantics

Every classification the per-entry path produces survives the batch path, except the tombstone-reap no-op (the last bullet):

- **Range-delete entries** bypass the pinned-floor gate and apply unconditionally (a range apply is naturally idempotent at the leaf layer).
- **Local-origin runs** classify every entry as `Dedup` with `HighWaterMark = HybridLogicalClock.Zero` and emit no grain calls.
- **Below-floor dedup** against the run's pinned causal floor is bit-equivalent to per-entry dedup against a freshly-read floor, because the floor is written only by `PinSnapshotAsync` and is therefore constant across a run.
- **Causal-park** is exercised per-entry; only the local-vector-clock fetch is lazy.
- **Per-entry instrumentation** (`ApplyDuration`, `ApplyLag`, `ApplyFifoViolations`) is recorded inside the per-entry loop so observability is preserved verbatim.
- **Single-entry batches** defer to `ApplyAsync` so behaviour is bit-identical with the legacy receiver for the trivial case.
- **Tombstone-reap envelopes** are not acknowledged as `dedup` on the batch path: it has no no-op branch for them, so a multi-entry batch carrying one faults on that entry, because the point-apply step has no rule for it. The dead-letter-tracking decorator then falls back to per-entry apply, which does acknowledge the envelope as a no-op. The sender never ships these envelopes, so only an older shipper or a hand-built caller can deliver one.

### Failure model

Per-entry failures inside the batch surface as `ApplyAsync`-equivalent exceptions. The gRPC receiver endpoint wraps the batch call in a transport-level exception so the sender's backoff/retry loop kicks in for the whole batch - partial-batch acceptance is not a guarantee the seam offers. The dead-letter-tracking applier decorator falls back to per-entry routing when any entry in the batch already has retry history, or when the batch call throws part-way, so its DLQ accounting is exact. On that per-entry fallback (and for a single-entry batch) the decorator itself records the inbound per-peer contact, which the batch path otherwise records once per run; entries from runs the failed batch attempt already recorded can therefore record contact a second time for the same push (see [Observability](observability.md#bidirectional-peerlast_contact_seconds-and-the-liveness-probe)).

### Parallel apply across independent runs

Under multi-tree load the per-run walk can serialise otherwise-independent work: a batch that interleaves runs from several trees applies them one after another even though they share no per-tree state, inflating apply latency and `apply.lag` (which now also drives receiver back-pressure, so slow applies translate directly into sender throttling).

`LatticeReplicationOptions.ApplyMaxParallelRuns` bounds how many **independent** runs the batch path may apply concurrently. Independence is defined at the **tree** granularity:

- Runs targeting **distinct trees** may apply in parallel. Distinct trees share no per-tree state - separate causal-apply buffers, shadow-forward dedupe caches, high-water-mark grains, and apply grains - so concurrent apply cannot reorder or interleave their work.
- Runs that **share a tree** (different origins of the same tree) stay in one ordered group and apply strictly sequentially in write-ahead-log order. The per-tree causal-apply buffer and shadow-forward dedupe cache are shared across a tree's origins, so keeping same-tree runs serialised guarantees those structures observe the exact access order the fully-sequential path produces.

Parallelism is therefore only ever introduced **across** independent runs, never **within** one. Every within-run ordering invariant holds unchanged regardless of the configured degree of parallelism: per-origin FIFO, the causal dependency gate and its bounded buffer, per-origin high-water-mark monotonicity, and atomic-batch (saga) apply boundaries. A multi-entry run still collapses to a single batched merge; an atomic batch still applies as a unit on its owning run.

The effective degree of parallelism for a given batch is the largest `ApplyMaxParallelRuns` configured for any tree in that batch (the option resolves per tree), clamped to the number of distinct trees present in it, and is bounded by a per-batch semaphore so concurrency can never amplify local WAL saturation beyond the configured cap. It is surfaced on the `apply.parallel_runs` histogram (see [observability](observability.md)).

**Default posture: fully sequential.** `ApplyMaxParallelRuns` defaults to `1`, which is exactly the historical behaviour - the batch path walks every run in order and awaits each before the next. The single-tree batch (the overwhelmingly common inbound shape, since the transport ships per-`(tree, peer)`) always takes the allocation-free sequential walk regardless of the configured value, because cross-tree parallelism is moot when there is only one tree. Raise the value conservatively, per workload, only after validating parallel apply for that topology.

## Cross-cluster atomic visibility - receiver seam

`SetManyAtomicAsync` sagas authored on the source cluster ride the
standard WAL replication transport: every prepared per-key write
emits a `Set` / `Delete` `WalRecord` with `IsPrepared = true` and a
non-empty `TransactionId`, and the saga's terminal phase emits one
`TxCommit` (or `TxAbort`) `WalRecord` per touched shard. The shipper
preserves these records verbatim; the receiver seam interprets them
through three additional internal apply hops:

| Apply hop | Wire trigger | Receiver behaviour |
|---|---|---|
| Prepared set | `Op == Set` && `IsPrepared == true` | Stages the write under the saga's `TransactionId` in the destination leaf's per-tx pending bucket. The visible projection is unchanged - public readers (`GetAsync`, `KeysAsync`, etc.) do not observe the prepared entry. |
| Prepared delete | `Op == Delete` && `IsPrepared == true` | Stages a tombstone under the saga's `TransactionId` in the same pending bucket. The pre-saga value remains visible to public readers until the terminal arrives. |
| Transaction terminal | `Op == TxCommit` or `Op == TxAbort` | Records this per-source-shard terminal arrival in the per-tree transaction registry (keyed by txid, source shard index, commit/abort outcome, and atomic shard count) to tally arrivals. While the tally is not final the registry mark stays unset and the receiver leaves' pending buckets stay in place so reads remain all-or-nothing. Only on the final arrival does the receiver mark the per-tree transaction-registry entry and pre-fan the terminal across the transitive split-forward closure of every observed source-shard in a single parallel hop. On commit every pending entry under the `TransactionId` flips into the visible projection; on abort the pending entries are dropped. |

The batch-apply classifier excludes any
entry with `IsPrepared == true` from the batched LWW fast-path so
prepared `Set` / `Delete` records are always routed through the
per-entry prepared-set / prepared-delete apply hops.
Without this exclusion the prepared writes would commit directly
into the receiver leaf's visible projection and the saga's terminal
mark would find no matching pending entries to flip - so the
cross-cluster reader would observe the prepared write as visible
before the registry gate flipped, purely as a function of whether
the inbound run happened to be batched or single-entry. Unprepared
writes continue to consume the batched merge path.

The per-source-shard arrival tally is the receiver-side
multi-shard atomic-visibility gate. A saga that touched **N** source
shards emits **N** independent terminal records, one per source
shard, that ship through the change feed under independent
backpressure / batching cadences. Each terminal carries the saga's
authoritative touched-shard count in the additive
`WalRecord.AtomicShardCount` slot, which the receiver feeds into
the terminal-arrival tally to compute finality. A receiver
running a pre-gate producer sees `atomicShardCount == 0` on every
terminal, which the gate treats as "no expected-total information"
and falls back to first-terminal-wins semantics - equivalent to the
pre-gate behaviour and wire-compatible across mixed-version
deployments.

The producer-side ship filter explicitly bypasses
per-tree `KeyFilter` and `KeyPrefixes` for `TxCommit` and `TxAbort`
records: a saga whose prepared keys passed the filter must have its
terminal delivered or the receiver-side pending bucket leaks. The
empty-origin guard and the cycle-break filter still run before the
bypass, so a malformed or self-loopback terminal is still rejected.

### Cross-tree terminals (receiver barrier)

A terminal that belongs to a **cross-tree** atomic write
(`IGrainFactory.SetManyAtomicAsync`) carries two additional slots -
`WalRecord.CrossTreeOperationId` and `WalRecord.CrossTreeParticipants`
(the canonical participant tree-id set). The applier threads these into
the transaction-terminal apply hop as a `crossTreeOperationId` plus a receiver-scoped
**wait set**. The wait set is the participant set intersected with the
trees this receiver actually replicates (its per-tree enrollment - the
`LatticeReplicationOptions.ReplicatedTrees` declaration or a runtime-enabled
tree); the tree that received the
terminal is always included. A participant tree not replicated here is
excluded, so a cross-tree batch spanning a mix of replicated and
non-replicated trees stays valid - the barrier completes on the present
subset rather than waiting forever on a tree that never ships here.

Once a tree's per-source-shard gate is final, a cross-tree terminal does
**not** flip that tree's registry directly. Instead the receiver durably
registers the tree's local txid as delegated to a **receiver coordinator
grain** (keyed by
`(originClusterId, operationId)`) and notifies it of this tree's arrival
and commit/abort vote. The coordinator decides only once a terminal has
arrived for every tree in the wait set, committing iff every arrived tree
voted commit. Before the decision, a delegated read on any participating
tree's registry resolves `InFlight` against the coordinator, so every
tree stays invisible (an unreachable coordinator resolves `Indeterminate`,
which likewise keeps the keys invisible); after it, the receiver flips every
participating tree together. The coordinator only ever returns the decision (it never
calls back into a tree grain); the calling tree grain performs the
per-tree finalizes - itself inline, siblings via their apply grains - so
there is no circular wait. A null/empty `crossTreeOperationId` routes the
terminal through the legacy single-tree gate unchanged.

Public readers therefore observe the receiver-side same-cluster
atomic-visibility property end-to-end: at every point in time,
either every key the saga prepared on the receiver is at its
post-saga value (after the commit terminal applies) or none of them
is (during the prepare window or after an abort). The HLC the
visible value carries is the source cluster's HLC verbatim - the
receiver's wall-clock progression does not bump it - so transitive
LWW resolution (A -> B -> C with A's HLC intact) holds across saga
output identically to single-key cross-cluster writes.

What ships today:

- `WalRecord.AtomicBatchSize`, `AtomicBatchIndex`, `AtomicShardCount`,
  `TransactionId`, and `IsPrepared` are preserved on the wire end-to-end.
  The receiver consumes `TransactionId` and `IsPrepared` to drive the
  prepared / terminal staging path, and `AtomicShardCount` to drive
  the per-source-shard arrival tally on terminal records. It also reads
  `AtomicBatchSize` to recognise saga prepare-phase entries (which
  bypass the pinned-floor and causal-dependency gates) and to reject a
  range delete that carries atomic-batch metadata, and it forwards
  `AtomicBatchSize` and `AtomicBatchIndex` with every prepared write
  into the per-transaction staging hop.
- The receiver-side multi-key atomic apply seam (and its associated
  `Atomic` + `Apply` value types) was deleted by the universal-
  visibility ship. Cross-cluster atomic visibility is provided
  exclusively by the per-key prepared / per-shard terminal-mark apply
  hops described above; the local `SetManyAtomicAsync` saga inside
  `Orleans.Lattice` uses the same point-apply seam as a non-saga
  write, with the `IsPrepared` flag selecting the staging behaviour.
- Local single-tree atomic visibility (within one cluster) is
  shipped end-to-end via the per-tree transaction registry
  linearization point; see
  [Atomic Writes](../lattice/atomic-writes.md) for the protocol
  and [Consistency](../lattice/consistency.md#atomic-visibility)
  for the read-path dial-back. The cross-cluster receiver seam
  reuses the same registry grain.
- The producer-side per-key WAL filter shipped earlier. Hosts that
  need to bound the change feed at commit time configure
  `ReplicatedTrees`, `KeyFilter`, or `KeyPrefixes` on
  `LatticeReplicationOptions` - see [`wal.md`](wal.md). `TxCommit`
  and `TxAbort` records are exempt from the per-key filter as
  described above.