--- title: "Replication modes" url: "https://nsta1.github.io/Orleans.Lattice/docs/lattice.replication/replication-modes.html" source: "https://github.com/NSTA1/Orleans.Lattice/blob/release/9.9/docs/lattice.replication/replication-modes.md" package: "Orleans.Lattice.Replication" version: "9.9.0" documents: "Orleans.Lattice 9.9.0 (release line 9.9)" built: "2026-10-04" all-pages: "https://nsta1.github.io/Orleans.Lattice/llms.txt" bundle: "https://nsta1.github.io/Orleans.Lattice/docs/lattice.replication/llms-full.txt" --- # Replication modes Part of the [Replication documentation](README.md). Every tree replicated by `Orleans.Lattice.Replication` declares a **`LatticeMergeMode`** at configuration time. The mode tells receivers how to merge the captured value bytes; the producer stamps it onto every emitted `WalRecord` so the receiver never has to guess. There is no implicit fallback. A tree that is not declared in `LatticeReplicationOptions.ReplicatedTrees` (or, with runtime replication configuration enabled, enabled at runtime - see [Runtime Replication Config](runtime-config.md)) is **not replicated**. This is deliberate - the core library stores every value as opaque `byte[]`, so the producer cannot recognise CRDT primitives by inspection. Implicit opt-in would silently fall back to last-writer-wins on bytes and risk concurrent-update data loss; explicit declaration removes the footgun. Receivers enforce the declaration independently. The apply path re-resolves the tree's enrollment and merge mode on the receiving cluster: an inbound entry for a tree that is not replicated there is dropped, and one whose wire mode disagrees with the locally resolved mode is dead-lettered as `mode_mismatch` (see [Replication apply](replication-apply.md#4-receiver-side-enrollment-and-merge-mode-gate)). ## Declaring a mode ```csharp verify siloBuilder.AddLatticeReplication(opts => { opts.ClusterId = "site-a"; opts.ReplicatedTrees = new Dictionary { ["users"] = LatticeMergeMode.LwwRegister, ["orders"] = LatticeMergeMode.LwwRegister, }; }); ``` `null` and an empty dictionary both mean "no trees are replicated" - the commit-time observer short-circuits before any sink call. ## Available modes | Mode | Status | Convergence guarantee | |------|--------|-----------------------| | `LwwRegister` | **Available** | Last-writer-wins ordered by `HybridLogicalClock`; an exact HLC tie falls to the replica-invariant tombstone, expiry, and value-byte fields before `OriginClusterId`. Concurrent writes from different clusters silently drop the loser; safe under single-writer-per-key discipline. | | `OrSet` | **Available** | Observed-remove set. State-based merge - concurrent active-active adds and removes from multiple clusters survive convergence with their causal dot context preserved. | | `PnCounter` | **Available** | Positive-negative counter. Pointwise-max merge on each replica's positive and negative components - concurrent increments and decrements from multiple clusters sum correctly. | | `VersionVector` | **Available** | Version vector. Pointwise-max merge on each replica's `HybridLogicalClock` entry. Late or duplicate delivery is a no-op. | | `MvRegister` | **Available** | Multi-value register. Dot-tagged state-based merge - concurrent writes from different clusters survive convergence as a conflict set the application resolves via `MvRegisterAccessor.ValuesAsync()`. | | `OrMap` | **Available** | Observed-remove map of `(TKey, TValue)` where `TValue` is itself a CRDT. Per-key values converge recursively through `ICrdt.MergeFrom`. Requires a one-time `siloBuilder.AddOrMapShape(treeName)` registration on each receiving silo so the applier can resolve the generic shape; an unregistered shape faults the apply rather than silently dropping the entry. | | `Sequence` | **Available** | Replicated Growable Array (RGA) sequence for collaborative ordered lists / text. Each insert ships the dot-explicit triple `(dot, parentDot, value)` and each remove ships the tombstoned dot, so concurrent active-active inserts and deletes from multiple clusters converge on an **identical ordered traversal** via the descending `(Counter, ReplicaId)` sibling tie-break. Author values through `ILattice.Sequence(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `OrFlag` | **Available** | Observed-remove (enable-wins) flag. Each key carries a single presence bit that is set while at least one enable dot survives: an observed-remove (disable) dot cancels every enable dot from the same replica at or below its counter. Concurrent enable and disable from multiple clusters converge enable-wins with their causal dot context preserved. The minimal observed-remove primitive for composite-key membership rows (e.g. a tag/key secondary index). Author through `ILattice.OrFlag(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `RwFlag` | **Available** | Remove-wins (disable-wins) flag - the inverse of `OrFlag`. Each key carries a single presence bit tracked by three grow-only dot lists (enables, disables, and the disable-dots an observed enable has tombstoned); the flag is enabled only when an enable dot survives and no live disable remains. Concurrent enable and disable from multiple clusters converge disable-wins, so ties and unobserved withdrawals fail closed. Author through `ILattice.RwFlag(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `GCounter` | **Available** | Grow-only counter. Pointwise-max merge on each replica's cumulative component - the monotonic-only counter `PnCounter` is built from, and the natural primitive for monotone metrics, sequence / event counters, and quota consumption where decrement never happens. Concurrent increments from multiple clusters sum correctly, and late or duplicate delivery is an idempotent no-op. Author through `ILattice.GCounter(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `GSet` | **Available** | Grow-only (G) set of opaque element byte arrays with value-equality by content. The merge is set union - trivially commutative, associative, and idempotent - so concurrent active-active adds from multiple clusters all survive convergence. Grow-only by design: it carries no dots and no tombstones and has no remove operation, making it the minimal set primitive for append-only workloads (tag sets, seen-ids, accumulating audiences); reach for `OrSet` when removal is needed. Author through `ILattice.GSet(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `RwSet` | **Available** | Remove-wins observed-remove set - the set-granularity generalisation of `RwFlag` (an `RwFlag` is a single-element `RwSet`, exactly as `OrFlag` is to `OrSet`). Per element it keeps a set of add dots, a set of remove dots, and a set of observed-add tombstones cancelling removes; an element is a member only when it carries an add dot and no remove dot survives. Concurrent active-active add and remove of the same element from different clusters converge remove-wins with their causal dot context preserved, so a revoke is never silently resurrected by a concurrent re-add. The remove-wins counterpart of the add-wins `OrSet`, for membership revocation lists and blocklists where a removal must win the tie. Author through `ILattice.RwSet(key)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `MaxRegister` | **Available** | Monotone max register - keeps the greatest totally-ordered value ever seen, paired with an explicit total-order key carried on the wire. The fold is directional max over that total order, so it is commutative, associative, and idempotent - a backwards write or a duplicate delivery is a no-op, and concurrent active-active writes from different clusters converge on the single greatest value without needing the domain comparer on the receiver. The high-water-mark primitive (a monotone gauge, a version ceiling, a max-seen reading). Author through `ILattice.MaxRegister(key, orderKeySelector)`. The descriptor is a global closed shape, so no per-tree registration is required. | | `MinRegister` | **Available** | Monotone min register - the inverse of `MaxRegister`, keeping the smallest totally-ordered value ever seen, paired with an explicit total-order key carried on the wire. The fold is directional min over that total order, so it is commutative, associative, and idempotent - a backwards write or a duplicate delivery is a no-op, and concurrent active-active writes from different clusters converge on the single smallest value without needing the domain comparer on the receiver. The low-water-mark primitive (a min-seen latency floor, a first-seen timestamp). Author through `ILattice.MinRegister(key, orderKeySelector)`. The descriptor is a global closed shape, so no per-tree registration is required. | The validator accepts every defined `LatticeMergeMode` value; only undefined integer values fail validation (it also rejects a null, empty, or whitespace tree-id key). ## When `LwwRegister` is the right choice `LwwRegister` is the right answer for keys with overwrite-with-latest semantics, but only under **single-writer-per-key discipline**. Each key must have at most one authoritative cluster at any given time (e.g. routed by tenant, by shard, or by ownership token). Under this discipline, last-writer-wins is correct: there is never a genuinely-concurrent write to resolve, and the HLC order (with its deterministic tie-break) just orders the unambiguous successor. If your workload allows concurrent writes from multiple clusters to the same key, last-writer-wins **silently drops the loser** - both writes return success on their respective clusters, but only one survives the merge. For those workloads, declare a typed CRDT mode and author values through the matching accessor on `ILattice`. ## How typed CRDT modes apply on the receiver For every typed CRDT mode, the producer-side accessor authors the matching public typed delta DTO into the single `WalRecord.Delta` slot at commit time. The `WalRecord.Value` slot is stripped on the wire from every committed CRDT-mode `Set` that carries a typed delta (prepared saga entries keep it) - the canonical payload travels only as the typed delta, not as a serialised post-merge snapshot. The receiver-side applier forwards the typed delta from `Delta` to the tree's CRDT-delta apply path, which folds it into the stored primitive with the primitive's `MergeDelta` operation inside a single grain turn and records a CRDT-delta revision, exactly as a locally-authored CRDT write is recorded. The merge is wrapped in a `LatticeOriginContext.With(originClusterId)` scope so the entry the receiver's commit appends to its own WAL carries the foreign origin, and the receiver's ship loop - which ships only locally-authored entries - filters it out: the same cycle-break semantics as LWW. Change-feed consumers that historically read `Value` directly on CRDT-mode entries must migrate to either reading `Delta` and folding it against their own prior observed state, or reading the post-merge state through the public lattice surface (`ILattice.GetAsync` / typed accessors). Typed-delta merge is commutative, associative, and idempotent: late or duplicate delivery converges to the same set / counter / vector / map regardless of arrival order. The snapshot-pinned causal floor and the shadow-forward identity cache still short-circuit redundant re-delivery, but correctness does not depend on them for typed CRDT modes. ### OR-Map shape registration `OrMap` is generic, so the receiver cannot infer `(TKey, TValue)` from the `WalRecord` alone. Each silo that may apply an OR-Map entry must register the concrete shape once at startup: ```csharp verify using Orleans.Lattice; siloBuilder.AddOrMapShape("tags-by-user"); ``` The registration installs a deserialiser / merger descriptor into the `CrdtShapeRegistry` singleton before silo activation; an apply against a tree configured for `LatticeMergeMode.OrMap` with no matching registration faults the apply with a clear configuration-error message rather than silently dropping the entry. ## Sequence mode back-pressure hazard `Sequence` mode replicates an RGA at **operation granularity**: every `InsertAtAsync` / `InsertAfterAsync` / `RemoveAtAsync` / `RemoveAsync` call commits one CRDT-delta WAL entry, and every WAL entry is one unit of replication ship work. This is the right granularity for convergence - the dot-explicit insert/tombstone is the minimal structural intent a receiver needs - but it makes a **high-frequency editor against a single sequence key a WAL-amplification hazard**: a collaborative text buffer that commits one keystroke per `InsertAfterAsync` generates one WAL entry, one change-feed event, and one shipped delta per keystroke. A sustained fast typist (or a programmatic bulk import that inserts character-by-character) can therefore saturate the WAL partition that key hashes to and the outbound shipper for that key. Mitigations, in order of preference: - **Producer-side debounce / coalescing.** Buffer keystrokes in the editor for a short window (for example 50-150 ms) and commit the batch as a run of inserts under the last-resolved parent dot, or as a single multi-character element when the application's element granularity permits. Coalescing N keystrokes into one commit cuts the WAL and ship rate by N without changing the converged order. The debounce lives in the application's edit loop, not in the lattice - the lattice deliberately commits exactly what it is told so the convergence contract stays exact. - **Coarser-grained whole-value storage under `LwwRegister` for cold sequences.** A sequence that is read-mostly and only occasionally rewritten wholesale (a rarely-edited document, a config list rebuilt on save) does not need per-keystroke convergence. Storing it as an opaque value under `LwwRegister` (or rebuilding and writing the whole list on save) replaces the per-edit WAL storm with one entry per save, at the cost of last-writer-wins on concurrent whole-document writes. Reserve `Sequence` mode for keys that are genuinely concurrently edited at fine granularity. The WAL saturation back-pressure surface (`IWalSaturationSignal` / `IWalSaturationObserver`) still applies: a producer that ignores the debounce guidance and drives a single hot sequence key past the per-tree WAL admission budget observes the standard saturation signal and `LatticeSaturatedException`, the same as any other write-amplifying workload. ## Single shape per tree A replicated tree is **single-shape**: every value in it is authored and shipped under the one `LatticeMergeMode` the tree is declared with. The mode is a property of the *tree*, not of the individual write - the producer stamps the declared mode onto every `WalRecord`, hoists it once per batch into the encoded batch header, and the receiver re-stamps it onto every decoded entry before dispatching the typed apply, overriding any mode carried on the entry itself, so a batch cannot carry a second shape, and the receiver never re-inspects the bytes to guess one. This means a write whose shape disagrees with the declared mode cannot converge on the peer. A plain last-writer-wins write to a tree declared as a CRDT mode ships value bytes the receiver tries to decode as a typed delta; a CRDT write under the wrong mode ships a delta the receiver decodes with the wrong shape. Either way the receiver's typed apply throws while decoding the payload under the declared shape, the entry is retried, and after `MaxApplyRetries` it is parked on the dead-letter queue. The origin cluster's own copy stays correct, so the divergence is silent - the peer simply never receives that key. To turn that silent, receiver-side divergence into a loud, origin-side failure, the public `ILattice` write surface **fails fast**. When a tree is declared for replication, any write whose shape does not match the declared mode throws `LatticeReplicationModeMismatchException` before it commits - nothing is written locally and nothing is shipped: - A plain last-writer-wins write (`SetAsync`, `SetManyAsync`, `SetManyAtomicAsync`, `SetIfVersionAsync`, `GetOrSetAsync`, `DeleteAsync`, `DeleteRangeAsync`, `BulkLoadAsync`, and their predicate variants) to a tree declared as any typed CRDT mode is rejected. Author the value through the matching accessor (`OrSet(key)`, `PnCounter(key)`, and so on) instead. - A CRDT write (`ApplyCrdtDeltaAsync`, reached through any CRDT accessor) whose mode differs from the declared mode is rejected - whether the tree is declared as a different CRDT mode or as `LwwRegister`. ```csharp verify try { // 'tree' is declared for replication as a typed CRDT mode (for example // OrSet). A plain last-writer-wins write is rejected before it commits, // because the receiver could not decode the bytes under the declared shape. await tree.SetAsync("k", new byte[] { 1 }, cancellationToken); } catch (LatticeReplicationModeMismatchException ex) { Console.WriteLine($"{ex.TreeId}: declared {ex.DeclaredMode}, attempted {ex.AttemptedMode}"); } ``` The guard is a no-op for trees that are not replicated (the merge-mode resolver returns `null`, so single-cluster hosts are never affected) and for writes whose shape already matches the declared mode, including a plain write to a tree declared as `LwwRegister`. It costs a single cached resolver reference and one per-tree dictionary read per write. The rejection covers the direct `ILattice` write surface. The [cross-tree atomic write](../lattice/api.md) builder stages writes through a separate coordinator saga; when a slice of a cross-tree batch targets a replicated tree, the same single-shape rule applies - stage the slice with the matching accessor's `Stage*` method rather than a plain value write. ## How the mode is resolved at commit time The commit-time observer routes every mutation through `ILatticeMergeModeResolver.Resolve(treeId)`: - The default implementation reads `LatticeReplicationOptions.ReplicatedTrees` and caches the per-tree outcome until `IOptionsMonitor.OnChange` fires. - A host that enables runtime replication configuration (`AddLatticeReplication(..., enableRuntimeConfig: true)`) gets a snapshot-backed resolver instead: it answers from the runtime configuration first, falls back to `ReplicatedTrees`, and returns `null` for a tree whose runtime mode is ambiguous even when the tree is also declared statically (see [Fail-closed ambiguity](runtime-config.md#fail-closed-ambiguity)). - Hosts can replace the registration to source the mode map from elsewhere (a control plane, a feature flag system, or a permissive test stub that opts every tree in to `LwwRegister`). - A `null` return value means "this tree is not replicated" and the observer returns immediately, before any sink call. The same resolver is consulted again downstream. The leaf commit-log writer stamps the resolved mode (or a CRDT write's own authored mode) onto the durable `WalRecord.Mode` at WAL append, so a storage replay can recover it, and the shipper resolves the mode again when it frames each batch, hoisting it once into the batch header that the receiver re-stamps onto every decoded entry - so receivers pick the correct apply algorithm without re-inspecting the value bytes. A `null` resolution does not pause a shipper that is already running. It keeps draining the tree's WAL and ships new entries under an `LwwRegister` batch header; a receiver that still resolves a CRDT mode dead-letters them as `mode_mismatch`, one that resolves no mode drops them, and either way the sender advances past them (see [Fail-closed ambiguity](runtime-config.md#fail-closed-ambiguity)).