Orleans.Lattice.Api.Replication architecture

This facade is a thin, fail-closed control layer over the replication config authority. It owns three responsibilities and nothing else: resolve the tree name and authorize the caller, delegate to the engine, and scope discovery to the caller's grant. The package's separate, read-only peer-status facade is described in The peer-status read path.

The single narrowest seam

Every mutating operation authorizes at exactly one choke point before it touches engine state; the config read uses the same choke point as a per-tree filter (see Permission-scoped discovery). EnableReplicationAsync and DisableReplicationAsync first resolve the caller-supplied, tenant-local tree name to its effective id, then authorize that same id - the whole target tree - for the dedicated LatticeOperation.Replication capability through the shared ILatticeAccessGate; only on an allow do they call the engine authority. A deny throws LatticeAuthorizationDeniedException and the engine is never consulted, so a denied caller cannot author a single config dot.

Because a control operation acts on a whole tree, a partial (key-filtered) allow cannot narrow it and is treated as a deny. This mirrors the whole-tree enforcement the sibling backup facade uses.

Delegation to the engine authority

The facade holds no replication state. It delegates to ILatticeReplicationConfigAuthority in Orleans.Lattice.Replication, which authors the sys-replication-config tree:

Enable fixes the merge mode at enable time by writing an add to the enablement RwFlag and setting the MvRegister mode. Enabling an already-enabled tree under a different mode - or one whose mode is currently ambiguous - is rejected with LatticeReplicationModeChangeRejectedException; enabling under the same mode is idempotent, including when two regions enable the tree concurrently under the same mode - their enablements converge on that mode, not on an ambiguous one. A runtime precondition failure (for example a flag-based mode without a configured local replica) surfaces as LatticeReplicationPreconditionFailedException.
Enable on a non-empty tree composes the existing snapshot bootstrap: when bootstrapSourceClusterId is supplied and the local tree already holds rows, the authority requests a receiver-driven snapshot (through ILatticeReplicationAdmin.RequestSnapshotAsync) that pulls the named source cluster's pre-existing rows - which the change feed will not carry - into the local tree, then reports BootstrapRequested = true. The bootstrap cannot push the local rows to remote peers, and an enable that names a source over an empty local tree, or finds the tree already enabled under the same mode, requests no bootstrap. The request takes the rate-limited routine re-seed path, so a repeat within LatticeReplicationOptions.OperatorReseedMinInterval (default 1 minute) of the last re-seed that silo honoured for the same tree and source cluster is not dispatched, and the result still reports BootstrapRequested = true.
Disable writes a disable-wins dot to the RwFlag and keeps the entry, with its last mode, in the config tree; it never purges peer data. Unless the static deployment map also declares the tree, its merge-mode resolution then returns no mode, but shipping does not pause: an already-active shipper keeps shipping the tree's new local writes, which a peer that has converged on the disable drops at its receiver-side enrollment gate. Disabling an absent or already-disabled tree is an idempotent no-op that authors nothing; disabling an enabled tree needs a configured local replica id to stamp the dot and otherwise fails with LatticeReplicationPreconditionFailedException. A later enable re-fixes the mode to the value it requests and, when bootstrapSourceClusterId is supplied for a tree that holds data, requests a fresh snapshot bootstrap; without it, no bootstrap runs.

Permission-scoped discovery

GetReplicationConfigAsync reads the authority's per-tree status set, then filters it: a tree is included only if the caller passes the same fail-closed authorization the mutating operations use. A per-tree denial is swallowed - the tree is silently omitted - so the report never reveals a tree the caller may not manage, and never throws on a partial grant.

Fail-closed ambiguity

The merge mode is stored in an MvRegister, so two clusters that concurrently enable the same tree under different modes both survive convergence. When the compiled snapshot sees more than one distinct live mode for a tree it marks the tree ambiguous and the resolver returns no mode rather than picking one. Shipping does not pause, though: an already-active shipper keeps shipping the tree's new local writes, which a converged peer - resolving no mode for the tree either - drops at its receiver-side enrollment gate. The facade surfaces this as ReplicationTreeConfigEntry.Ambiguous = true with a null Mode. The engine-side detail is documented in runtime replication configuration.

Ordering guard

AddLatticeReplicationApi() checks at registration that ILatticeReplicationConfigAuthority has been registered and throws with an actionable message if it has not, so a host that forgot enableRuntimeConfig: true on AddLatticeReplication(...) fails fast rather than at first call.

The peer-status read path

ILatticeReplicationStatus is a separate, read-only facade with its own registration, AddLatticeReplicationStatusApi(). It reads the replication engine's per-link telemetry rather than the config authority, so it needs AddLatticeReplication(...) registered first - it checks this at registration and throws InvalidOperationException otherwise - but neither enableRuntimeConfig: true nor AddLatticeReplicationApi().

Where the numbers come from. Each silo keeps the per-link counters of the shippers and appliers it hosts in memory. Each read fans out to every active silo through a per-silo grain service, each silo answers with a bounded number of rows after the cursor, and the answers are merged. When more than one silo reports the same link - a shipper rebalanced to another silo leaves a stale copy behind, and an inbound link exists on every silo that applied a batch from that peer - the copy with the most recent successful contact is kept whole, so its backlog, error streak, and in-flight count are never mixed across silos; when no copy has made contact, the one with the longest error streak, then the larger backlog, is kept. A silo that fails to answer fails the whole read, because a partial answer would present that silo's links as absent rather than unknown. The read path holds no state, schedules nothing, and never touches the ship or apply path.
Tenant and authorization. Every call confirms that the caller's tenant resolves and fails closed with LatticeTenantAccessDeniedException otherwise. A tree filter is resolved to its effective, tenant-scoped id and authorized for LatticeOperation.Replication over the whole tree before any telemetry is read; a denied filter returns an empty page. Without a filter, each tree's verdict is computed once per call and the rows of a tree the caller may not manage are skipped, so the report never reveals that tree.
Paging. Rows are ordered by effective tree id, then peer region id, then direction. Each read asks for up to one row more than the page size, so a full page learns whether anything follows without another round trip, and rows the caller may not see are skipped by reading further rather than by ending the page early. The continuation token is opaque and versioned and encodes only the key of the last row the caller was shown. It is treated as a position to validate, never as authority: every row after it is authorized again. A malformed token, a token longer than 4,096 characters, or a token of an earlier format version is rejected with ArgumentException.
Health. Each row's ReplicationLinkHealth is derived from that row alone against the LatticeReplicationStatusOptions thresholds, without reading a clock: the time since last contact is measured when the row is read. See Configuration.

Table of Contents