Snapshot cursors (zero observable writes)

ILattice.OpenSnapshotKeyCursorAsync(...) and ILattice.OpenSnapshotEntryCursorAsync(...) open a strict snapshot-isolation cursor: every page returned by the cursor reflects the tree state captured at open time, and no concurrent write - foreground SetAsync / DeleteAsync, saga SetManyAtomicAsync, range delete, or replication apply - is ever visible to the cursor for the remainder of its lifetime.

Snapshot cursors compose with the live cursors documented in Durable Cursors: the cursor ID, pagination contract, and CloseCursorAsync lifecycle are identical. Only the visibility semantics differ.

When to use a snapshot cursor

Scenario	Cursor flavour
Long-running export, audit, or report that must reflect a single instant	Snapshot (`OpenSnapshot*CursorAsync`)
Pagination where the latest writes should appear on later pages	Live (`OpenKeyCursorAsync` / `OpenEntryCursorAsync`)
Stable saga-decision view across pages, but mid-page foreground writes are fine	Point-in-time live (`pointInTime: true`) - see api.md
Strict isolation against every concurrent write and every concurrent saga	Snapshot

How the snapshot is captured

OpenSnapshot*CursorAsync performs these deterministic steps before returning the cursor ID:

Routing capture. The tree's routing map is re-read from the registry (not taken from a cached copy) and its version pinned on the coordinate - with the full map as well when the tree has more than one physical shard - so all paging fan-outs target the same shard layout.
Per-shard frozen-baseline capture. Every shard root walks its leaf chain through IShardRootGrain.CaptureSnapshotBaselineAsync, freezing each BPlusLeafGrain's committed projection and folding its own (leaf_frontier, capturedHead] WAL tail exactly once (CRDT folds are not idempotent, so each record is applied to a single leaf a single time). The per-leaf results are unioned into one fully materialised per-shard projection and seeded in memory into the transient snapshot leaf for that shard, keyed by {physicalTreeId}/{shardIndex}/{baselineToken:N} - the physical tree the capture resolved, so a later alias cutover of the logical tree still reaches the seeded leaf. The seed is not written durably at capture - see Lazy baseline persistence below. The uniform capturedHead (each shard's per-partition WAL head, read after every leaf has frozen) is the bound recorded on the coordinate. To avoid blocking every shard root at once (each capture runs on the shard root's non-reentrant turn, contending with replication applies and reads), the fan-out is bounded to LatticeOptions.MaxConcurrentSnapshotCaptures shard captures at a time (default 4); the open then proceeds in waves. The captured baseline is identical under any cap - only the dispatch schedule changes. Within a single shard, the fold pass is itself fanned out over a sliding window of LatticeOptions.MaxConcurrentSnapshotBaselineFolds leaves (default 4) to cut that shard's hold; the freeze pass stays sequential because the chain is discovered one sibling hop at a time. Fold results are consumed in strict leaf-chain order, so the baseline is byte-identical to a serial fold. The whole capture is additionally covered by the hard end-to-end stall ceiling (MaxScanPageStallDuration), so a shard root can never be held indefinitely by an unresponsive leaf: the capture is abandoned instead, which is safe because it performs no observable writes until the baseline is seeded.
Saga-decision snapshot. The tree's transaction registry is read, as for a point-in-time cursor, but its decisions are not carried into the cursor: the coordinate records only a diagnostic registry clock, which is currently always zero, and a transport failure of that read does not fail the open (any other registry fault does). Saga visibility is fixed by the frozen baseline itself - a batch whose commit terminal lands after the captured head stays pending, and so invisible, on every leaf it touched.

The captured values are packaged as a LatticeSnapshotCoordinate (Orleans-serializable; alias ol.lsc) and persisted on LatticeCursorState.SnapshotCoordinate. The coordinate carries a fresh per-open SnapshotBaselineToken that identifies the durable baseline rows. The coordinate is deterministic - replaying with the same coordinate yields the same page sequence, even after silo failover.

Why a frozen baseline rather than open-time WAL replay? A snapshot scan is an ephemeral reader, not a registered WAL cursor. Earlier versions replayed each shard's WAL from offset 0 to the captured head at every page; once LatticeWalGc trimmed the committed prefix the reader depended on, that from-zero replay silently returned empty or partial results (and restarted any CRDT counter fold from zero). Freezing and materialising the projection once at open, then serving those durable rows, removes the dependency on the WAL prefix entirely: a later trim cannot perturb an already-frozen baseline, and a leaf rebuilt after eviction reloads the same rows for a stable point-in-time view across failover.

How pages are materialised

On every NextKeysAsync / NextEntriesAsync call, the cursor grain:

Resolves the per-page sub-range from the cursor's persisted bookmark.
Fans out to the per-shard transient snapshot leaves addressed by {physicalTreeId}/{shardIndex}/{baselineToken:N}. Each snapshot leaf was seeded in memory at capture from the materialised per-shard projection. Pages are served straight from that in-memory baseline - on a multi-shard tree through a donor-orphan / virtual-slot ownership filter that resolves each key's owner against the coordinate's pinned routing map (the point-in-time counterpart of the live read path's moved-away guard) - with no WAL replay. If the leaf was evicted and later reloads, it reloads from its durable baseline row (written lazily; see below). A coordinate persisted before the frozen-baseline store existed (empty SnapshotBaselineToken) falls back to the legacy from-zero WAL replay for wire compatibility.
Performs a k-way merge of the per-shard pages back into the cursor's scan order, advancing the persisted bookmark.

Lazy baseline persistence

A snapshot whose entire range drains in a single page is the common case (small range, or a generous page size). For that case the durable baseline write at capture plus the durable delete at close are pure write-amplification: the snapshot never survives past the first page, so the in-memory seed is sufficient.

The frozen baseline is therefore seeded in memory at capture and persisted lazily: the cursor flushes every shard's baseline durably (ISnapshotLeafGrain.EnsurePersistedAsync) the first time a page reports HasMore == true, before it returns that page or writes its continuation bookmark. The client only ever observes a continuation token after every shard baseline is durable, so any cursor that survives past page 1 - and can therefore fail over to another silo - is always backed by a durable baseline.

Outcome	Durable writes
Snapshot drains in one page (`HasMore == false`)	none - served from the in-memory seed, close skips the durable delete
Snapshot spans multiple pages	one baseline write per shard on the first `HasMore == true` page, deleted at close

Eviction before the first read. Because nothing is persisted until the first multi-page boundary, a seeded leaf that idle-evicts before its first page is read loses the unread snapshot. The next read fails fast with LatticeSnapshotExpiredException (an InvalidOperationException), and the caller simply reopens the snapshot. This is an availability edge only - a wrong or partial point-in-time view is never returned.

The snapshot leaves are activation-cached and idle-evict under the silo's ordinary Orleans grain-collection policy (GrainCollectionOptions.CollectionAge). LatticeOptions.SnapshotLeafIdleTtl (default 30 minutes) is described as this eviction window, but no runtime code currently reads it, so changing it has no effect. A subsequent page after eviction transparently rebuilds the leaf by reloading the same durable frozen baseline, so the view stays stable regardless of any WAL trimming that happened in the meantime.

WAL retention

A snapshot cursor still registers itself with the WAL cursor registry through IWalCursorRegistry.ReportCursorAsync(...) for the lifetime of the cursor, but it reports the coordinate's registry clock (always zero for a coordinate captured by the current build) with no blocked floor - a registration the WAL garbage collector excludes from its trim floor - so it does not hold back WAL trimming. That is safe because pages are served from the frozen baseline rather than the WAL: the registration is a diagnostic anchor (it is what orleans.lattice.snapshot.pins counts), not a correctness dependency, and trimming the WAL cannot perturb an already-captured baseline. Any durable per-shard baselines (written only once a snapshot spans multiple pages) are deleted when the cursor is closed (CloseCursorAsync) or evicted by the idle-TTL reminder; a single-page snapshot persists nothing and so deletes nothing.

Baseline TTL leak-guard

A durable baseline is normally deleted at close, but an interrupted close (silo crash, lost client) can orphan a baseline row keyed by a per-open token no other cursor reuses. To bound that leak, every persisted baseline carries a sliding time-to-live governed by LatticeOptions.SnapshotBaselineTtl (default 6 hours; set to Timeout.InfiniteTimeSpan to disable). SnapshotBaselineStorageGrain arms a self-clearing Orleans reminder when a baseline is written and slides it forward while the snapshot is actively served, so a long-running scan keeps its baseline alive while an abandoned one is reclaimed once the TTL lapses. Both the slide on write and the serving leaf's keep-alive touch are throttled (no more than once per SnapshotBaselineTtl / 2, floored at one minute) so the reminder table is not rewritten on every page.

A reclaimed baseline is gone for good. A cursor left idle past SnapshotBaselineTtl keeps serving pages only while its snapshot leaves stay activated; once a leaf has to reload its baseline (after idle eviction or a failover) and finds the row reclaimed, the page throws LatticeSnapshotExpiredException, exactly as for a baseline that was never persisted, and the caller reopens the snapshot.

Bounding the cost

Open-time cost is gated by LatticeOptions.MaxSnapshotReplayEntries (default 10 million entries per shard). With the frozen-baseline store the per-shard cost is the materialised baseline row count - what the snapshot leaf seeds into memory - rather than the captured WAL head: after a GC trim the head can be arbitrarily large while the real projection is small. The gate is checked once every shard's capture has returned: if the deepest shard's baseline exceeds this budget, OpenSnapshot*CursorAsync throws LatticeSnapshotReplayBudgetExceededException and no cursor is created. The budget does not bound the capture itself - by the time it is compared, every shard's baseline has already been built and seeded in memory into its snapshot leaf. Those seeds are never persisted, and they are released when the orphaned leaves idle-evict.

Admission control under saturation

A snapshot open is heavier than a single write: it freezes and materialises every shard's leaf chain on the non-reentrant shard roots. Admitting one into a tree that is already WAL-saturated piles that work onto roots collapsing under write back-pressure, starving replication applies and reads queued on the same roots, and a client that retries on the resulting timeout sustains a scan storm.

To prevent that, when LatticeOptions.ShedSnapshotOpensWhenSaturated is enabled (the default) and the tree's per-silo WAL saturation signal reports Saturated at the moment of the open, OpenSnapshot*CursorAsync sheds the open at admission - before any per-shard baseline capture is fanned out - by throwing a LatticeSaturatedException carrying the tree id, with SaturationSource SnapshotCursorOpen (counted on orleans.lattice.saturation.refusals under source=snapshot_cursor_open). That source is not automatically retryable: an immediate retry would re-fan the capture onto the saturated shard roots the shed protects. Only Saturated (the "pause new appends" regime) sheds; a Throttled tree is unaffected and stays browsable. The check reads the signal under the id the tree was addressed by, while the signal is sampled under the id of the write-ahead log the tree's writes land in, so on an aliased tree - after a resize, a shadow-cutover restore or a schema remediation, whose writes land in the physical copy's log - the check does not see that log's saturation and does not shed the open. The caller contract is the same as every other LatticeSaturatedException source: back off briefly and retry once the tree drains.

Over the read-only state API this refusal is mapped to gRPC ResourceExhausted; the Explorer catches that code, leaves the connection connected (other trees stay browsable) and does not auto-retry (which would amplify the storm). The Explorer browses a tree live by default and opens a snapshot cursor only when a scan is switched to its Snapshot mode; a shed open then shows as keys that did not load in time, with a Try again action. Set the option to false to restore the prior always-open behaviour.

Observability

Instrument	Kind	Tags	Description
`orleans.lattice.snapshot.replay.duration`	Histogram (ms)	`tree`, `shard`, `tenant`	Per-shard wall-clock WAL replay time when a snapshot leaf opens from a legacy coordinate (one persisted before the frozen-baseline store, with no baseline token). A frozen-baseline snapshot never replays the WAL, so it never records this.
`orleans.lattice.snapshot.replay.entries`	Counter (`{entry}`)	`tree`, `shard`, `tenant`	WAL entries consumed by that legacy snapshot-leaf replay. Not recorded for a frozen-baseline snapshot.
`orleans.lattice.snapshot.pins`	ObservableGauge (`{pin}`)	`tree`, `tenant`	Live WAL cursor-registry pins held by snapshot cursors (registrations that do not hold back trimming - see WAL retention), derived from the WAL cursor registry rather than accumulated.

orleans.lattice.snapshot.pins reports the pins held now, derived from the WAL cursor registry, rather than accumulating a +1 on open and a -1 on close. That is what makes it self-healing: an activation lost with its silo never emits a -1, but there is no compensating write to lose, so the series falls to zero the moment the registry stops holding the pin. A value that climbs and stays up is a real pin leak.

Two properties of that derivation are worth knowing before you trust a reading:

The value is as of the tree's last WAL GC pass, not as of the scrape. Opening and closing a cursor updates the series immediately, but the re-derivation that drops a pin no one could report the release of runs once per GC pass. So a pin lost with its silo clears within that tree's effective GC interval, which the scheduler varies between WalGcMinInterval and WalGcInterval (30 s to 1 h by default). A quiet tree relaxes toward the ceiling, so the worst case on a healthy tree is the ceiling; a tree blocked by an unusable pin holds at the floor, so the tree most likely to be under investigation is also the one that reconciles most often.
Setting WalGcInterval to zero or less disables the scheduler, and with it the self-healing. Neither the per-tree zero-priming nor the re-derivation runs. Opens and closes still keep the series correct, and it still cannot ratchet, so it remains more accurate than a plain accumulating counter - but a pin lost with its silo will never age out of the gauge, because nothing re-reads the registry. Re-enable WAL GC if you need this series to self-heal.

Examples

Manual lifecycle (durable cursor shape)

This shape is the one to use when the cursor must outlive the local scope - the cursorId is a string so it can be persisted to a database, sent to a queue, or resumed after a process restart.

var cursorId = await lattice.OpenSnapshotEntryCursorAsync();
try
{
    while (true)
    {
        var page = await lattice.NextEntriesAsync(cursorId, pageSize: 500);
        foreach (var kv in page.Entries)
        {
            // Process kv.Key / kv.Value. Values reflect the tree state at
            // the moment OpenSnapshotEntryCursorAsync returned, regardless
            // of concurrent writes elsewhere.
        }
        if (!page.HasMore) break;
    }
}
finally
{
    await lattice.CloseCursorAsync(cursorId);
}

Scoped lifecycle (`await using`)

For the common case where the cursor's lifetime is bounded by a single stack frame, the LatticeExtensions.Open*CursorScopeAsync family returns a LatticeScopedCursor that implements IAsyncDisposable. Disposing the scope calls CloseCursorAsync exactly once, even if the body throws. The scope is implicitly convertible to its underlying string cursor ID, so it can be passed directly to NextKeysAsync / NextEntriesAsync / DeleteRangeStepAsync.

await using var scope = await lattice.OpenSnapshotEntryCursorScopeAsync();
while (true)
{
    var page = await lattice.NextEntriesAsync(scope, pageSize: 500);
    foreach (var kv in page.Entries)
    {
        // Same snapshot semantics as the manual shape - the only
        // difference is who calls CloseCursorAsync.
    }
    if (!page.HasMore) break;
}
// scope.DisposeAsync() runs here and closes the cursor.

The *Scope family covers the five unfiltered cursor flavours - OpenKeyCursorScopeAsync, OpenEntryCursorScopeAsync, OpenSnapshotKeyCursorScopeAsync, OpenSnapshotEntryCursorScopeAsync, and OpenDeleteRangeCursorScopeAsync - so the choice between scoped and manual is independent of the cursor's semantics. A predicate-filtered cursor (the Open*CursorWherePredicateAsync methods) has no scope helper; wrap its id in new LatticeScopedCursor(lattice, cursorId) for the same close-on-dispose behaviour. Pick the scoped shape when the cursor lives and dies inside one method; pick the manual shape when the cursor ID must survive a serialization or process boundary.

Out of scope

Cross-cluster snapshot rendezvous. A snapshot is local to the cluster that opened it.
Writable snapshots. There is no snapshot variant of OpenDeleteRangeCursorAsync; a delete-range cursor spec is rejected under zero-observable-writes.
Snapshot reuse across cursors. Two cursors opened at logically identical coordinates do not share their materialised snapshot leaves: every open mints a fresh baseline token, and the snapshot leaves and their baseline rows are keyed by it.
Live-vs-snapshot diff. Compute the diff in the caller by running a live cursor against the same range.

Table of Contents