--- title: "Release 2026-09-05: Fixed - Changelog" url: "https://nsta1.github.io/Orleans.Lattice/changelog/2026-09-05-2.html" source: "https://github.com/NSTA1/Orleans.Lattice/blob/release/9.9/CHANGELOG.md?plain=1#L1217-L1288" documents: "Orleans.Lattice 9.9.0 (release line 9.9)" built: "2026-10-04" all-pages: "https://nsta1.github.io/Orleans.Lattice/llms.txt" --- # Release 2026-09-05: Fixed Part of [Release 2026-09-05](../changelog/2026-09-05.md), in [Changelog](../CHANGELOG.md). ## Fixed - **Three public read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal.** `LeafProjectionDigest` (returned from `ILattice.GetLeafProjectionDigestAsync` / `...ForRangeAsync`), `ViewDigest` (returned from `ILatticeView.ComputeDigestAsync`) and `EntryRevision` (returned from `ILattice.ScanEntryHistoryAsync`) are serializable value records whose `byte[]` members (`Hash`, and `EntryRevision`'s `ValuePreview` / `Delta`) were left to the compiler-generated record-struct equality, which compares an array with `EqualityComparer.Default` - reference equality. Two digests or revisions carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self, defeating the cross-silo divergence and view-drift checks these types exist for - the same class of defect already fixed for `LatticePredicateNode`, `LatticeValueTransform` and `RuntimeViewRegistration`. All three now hand-write `Equals` / `GetHashCode` comparing every scalar field ordinally and their array bytes by content (`ViewDigest`'s existing `ContentEquals` becomes the named alias the new value equality delegates to). No public member is removed or changed and no wire alias moves; `EntryRevision.VectorClock` keeps its own equality. (`Orleans.Lattice` 9.6.0) - **Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal.** `CrdtMemberChange` and `CrdtMemberValue` (the element-level provenance event and the live-member projection the State API surfaces from a CRDT's decoded history and folded state) and `EntryRevisionRecord` (returned from `ILatticeStateQuery.GetEntryHistoryAsync`) are serializable records whose `byte[]` members (`Element`, and `EntryRevisionRecord`'s `ValuePreview` / `Delta`) were left to the compiler-generated record equality, which compares an array with `EqualityComparer.Default` - reference equality. Two members or revisions carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self - directly contradicting `CrdtMemberChange`'s documented "identity is by content (byte equality)" contract. This is the same class of defect already fixed for `LeafProjectionDigest`, `ViewDigest` and `EntryRevision`; `EntryRevisionRecord` is the abstraction-layer twin of `EntryRevision` that the earlier sweep missed. All three now hand-write `Equals` / `GetHashCode` comparing every scalar field ordinally and their array bytes by content, and `EntryRevisionRecord` additionally compares its `MemberChanges` list element by element. No public member is removed or changed and no wire alias moves. (`Orleans.Lattice` 9.6.0, `Orleans.Lattice.Api.Abstractions` 9.6.0) - **Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal.** `EntryRecord` (the read-only entry projection returned by `ILatticeStateQuery`'s entry scan and detail reads) and the two MCP data-tool DTOs `DataGetToolResult` (the `data_get` point-read result) and `DataEntryDto` (the key/value entry the `data_read_range` read returns and the write batch accepts) are records whose `byte[]` members (`EntryRecord.ValuePreview`, `DataGetToolResult.Value`, and `DataEntryDto.Value`) were left to the compiler-generated record equality, which compares an array with `EqualityComparer.Default` - reference equality. Two entries, results, or records carrying identical content therefore never compared equal, and a value that round-tripped through serialization never equalled its pre-serialization self. This continues the same sweep already applied to `DataReadResult`, `DataEntry`, `DeadLetterEntryRecord`, `CrdtMemberChange` and `CrdtMemberValue`, closing the read-surface siblings the earlier passes missed: `DataGetToolResult` and `DataEntryDto` are the MCP tool twins of the already-fixed facade records `DataReadResult` and `DataEntry`, and `EntryRecord` is the state-inspection projection that carries the same `CrdtMemberValue` members those passes hardened. All three now hand-write `Equals` / `GetHashCode` comparing every scalar field ordinally and their array bytes by content, and `EntryRecord` additionally compares its `CurrentMembers` list element by element. No public member is removed or changed and no wire alias moves. (`Orleans.Lattice.Api.Abstractions` 9.6.0, `Orleans.Lattice.Api.Mcp` 9.6.0) - **Three more serializable value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal.** `HistoryRow` (the durable per-key history-view row - the serialized value stored at each history-view key), `ViewWrite` (the effect an `ILatticeViewProjection` emits for the view maintainer to apply to a view tree) and `VersionedValue` (returned from `ILattice.GetWithVersionAsync`) are serializable value records whose `byte[]` members (`HistoryRow.Value` and `HistoryRow.Delta`, `ViewWrite.Value`, and `VersionedValue.Value`) were left to the compiler-generated record equality, which compares an array with `EqualityComparer.Default` - reference equality. Two rows, writes, or results carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self. This continues the same sweep already applied to `LeafProjectionDigest`, `ViewDigest`, `EntryRevision`, `CrdtMemberChange`, `CrdtMemberValue`, `EntryRevisionRecord`, `DataReadResult`, `DataEntry` and `DeadLetterEntryRecord`; `HistoryRow` is the durable history-view twin of the already-fixed `EntryRevision`. All three now hand-write `Equals` / `GetHashCode` comparing every scalar field ordinally and their array bytes by content. No public member is removed or changed and no wire alias moves. (`Orleans.Lattice` 9.6.0) - **Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal.** `DataReadResult` and `DataEntry` (the point-read result and the key/value entry the data plane returns and accepts, from `ILatticeDataApi.GetAsync` and the bounded range read) and `DeadLetterEntryRecord` (returned from `ILatticeStateQuery.ListDeadLettersAsync`) are serializable records whose `byte[]` members (`DataReadResult.Value`, `DataEntry.Value`, and `DeadLetterEntryRecord.ValuePreview`) were left to the compiler-generated record equality, which compares an array with `EqualityComparer.Default` - reference equality. Two entries or records carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self. This continues the same sweep already applied to `LeafProjectionDigest`, `ViewDigest`, `EntryRevision`, `CrdtMemberChange`, `CrdtMemberValue` and `EntryRevisionRecord`, closing the public read-surface siblings on the data and dead-letter surfaces that the earlier passes missed. All three now hand-write `Equals` / `GetHashCode` comparing every scalar field ordinally and their array bytes by content. No public member is removed or changed and no wire alias moves. (`Orleans.Lattice.Api.Abstractions` 9.6.0) - **A routine WAL materialiser pin flush no longer rewrites every co-routed leaf on its shard.** The durable pin store keeps one grain-state blob per shard with an entry per routed leaf consumer, and Orleans writes grain state whole, so persisting one leaf's frontier rewrote every neighbour on that shard - O(consumers per shard), 1.33 MB on an observed 7,841-file tree. `LeafCursorReporter.FlushDurableMaterialiserFrontierAsync` routed that routine flush through the write-through birth path, so every leaf activation and graceful deactivation forced a full awaited blob rewrite serialized through one non-reentrant grain. Under activation collection with roughly 9,500 leaves this backed the pin grain's non-reentrancy queue up to 165 deep with a single write taking 42.9 s, past the 30 s response timeout; the durable pin floor then never advanced, WAL GC could not trim, cold leaves replayed past `MaxLeafReplayEntries`, and callers timed out. The retention flush now goes through the coalesced report path, so the merge stays immediate and only the durable write is debounced - safe because the pin is a monotonic-max retention floor, and a durable pin that lags the in-memory frontier only ever retains more WAL. The pin grain additionally amortises its coalesced flush against its own measured cost, deferring a timer tick unless nine times the last write's duration has elapsed since it completed, which bounds the grain's write duty cycle to one tenth and leaves the rest for draining its queue; it is self-tuning, needs no configuration, and never defers an explicit durability point. Only the birth block-pin seed keeps write-through, where the barrier is genuinely load-bearing. (`Orleans.Lattice` 9.6.0) - **The replication health check's sustained-degraded window no longer documents the opposite of what it enforces.** `LatticeReplicationHealthCheckOptions.UnhealthyAfter` claimed that `TimeSpan.Zero` would treat every degraded sample as unhealthy on the next probe. It does the reverse: any non-positive value disables sustained-degraded escalation entirely, so an operator reaching for the strictest possible gating silently got the loosest. The XML doc, the prose page, and a test whose name asserted the opposite of its own assertion now all agree with the code, and the negative, infinite and small-positive cases are pinned by tests. No behaviour or public member changes. (`Orleans.Lattice.Replication` 9.6.0) - **A page fill that finds nothing live no longer walks a shard's entire leaf chain in one call, and `repocontext_list_repos` no longer re-counts every embedding on every call.** Two independent defects that compounded into a 605-second `GetSortedEntriesBatchAsync` timeout, and a partition head-of-line blocked behind it, on a repository mid-back-fill. (1) The shard root's leaf-walk budget - the leaf cap (`MaxLeavesPerScanPage`, default 64) and the wall-clock deadline (`MaxScanPageDuration`, default 5 seconds) that bound every read page fill - was consulted as `ShouldYield(resultsCollected)` and returned `false` outright whenever nothing had been collected yet. That was a guard against emitting an empty page the caller could not advance past, but it disarmed **both** bounds for the case that needs them most: a *sterile* run of leaves, where every row is moved-away, tombstoned, TTL-expired, or rejected by the predicate or the slot filter. A scan crossing thousands of such leaves - routine during a back-fill, a bulk delete, or a slot-filtered read - held the non-reentrant shard root for the whole chain and queued every other request to that shard behind it. The budget is now a pure work bound, and the ability to *act* on it is what each call site owns: the six page-fill methods return a leaf-boundary **resume key** on the page (`EntriesPage.ResumeFromKey` / `KeysPage.ResumeFromKey`, both additive) and accept it back, so a sterile page yields with a position the caller can genuinely continue from. Where no such position exists the walk still refuses to yield, so a bounded page is never one the caller cannot advance past. The wall-clock half of the budget now starts when the grain call begins rather than when it reaches its leaf loop, so the time already spent preparing the grain and traversing down to the start leaf counts against the page - a request that had already held the shard for minutes on a cold activation is no longer granted a further full `MaxScanPageDuration` of leaf walking on top of it. The deadline is sampled between leaf reads, so it bounds how many individually slow leaves one page fill chains together rather than preempting a leaf read already in flight. The boundary is direction-aware because the semantics differ: forward it is the visited leaf's exclusive high key, which is an **inclusive** lower bound for the resumed walk and therefore cannot be returned as an exclusive continuation token, while in reverse it is the leaf's inclusive low key, which is already exclusive as an upper bound. Resuming is strictly monotone in both directions, so a resumed page cannot revisit the leaf it resumed from. The same fix closes a latent hole on the caller side: the entry and key cursors treated an empty page as the end of the walk even when it claimed more, silently truncating any result set whose page happened to come back empty. (2) `repocontext_list_repos` reports `embeddedVectorCount` per repository, and computing it exactly means walking the whole vector-membership tree - the largest in the store. It was memoized against a cache generation that **any** membership write advances, so during an active ingest, which is exactly when an operator lists repositories to watch progress, the memo never hit and every call re-scanned tens of thousands of entries synchronously on the request thread. The read no longer performs that walk: it serves the last completed figure, states its currency through a new `embeddedVectorCountPending` flag, and schedules at most one out-of-band refresh per repository. A repository nothing has counted yet omits the count entirely rather than reporting `0`, because "not measured" and "no vectors landed" are different answers and an operator acts differently on each. An exact count remains available, and priced, through the writer's explicit scan. The approximate-index sweep, which reads only repository ids, no longer builds a full summary per repository to discard everything but the id. ([#1992](https://github.com/NSTA1/Orleans.Lattice/issues/1992)) (`Orleans.Lattice` 9.6.0, `Orleans.Lattice.Api.Mcp.RepoContext`) - **A read page fill now has a hard end-to-end ceiling, so a stalled prologue or a single leaf read that never returns can no longer hold a shard indefinitely.** [#1992](https://github.com/NSTA1/Orleans.Lattice/issues/1992) moved the page budget's clock to the start of the grain call, but the budget is *cooperative*: `MaxScanPageDuration` is sampled between leaf reads, so it can only stop a walk where the walk can name a resume position. Two shapes never reach such a point, and that residual limit was recorded at the time as deliberately out of scope. A prologue or descent that parks - preparing the activation, resolving options, or traversing down to the start leaf - never enters the leaf loop, so nothing samples the budget at all; and a leaf read already in flight cannot be preempted, so one read that never returns is unbounded no matter what the budget says. Either shape holds the deliberately non-reentrant shard root for as long as the underlying call takes and head-of-line blocks every other request to that shard: the reported incident was a single `GetSortedEntriesBatchAsync` holding a shard for 576.8 seconds against a 5 second budget, 115 times over. A new option, `MaxScanPageStallDuration`, is the hard ceiling that covers both. It defaults to `null`, meaning *derive it*: the effective ceiling is the silo's configured Orleans `ResponseTimeout` minus a five second headroom, floored at `MaxScanPageDuration`. That relationship is the load-bearing one - a ceiling at or above the response timeout is dead configuration, because the caller's own RPC deadline expires first, so the caller sees an anonymous Orleans timeout rather than the typed fault and, critically, nothing releases the shard. Deriving keeps that ordering true for any configured timeout instead of only for the Orleans default, so a deployment that tightens `ResponseTimeout` does not silently lose the guard; an explicit value overrides it, and `Timeout.InfiniteTimeSpan` disables it. When it fires the call is abandoned and faulted with a typed, retriable `ScanPageStalledException` (a `TimeoutException`), which releases the shard so its queue drains and lets the caller retry from its last continuation token; abandoning a parked await is the same shape the activation-readiness and shadow-forward deadlines already use, and the deadline is plumbed with a `CancellationTokenSource` rather than `Task.WaitAsync(TimeSpan)`, which this codebase has observed failing to fire. `MaxScanPageDuration` is unchanged and remains the graceful bound that yields a resumable page - the new ceiling only fires where the graceful one structurally cannot, and validation requires it to sit strictly above it. Every fault names the phase it stalled in (prologue, descent or leaf-walk), so the next occurrence is self-diagnosing rather than a bare timeout, and a new counter, `orleans.lattice.shard_root.scan_page.stalls`, is dimensioned by that phase and charted on the existing shard-root wedge-guard panel. Closing the prologue gap also **removes a grain call from each of the fifteen page-fill entry points**: all three bounds are plain per-tree options rather than structural pins, so they resolve synchronously from the options monitor instead of through the registry round trip every site previously awaited - which is precisely what lets the budget be armed as the literal first statement of the call. Each entry point is therefore now a synchronous wrapper over an `async` core, and a new structural test asserts that of all fifteen, so a future edit cannot silently reintroduce an `await` in front of the clock. That test also found four further defects, all of them the same [#1992](https://github.com/NSTA1/Orleans.Lattice/issues/1992) defect surviving in the shard root partials that sweep missed: the operator projection rebuild and the materialiser-lag walk in `ProjectionAdmin`, and the shard diagnostics page and the leaf byte-footprint refresh in `Diagnostics`, each built its budget after an `await` and so had **no start clock at all**, leaving its entire prologue unbounded - the lag walk's worst of all, since that prologue awaits a WAL head offset per partition before it ever reaches a leaf. All four are fixed here with the rest, and all fifteen entry points are now covered by the structural test. The per-call phase probe and its cancellation source are pooled and the guard adds no state machine when it is disabled or the page completes synchronously, so the ceiling costs nothing on a healthy read. ([#2002](https://github.com/NSTA1/Orleans.Lattice/issues/2002)) (`Orleans.Lattice` 9.6.0, `Orleans.Lattice.Dashboards` 9.6.0) - **Four read-only leaf-chain walks in the shard root no longer hold the shard for the whole chain, and two uncapped tree descents can no longer spin forever.** The sweep that bounded the shard's traversal walks ([#1955](https://github.com/NSTA1/Orleans.Lattice/issues/1955), [#1957](https://github.com/NSTA1/Orleans.Lattice/issues/1957), [#1971](https://github.com/NSTA1/Orleans.Lattice/issues/1971)) audited `ShardRootGrain.Traversal.cs`, so four walks living in the `Diagnostics` and `ProjectionAdmin` partials were never reached: a deep diagnostics scrape, a materialiser-lag query, an operator-forced storage-usage re-anchor, and an operator projection rebuild each walked every leaf inside one call on a non-reentrant grain, so on a wide shard a single dashboard scrape or admin verb queued every other request to that shard behind it for the length of the chain. All four are now work-bounded on the same `MaxLeavesPerScanPage` / `MaxScanPageDuration` budget as the rest of the shard, resuming by **key** rather than by leaf grain id so a leaf split or consolidated away between batches is re-descended rather than read as a clean end of chain. Three of the four route through the same `BoundedLeafWalk` helper the background coordinators adopted in [#1973](https://github.com/NSTA1/Orleans.Lattice/issues/1973), reached through a new in-shard entry point that takes an already-resolved start leaf, because a shard resolving its own start leaf through `IShardRootGrain` would self-call and deadlock; sharing that helper also makes these walks immune by construction to the disarmed-budget hole a sterile run can otherwise hit ([#1992](https://github.com/NSTA1/Orleans.Lattice/issues/1992)). The projection rebuild is the one deliberate exception and says so at the site: it deactivates each leaf as it goes, so it must read the sibling pointer and the resume key **before** the rebuild rather than after, which is the one ordering the shared helper cannot express - reading them afterwards would force an immediate WAL replay purely to obtain a cursor and turn the deliberately lazy rebuild into an inline one. **Every answer is bit-for-bit what the single-call walk produced.** The WAL heads a lag query measures against are captured on the first batch only, so a tree committing mid-walk cannot inflate the reported lag; the storage-usage totals are re-anchored only on the batch that completes the walk, and only from the whole-chain figure, so the shard never advertises a fraction of its own footprint to a concurrent reader; and the diagnostics report's O(1) shard-wide fields (depth, hotness, lifecycle flags) are computed on the first batch only rather than re-derived per batch for an identical answer. Each legacy verb is retained as a wire-compatible shim that drives the bounded protocol to completion in one call, so a caller from an older build is unaffected. Separately, the two leftmost-path descents behind the diagnostics and footprint walks were written `while (!childrenAreLeaves)` rather than the bare `while (true)` that audit swept for, so both were still uncapped: a corrupt or cyclic leftmost-child pointer spun the non-reentrant shard root forever, and both now fail with a typed exception at the same `MaxTreeDescentLevels` cap every other descent uses. One walk is deliberately left unbounded and is instead instrumented through the existing slow-atomic-walk warning: the transaction-terminal chain collection must observe one terminal HLC across the whole chain, and yielding mid-walk would tear the saga it commits. ([#1972](https://github.com/NSTA1/Orleans.Lattice/issues/1972)) (`Orleans.Lattice` 9.6.0) - **Two distinct caller identities that alias under a bare-delimiter join no longer coalesce onto one shared metrics sampling loop.** `SharedMetricsSampler.BuildIdentityComponent` renders the visibility-determining identity of a caller - its subject id, transitively-expanded group closure, and claim bag - into the signature that decides when two dashboard subscribers may share a single identity-filtered sampling loop. It joined the group ids with a bare `,` and each claim as `key=value` without length-prefixing any field, so two distinct identities could collapse to the same string: groups `["a,b"]` and `["a","b"]` both rendered `a,b`, and claims `{"a=b":"c"}` and `{"a":"b=c"}` both rendered `a=b=c`. A wrongly shared loop serves one subject the first subscriber's identity-filtered view, so an aliased pair breaks the "identical signature implies identical read access" guarantee (#971) - the same delimiter-aliasing class the [#1952](https://github.com/NSTA1/Orleans.Lattice/issues/1952) projection-version fix closed. Every subject id, group id, and claim key and value is now length-prefixed, making the signature injective so no delimiter a field happens to contain can shift a field boundary. The signature is ephemeral in-memory coalescing state, so no persisted data or rebuild is affected. (`Orleans.Lattice.Api.State` 9.6.0) - **A sliding-expiration blob cache entry that degrades to "never expires" no longer reports a spurious slid deadline.** `BlobCacheEntryExpiration.Slide` computes the next sliding deadline for an entry, but inverted its guard for the no-effective-expiry case: a null effective expiry is the documented never-expires state that the sibling `IsExpired` and the `FromMetadata` reader already honour (a partially written or corrupt blob whose sliding window parsed but whose absolute expiry did not degrades to never-expires), yet `Slide` returned the freshly computed candidate deadline instead of null there, turning a never-expiring entry into one that expires at the slid time. The guard now returns the candidate only when it strictly extends a set effective expiry, and null whenever the entry never expires, matching the never-expires semantics its own siblings enforce. (`Orleans.Lattice.Caching.AzureBlob` 9.6.0) - **A redefined view whose filter differs only in where a colon falls can no longer reuse a stale projection version and skip its rebuild.** The view maintainer gates a rebuild on the projection version, and the three built-in projections (`PredicateLatticeViewProjection`, `LatticeFoldProjection`, `AggregationLatticeViewProjection`) derived that version by joining each predicate node's caller-controlled member path and constant with a bare `:` delimiter. A value containing `:` could shift a field boundary so that two structurally different filters serialized identically and hashed to the same version, leaving a redefined view serving results built for the old filter (the same aliasing class as the [#1952](https://github.com/NSTA1/Orleans.Lattice/issues/1952) operation-id fix). Both variable-length fields are now length-prefixed, making the node encoding injective. Because this changes the projection-version hash for every filtered view, the first drain after upgrading recomputes the version and performs a one-time rebuild - the designed response to a version change - after which the version is stable again. (`Orleans.Lattice` 9.6.0) - **A misconfigured clamp bound no longer lets a state-API request exceed its configured maximum.** The four query clamp helpers in `LatticeStateQuery` (`ClampHistoryLimit`, `ClampHistoryPreviewBudget`, `ClampPageSize`, `ClampScanPreviewBudget`) fell back to the raw configured default whenever the requested value was below one, without also capping that default at the configured maximum - and nothing validates `LatticeApiStateOptions`, exactly as the sibling `LatticeDataApi` clamp path already notes. A default configured above its own maximum therefore let a defaulted request return more history rows, a larger preview budget, or a larger page than the maximum was meant to cap, while the same helpers still capped an explicitly requested value correctly. Each below-one fallback now returns `Math.Min(default, maximum)`, so a defaulted request can never exceed the maximum; a correctly ordered configuration (default at or below maximum) is unaffected. (`Orleans.Lattice.Api.State` 9.6.0) - **An effectively-unbounded backup retention window no longer faults the whole retention pass.** `BackupSchedulerGrain.ComputeKeepSet` computed its age cutoff as `now - RetentionMaxAge`, but the schedule-options validator rejects only a non-positive window, so a very large `RetentionMaxAge` (up to `TimeSpan.MaxValue`) is permitted and underflowed `DateTimeOffset`, throwing `ArgumentOutOfRangeException` and failing the retention pass after an otherwise successful capture. A "retain for practically forever" window means keep every manifest, so the cutoff now saturates at `DateTimeOffset.MinValue` instead of faulting, retaining all manifests as intended. (`Orleans.Lattice.Backup` 9.6.0) - **A backup health-monitor shutdown race no longer faults the host on the way out.** `BackupHealthMonitorActivationService` retries a transient startup failure - the silo not yet being ready to serve its grain call - and caught that failure with `catch (Exception ex) when (!stoppingToken.IsCancellationRequested)`. An exception filter is evaluated at *throw* time, so a stop request arriving between the grain call failing and the filter running made the filter false and left the exception **uncaught**. It escaped `ExecuteAsync` and faulted the `BackgroundService` task, and because `BackgroundServiceExceptionBehavior` defaults to `StopHost`, a benign "silo not ready" raised on the way out could take the whole host down - a shutdown-time race turning a retryable startup condition into a host stop. The failure is now caught unconditionally with the cancellation check moved inside the handler: once stopping has been requested the call was going to be abandoned anyway, so whatever it threw is not a fault. A regression test drives the interleaving deterministically (cancel, then throw) and fails against the old handler. The existing cancellation test was also unsound - it signalled its completion source *before* throwing, so the case sometimes exercised the throw-racing-cancellation path instead of the delay-cancellation path it is about - and now signals in a `finally`, with the other path covered by its own test. ([#1959](https://github.com/NSTA1/Orleans.Lattice/pull/1959)) (`Orleans.Lattice.Backup` 9.6.0) - **Two more MCP tool-argument enum parsers no longer accept a numeric ordinal or a comma-combined name that folds onto a defined member.** The backup tool's `scopeKind` and restore `mode` parsers (`BackupToolMappings.ParseScopeKind` and `ToRestoreMode`) matched their value with `Enum.TryParse` plus an `Enum.IsDefined` guard - the same guard the [#1941](https://github.com/NSTA1/Orleans.Lattice/issues/1941) hardening added to three sibling parsers - but that guard rejects only an out-of-range ordinal. Both still accepted an in-range numeric string (`"1"` binds to the defined `BackupScopeKind.Prefix` and `LatticeRestoreMode.ShadowCutover`) and a comma-combined name, which `Enum.TryParse` treats as a bitwise OR: `"WholeTree,Prefix"` = 0|1 = 1 folds onto `Prefix` and `"InPlace,ShadowCutover"` = 0|1 = 1 folds onto `ShadowCutover`, so a wire caller could select a backup scope or restore mode it never named, each violating the parser's documented "reject unrecognised value" contract. These were the two members the #1941 sweep missed. Each now adds the ordinal round-trip name-equality check (`string.Equals(parsed.ToString(), value, StringComparison.OrdinalIgnoreCase)`) the sibling parsers use, which rejects both the numeric and the combined forms while still parsing a genuine name in any case. (`Orleans.Lattice.Api.Mcp` 9.6.0) - **A misconfigured range-read page-size bound no longer faults every bounded range read.** `LatticeDataApi.ReadRangeAsync` clamped its page size from above with `Math.Min(size, MaxRangePageSize)` but, unlike the sibling range-delete path that floors its step at one (`Math.Max(1, RangeDeleteStepSize)`), applied no floor of its own - and nothing validates `LatticeApiDataOptions`. A non-positive `MaxRangePageSize` (or `DefaultRangePageSize`) therefore collapsed every resolved page size to zero, which the cursor grain rejects with `ArgumentOutOfRangeException`, so a single misconfigured option turned every bounded range read into a fault instead of a bounded page. The clamp now floors at one, mirroring the range-delete step, so a non-positive bound yields a one-entry page that still paginates rather than a fault. (`Orleans.Lattice.Api.Data` 9.6.0) - **Seventeen broken in-page documentation links now land where they say they do.** Cross-references into the core API reference, the metrics and chaos-test documents, the tree-storage guide, and three replication documents pointed at anchors that do not exist, so following one arrived on the correct page at the wrong position - on github.com as much as in any rendered site. The failures were the usual heading-slug traps: an anchor written for a shortened heading (`#resize` for "Resize and reshard"), and headings whose punctuation leaves a doubled hyphen in the slug ("Saga / coordinator / lifecycle" becomes `#saga--coordinator--lifecycle`). Every replacement was taken from the anchor the renderer actually generates rather than derived by hand. The new documentation pull-request check fails closed on any broken relative link or in-page anchor, so this class of rot cannot silently return. (`Orleans.Lattice` 9.6.0, `Orleans.Lattice.Replication` 9.6.0) - **Emptiness probes and the remaining shard count paths no longer walk a whole leaf chain in one call.** `AnyAsync`, `CountWithMovedAwayAsync`, and `CountForSlotsAsync` each walked every leaf in the shard inside a single non-reentrant call, head-of-line-blocking every other request to that shard for the duration. Each now has a work-bounded form that returns a resume position, with the original retained as a wire-compatible wrapper, and the real callers - the tree emptiness probe and the two per-slot count fan-outs - drive the bounded form so the bound actually engages. `AnyAsync` short-circuits at the first non-empty leaf, so its bound only ever engages on the empty or fully-tombstoned shard that was the unbounded case; a negative answer is believed only once the shard reports no resume position, so a bounded batch can never be mistaken for a genuinely empty shard. The moved-away slot sets are unioned across batches rather than replaced, because a slot observed only in an earlier batch is as real as one observed in the last. `CountForSlotsAsync` also gains the past-range exit, paid only when a leaf yields no in-range keys. Resumption is by key throughout, never by leaf grain id, since Orleans grains are virtual and a leaf reclaimed between batches would otherwise activate empty, report no sibling, and silently truncate the walk. ([#1971](https://github.com/NSTA1/Orleans.Lattice/issues/1971)) (`Orleans.Lattice` 9.6.0) - **Counting a shard no longer walks its whole leaf chain in one call, and a narrow range no longer costs a whole-shard walk.** `CountAsync` on a shard walked every leaf inside a single non-reentrant call, so it head-of-line-blocked every other request to that shard for the duration - and, uniquely among the range-scoped reads, it had no past-range exit at all, so counting a narrow range still walked to the end of the chain. The shard now exposes a work-bounded batch that counts a bounded number of leaves and returns a resume position, and `LatticeGrain` drives those batches to completion, so `ILattice.CountAsync` remains one logical call that simply makes several short shard calls instead of one long one. Resumption is by key rather than by leaf grain id, because Orleans grains are virtual and a leaf reclaimed between batches would otherwise activate empty, report no sibling, and silently end the walk. A resume position is emitted only when a next leaf actually exists: unlike a repeated range delete, which is idempotent, re-counting a leaf would inflate the answer and never terminate. Atomic visibility is unchanged, because it comes from the saga-decision snapshot pinned for the lifetime of the enumeration rather than from the granularity of any one shard call, and the multi-batch loop stays inside a single `LatticeGrain` invocation so that pin survives it. The past-range probe is paid only when a leaf contributes nothing, so a productive leaf keeps its single-call cost. ([#1971](https://github.com/NSTA1/Orleans.Lattice/issues/1971)) (`Orleans.Lattice` 9.6.0) - **Tombstone compaction can no longer be silently truncated by a stale resume cursor.** `TombstoneCompactionGrain` walks a shard in batches and persists its resume position as a leaf grain id. Orleans grains are virtual, so a cursor naming a leaf whose state no longer exists does not fail - it activates a fresh, empty grain that reports no sibling, which the walk reads as a clean end of shard. It would then report the shard fully compacted with the remainder never visited: no exception, no metric, and no log line to distinguish it from genuine completion. This was not reachable in practice, because the leaf chain is grow-only (splits and bulk-load grafts insert, nothing unlinks) and leaf state is cleared only when a tree is purged, which by contract runs against an already-offline tree - but that safety was an emergent property of the rest of the design rather than anything the compaction code asserted, so a later change that reclaimed leaves would have reintroduced it silently. The resume path now validates the cursor and falls back to re-walking the shard from its leftmost leaf when it does not resolve, which is safe because per-leaf compaction is idempotent, and is the same trade the sibling dirty-set resume path already made. A probe that throws is treated as live rather than dead, so a transient fault is left to the existing per-leaf retry instead of restarting the whole shard. ([#1970](https://github.com/NSTA1/Orleans.Lattice/issues/1970)) (`Orleans.Lattice` 9.6.0) - **The replication config report now names every tree that actually replicates, not just the ones enabled at runtime.** A replication-enabled host decides whether to ship a tree from two sources: the runtime configuration tree an operator writes through enable and disable, and the static replicated-tree map supplied as deployment configuration, which acts as a fallback floor the runtime tree layers on top of. The report described only the first, so an estate configured entirely through deployment configuration - which is exactly what the reference architecture's own `-ReplicationTrees` flag produces - answered "no trees" in every region while demonstrably converging writes across three of them and refusing writes that disagreed with its declared merge mode. This is the one tool an operator reaches for to answer "which trees replicate here?", and on a correctly configured estate it answered "none". Both sources are now reconciled under exactly the precedence the commit path applies, so the report describes what the host does rather than what one of its two inputs says: a divergent runtime mode still fails closed and is never resolved by a static declaration, an enabled unambiguous runtime mode still wins, and otherwise the static declaration is the floor that keeps the tree shipping. Each entry carries a new `Source` (`Runtime`, `Static`, or `RuntimeAndStatic`) naming which one is in force, which also surfaces a trap that was previously invisible: because the static map is a floor, a tree reported `Static` keeps shipping after a runtime disable and is turned off by editing deployment configuration instead. Nothing on the shipping path changed, and the addition is wire-compatible - an entry from a peer that predates the field reads as `Runtime`, the only source the report used to describe. ([#1940](https://github.com/NSTA1/Orleans.Lattice/issues/1940)) (`Orleans.Lattice.Replication` 9.6.0, `Orleans.Lattice.Api.Abstractions` 9.6.0, `Orleans.Lattice.Api.Replication` 9.6.0, `Orleans.Lattice.Api.Replication.Grpc` 9.6.0, `Orleans.Lattice.Api.Mcp` 9.6.0) - **Installing the moved-away seal during a shard split no longer holds the shard for the sum of its per-leaf writes.** When an adaptive split or a consolidation migrates virtual slots, the source shard records the change on every leaf in its chain, and it did so one leaf at a time. `ShardRootGrain` is non-reentrant and every read path reaches a leaf through it, so the shard was unavailable to every other caller - point reads, writes, further splits, health probes - for the total of those writes, on a shard already hot enough to have triggered the split. The walk still happens inside a single grain turn, which is what makes the installation unobservable and is load-bearing: a half-installed seal is wrong in the dangerous direction, because an already-sealed leaf reports a key the source shard still owns as missing. What changed is that the per-leaf writes, which have no ordering relationship with one another, now issue with bounded concurrency instead of serially, while chain discovery stays sequential. Measured over a real cluster reshard at 121 leaves per shard, the walk fell from roughly 190 ms to roughly 60 ms per shard against in-memory storage, where a per-leaf write is nearly free; against a real store, where each is a network round trip, the gap widens. ([#1960](https://github.com/NSTA1/Orleans.Lattice/issues/1960)) (`Orleans.Lattice` 9.6.0) - **A leaf grain shutting down no longer overruns its deactivation deadline and floods the log.** `BPlusLeafGrain.OnDeactivateAsync` performs storage I/O, and three of its four awaits accepted no cancellation token - one passing `CancellationToken.None` to a reporter that accepts one, and the deactivation-time snapshot capture hard-coding it on the blob write, the most expensive operation on the path - so none of them could be interrupted when Orleans' deactivation deadline fired. The hook already swallowed storage failures, which made it look defended, but an uninterruptible await means the grain never *returns*: Orleans reports the overrun as a `TaskCanceledException` raised in its own frame, which the grain's own exception handling can never observe. The effect arrives as a burst rather than a trickle, because at the end of a WAL replay several thousand leaves go idle together, contend for the same storage provider, and slow each other past the deadline - observed on a single-silo deployment as roughly 1,100 error-level lines in a single minute, accounting for over 99% of all errors it logged. All three now honour the token and abandon promptly, which the existing handler absorbs, and an abandoned capture is swallowed without logging so one flood is not traded for another. None is correctness-critical on this path: the digest is staleness-tolerant and republished on the next mutation, the durable frontier is re-reported on the next activation, an abandoned snapshot simply leaves coverage unadvanced so the write-ahead log is retained rather than trimmed ahead of it, and the log remains the durability boundary throughout. ([#1965](https://github.com/NSTA1/Orleans.Lattice/issues/1965)) (`Orleans.Lattice` 9.6.0) - **Three MCP tool-argument enum parsers no longer accept a numeric ordinal or a comma-combined name that folds onto a defined member.** `ReplicationToolMappings.ToMergeMode`, the repository-context `Scan` handler's `scope` argument, and the repository-context `Remember` handler's `kind` argument each matched their value with `Enum.TryParse` plus an `Enum.IsDefined` guard - the hardening [#1941](https://github.com/NSTA1/Orleans.Lattice/issues/1941) applied - but that guard only rejects an *out-of-range* ordinal. It still accepted an *in-range* numeric string (`"11"` binds to the defined `RwSet`) and a comma-combined name, which `Enum.TryParse` treats as a bitwise OR: `"OrSet,GSet"` = 1|10 = 11 folds onto the distinct defined `LatticeMergeMode.RwSet`, `"Packages,Symbols"` folds onto `RepoContextScanScope.Memory`, and `"Decision,Note"` folds onto `MemoryKind.Memory` - so a wire caller could enable a tree under a merge mode, scan a scope, or file a durable memory entry under a kind it never named, each violating the parser's documented "reject unrecognised value" contract. Each now adds the ordinal round-trip name-equality check (`string.Equals(parsed.ToString(), value, StringComparison.OrdinalIgnoreCase)`) that `CrdtProvenanceDecoderRegistry.TryGet` already uses, which rejects both the numeric and the combined forms while still parsing a genuine name in any case. (`Orleans.Lattice.Api.Mcp` 9.6.0, `Orleans.Lattice.Api.Mcp.RepoContext`) - **A single shard range-scan call can no longer hold a shard for minutes, blocking every other request to it.** The six paged range-scan loops in `ShardRootGrain` were bounded by their *output* - the number of results collected - and walked the leaf sibling chain until the page filled. A leaf can cost a grain call, plus a snapshot rehydration or WAL replay when cold, and still contribute nothing to the page: its entries may all be filtered as moved-away by an adaptive split, tombstoned and still inside `TombstoneGracePeriod`, TTL-expired, or rejected by a pushed-down predicate. The worst case was therefore O(leaves in range) inside one call, and because shard reads are deliberately non-reentrant - the alternative reintroduced a chaos-reshard mid-saga invariant violation and was reverted - that call head-of-line-blocks every other request to the shard for its whole duration. On a live deployment one such call held a shard for roughly six minutes, queueing requests behind it for 159-360s and emitting 14,363 Orleans long-request warnings. Each page fill is now also bounded by the *work* it does, via the new [`MaxLeavesPerScanPage`](../docs/lattice/configuration/options-reference-2.md#maxleavesperscanpage) (64 leaves) and a [`MaxScanPageDuration`](../docs/lattice/configuration/options-reference-2.md#maxscanpageduration) safety net for individually slow cold leaves, returning a partial page and releasing the shard so other traffic interleaves. The bound is applied only once the page holds at least one result, so a caller can always derive its next continuation token and make forward progress - an empty page reporting more available would strand a caller that resumes from the last returned key - which keeps every existing caller correct with no wire-format change. Atomic visibility is unaffected: it comes from the saga-decision snapshot pinned for the lifetime of the enumeration, not from the granularity of an individual shard call, so an extra page boundary observes the identical view. Alongside it, five bare `while (true)` tree descents now honour the `MaxTreeDescentLevels` cap the write descent already used, so a cyclic routing pointer surfaces as a typed exception rather than an unbounded loop. ([#1955](https://github.com/NSTA1/Orleans.Lattice/issues/1955)) (`Orleans.Lattice` 9.6.0) - **A wide range delete no longer holds a shard for its whole duration.** `ShardRootGrain`'s range delete walked its entire leaf chain inside one non-reentrant call, so deleting a wide range blocked every other request to that shard - point reads, writes, splits and health probes alike - until the walk finished. Each shard's walk is now work-bounded by the same [`MaxLeavesPerScanPage`](../docs/lattice/configuration/options-reference-2.md#maxleavesperscanpage) budget the read paths use: it tombstones a bounded number of leaves, returns the key to resume from, and releases the shard so other traffic interleaves, with the tree-level call driving each shard to completion. Resumption is by key rather than by leaf grain id, because Orleans grains are virtual and a leaf reclaimed between two batches would activate empty with no siblings, silently ending the walk and leaving part of the range undeleted. This weakens no documented guarantee: `ILattice.DeleteRangeAsync` already fanned out to every physical shard in parallel with no cross-shard atomicity, and its documented visibility is per-key only - "a concurrent saga may be observed as committed for some keys and pending for others" - so the whole-shard atomicity of the old walk was an implementation artifact rather than a contract. The end-state guarantee, that every key in the range is tombstoned, is unchanged. Each batch publishes its own `DeleteRange` observer notification carrying that batch's matched-key closure; their union is the closure the single notification previously carried, so replication apply still reproduces the tombstone set without re-evaluating the predicate. ([#1956](https://github.com/NSTA1/Orleans.Lattice/issues/1956)) (`Orleans.Lattice` 9.6.0) - **A shard held by one of the remaining atomic leaf-chain walks now says which walk is holding it.** Four leaf-chain walks cannot be work-bounded, because for each the fact that no other message runs on the shard between the first and last leaf is the invariant the surrounding protocol depends on: marking moved-away slots runs immediately before a splitting shard enters Reject phase so no read crosses the Swap boundary observing an unmarked leaf (a half-installed seal is wrong in the dangerous direction - an already-sealed leaf reports a key the source still owns as missing), unmarking them during a consolidation is the mirror image against the donor freeze, a snapshot-cursor baseline reads its captured WAL head only after every leaf has frozen so the baseline describes a single instant, and shard purge runs on an already-offline tree and is destructive so cannot resume once a sibling pointer is cleared. Each now records that reasoning in place, so a later change cannot apply the work bound uniformly without first reading why it must not, and each logs a warning naming the operation and its leaf count when it holds the shard past a short threshold - Orleans' own long-request warnings name the blocked messages and say nothing about the blocker. For the two where the stall is worth removing rather than explaining, the real fix is a redesign rather than a budget and is tracked separately ([#1960](https://github.com/NSTA1/Orleans.Lattice/issues/1960), [#1961](https://github.com/NSTA1/Orleans.Lattice/issues/1961)). ([#1956](https://github.com/NSTA1/Orleans.Lattice/issues/1956)) (`Orleans.Lattice` 9.6.0) - **The CRDT provenance decoder registry no longer resolves a decoder for a numeric or combined shape string.** `CrdtProvenanceDecoderRegistry.TryGet(string?)` matched the shape tag with a bare `Enum.TryParse`, which also accepts a numeric ordinal string and a comma-separated combination, so `"3"` resolved the ordinal-3 `VersionVector` decoder and `"OrSet,GSet"` folded to the defined `RwSet` ordinal and resolved a different decoder again - both violating the documented contract that a shape tag is the mode's `Enum.ToString()` form (e.g. `"OrSet"`) and that an unrecognised shape yields `false`. The shipped decode path only ever supplies an enum name or `null` (through `ResolveCrdtShape`), so no product caller reached the defect, but the public method now guards with `Enum.IsDefined` and an ordinal round-trip name-equality check - the same `Enum.IsDefined` hardening #1941 applied to the MCP argument parsers, extended with the round-trip check that also rejects an in-range ordinal like `"3"`. (`Orleans.Lattice` 9.6.0) - **The replication accepted-secret change token no longer aliases distinct secret sets that differ only around a NUL character.** `StableSecretSetHash.Compute` delimited entries with a single `0x00` separator byte, which is indistinguishable from a `0x00` that occurs inside an entry - both an embedded `'\0'` and the high byte of every ASCII code unit hash as `0x00` - so two distinct ordered secret sets could produce the identical byte stream and the same `Version` token (for example `{"a\0\0c"}` and `{"a", "\0", "c"}`, or `{"\0"}` and three empty entries). A secret source bound from configuration can reach these shapes, and an aliased token lets a consumer skip a rebuild that a genuine secret-set change should trigger. Each entry is now length-prefixed so its extent frames its content in the digest, mirroring the sibling `ReplicationContentHash` that was corrected the same way; the token is an in-memory change token only and never travels on the wire, so the algorithm change is safe. (`Orleans.Lattice.Replication` 9.6.0) - **The tenant quota report now names the burst ceiling admission control actually enforces.** `TenantQuotaUsageMapping.BurstCeiling` computed the burst-adjusted ceiling as `ceiling / 100 * burstPercent` (divide before multiply), which floors the allowance to zero for any ceiling below 100 and understates any ceiling that is not a multiple of 100 - so a tenant with a 50% burst over a ceiling of 10 was reported as admitting 10 while the evaluator admits 15, and a 10% burst over 250 was reported as 270 while the evaluator admits 275. The tenancy engine's `TenantQuotaEvaluator` had already been corrected to multiply through a 128-bit intermediate and clamp to `long.MaxValue`; the read-only report mapping had drifted from it. The mapping now mirrors the evaluator exactly, so every surface that consumes the report (Explorer, MCP, and the gRPC binding) names the same ceiling the engine enforces. (`Orleans.Lattice.Api.TenantAdmin` 9.6.0) - **A `ulong` transform constant above `long.MaxValue` is no longer silently corrupted.** `LatticeValueTransformTranslator` captured a `ulong` literal via `Integer(unchecked((long)ul))`, wrapping any value above `long.MaxValue` to a negative integer (`ulong.MaxValue` became `-1`) and lowering to a projection constant that no longer matched the intended value. It now captures such a value as a `Double` when it exceeds `long.MaxValue` and an exact integer otherwise, matching the contract the sibling `LatticePredicateTranslator` already enforces. (`Orleans.Lattice.Schema` 9.6.0) - **The WAL rebalance recommendation now names the real hot partition on the default storage account.** `StoragePressureCollector` normalises a null partition provider key to the default catalogue key when it attributes retained bytes, so the hottest account it selects carries the normalised key; its partition search, however, compared the raw (still null) partition key against that normalised key, never matched, and collapsed the advice to the empty-tree, partition-zero fallback. The search now applies the same normalisation, so a recommendation for the default account names the actual hot tree and partition instead of `""` / `0`. (`Orleans.Lattice.Scaling` 9.6.0) - **The MCP tool surface now rejects malformed tool arguments instead of silently accepting them.** Three independent argument-validation defects shared one root cause: a value was trusted without confirming it named a real, in-range option. First, every facade tool wrapped by `CredentialStampingTool` silently ignored an unknown argument name, so a misspelled or unsupported argument (`treeid` for `treeId`, say) was dropped and the call proceeded as if it had never been supplied, masking the caller's mistake; the wrapper now computes the accepted argument names from the tool's own input schema and throws an `McpException` naming the offenders. Second, the backup tools' `ToRestoreMode` and scope parsers used `Enum.TryParse` without `Enum.IsDefined`, so a numeric enum string such as `"99"` parsed to an undefined enum value rather than raising the documented `ArgumentException`. Third, the repository-context `Scan` and `Remember` handlers had the same `Enum.TryParse`-without-`Enum.IsDefined` gap on their `scope` and `kind` arguments, so an out-of-range numeric ordinal bypassed the documented `McpException`. All three now guard with `Enum.IsDefined` (or an explicit allow-set), matching the existing `ReplicationToolMappings.ToMergeMode` pattern. ([#1941](https://github.com/NSTA1/Orleans.Lattice/issues/1941)) (`Orleans.Lattice.Api.Mcp` 9.6.0, `Orleans.Lattice.Api.Mcp.RepoContext`) Newer release: [Release 2026-09-05: Added to Changed](../changelog/2026-09-05-1.md). Older release: [Release 2026-09-03](../changelog/2026-09-03.md). Contents: [Changelog](../CHANGELOG.md).