Table of Contents

Release 2026-09-05: Fixed

Part of Release 2026-09-05, in Changelog.

Fixed

  • Three public read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal. LeafProjectionDigest (returned from ILattice.GetLeafProjectionDigestAsync / ...ForRangeAsync), ViewDigest (returned from ILatticeView.ComputeDigestAsync) and EntryRevision (returned from ILattice.ScanEntryHistoryAsync) are serializable value records whose byte[] members (Hash, and EntryRevision's ValuePreview / Delta) were left to the compiler-generated record-struct equality, which compares an array with EqualityComparer<byte[]>.Default - reference equality. Two digests or revisions carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self, defeating the cross-silo divergence and view-drift checks these types exist for - the same class of defect already fixed for LatticePredicateNode, LatticeValueTransform and RuntimeViewRegistration. All three now hand-write Equals / GetHashCode comparing every scalar field ordinally and their array bytes by content (ViewDigest's existing ContentEquals becomes the named alias the new value equality delegates to). No public member is removed or changed and no wire alias moves; EntryRevision.VectorClock keeps its own equality. (Orleans.Lattice 9.6.0)

  • Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal. CrdtMemberChange and CrdtMemberValue (the element-level provenance event and the live-member projection the State API surfaces from a CRDT's decoded history and folded state) and EntryRevisionRecord (returned from ILatticeStateQuery.GetEntryHistoryAsync) are serializable records whose byte[] members (Element, and EntryRevisionRecord's ValuePreview / Delta) were left to the compiler-generated record equality, which compares an array with EqualityComparer<byte[]>.Default - reference equality. Two members or revisions carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self - directly contradicting CrdtMemberChange's documented "identity is by content (byte equality)" contract. This is the same class of defect already fixed for LeafProjectionDigest, ViewDigest and EntryRevision; EntryRevisionRecord is the abstraction-layer twin of EntryRevision that the earlier sweep missed. All three now hand-write Equals / GetHashCode comparing every scalar field ordinally and their array bytes by content, and EntryRevisionRecord additionally compares its MemberChanges list element by element. No public member is removed or changed and no wire alias moves. (Orleans.Lattice 9.6.0, Orleans.Lattice.Api.Abstractions 9.6.0)

  • Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal. EntryRecord (the read-only entry projection returned by ILatticeStateQuery's entry scan and detail reads) and the two MCP data-tool DTOs DataGetToolResult (the data_get point-read result) and DataEntryDto (the key/value entry the data_read_range read returns and the write batch accepts) are records whose byte[] members (EntryRecord.ValuePreview, DataGetToolResult.Value, and DataEntryDto.Value) were left to the compiler-generated record equality, which compares an array with EqualityComparer<byte[]>.Default - reference equality. Two entries, results, or records carrying identical content therefore never compared equal, and a value that round-tripped through serialization never equalled its pre-serialization self. This continues the same sweep already applied to DataReadResult, DataEntry, DeadLetterEntryRecord, CrdtMemberChange and CrdtMemberValue, closing the read-surface siblings the earlier passes missed: DataGetToolResult and DataEntryDto are the MCP tool twins of the already-fixed facade records DataReadResult and DataEntry, and EntryRecord is the state-inspection projection that carries the same CrdtMemberValue members those passes hardened. All three now hand-write Equals / GetHashCode comparing every scalar field ordinally and their array bytes by content, and EntryRecord additionally compares its CurrentMembers list element by element. No public member is removed or changed and no wire alias moves. (Orleans.Lattice.Api.Abstractions 9.6.0, Orleans.Lattice.Api.Mcp 9.6.0)

  • Three more serializable value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal. HistoryRow (the durable per-key history-view row - the serialized value stored at each history-view key), ViewWrite (the effect an ILatticeViewProjection emits for the view maintainer to apply to a view tree) and VersionedValue (returned from ILattice.GetWithVersionAsync) are serializable value records whose byte[] members (HistoryRow.Value and HistoryRow.Delta, ViewWrite.Value, and VersionedValue.Value) were left to the compiler-generated record equality, which compares an array with EqualityComparer<byte[]>.Default - reference equality. Two rows, writes, or results carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self. This continues the same sweep already applied to LeafProjectionDigest, ViewDigest, EntryRevision, CrdtMemberChange, CrdtMemberValue, EntryRevisionRecord, DataReadResult, DataEntry and DeadLetterEntryRecord; HistoryRow is the durable history-view twin of the already-fixed EntryRevision. All three now hand-write Equals / GetHashCode comparing every scalar field ordinally and their array bytes by content. No public member is removed or changed and no wire alias moves. (Orleans.Lattice 9.6.0)

  • Three more serializable read-surface value types no longer compare their byte arrays by reference, so structurally identical instances (and any that round-trip through serialization) now compare equal. DataReadResult and DataEntry (the point-read result and the key/value entry the data plane returns and accepts, from ILatticeDataApi.GetAsync and the bounded range read) and DeadLetterEntryRecord (returned from ILatticeStateQuery.ListDeadLettersAsync) are serializable records whose byte[] members (DataReadResult.Value, DataEntry.Value, and DeadLetterEntryRecord.ValuePreview) were left to the compiler-generated record equality, which compares an array with EqualityComparer<byte[]>.Default - reference equality. Two entries or records carrying identical content therefore never compared equal, and a value that crossed the Orleans grain boundary never equalled its pre-serialization self. This continues the same sweep already applied to LeafProjectionDigest, ViewDigest, EntryRevision, CrdtMemberChange, CrdtMemberValue and EntryRevisionRecord, closing the public read-surface siblings on the data and dead-letter surfaces that the earlier passes missed. All three now hand-write Equals / GetHashCode comparing every scalar field ordinally and their array bytes by content. No public member is removed or changed and no wire alias moves. (Orleans.Lattice.Api.Abstractions 9.6.0)

  • A routine WAL materialiser pin flush no longer rewrites every co-routed leaf on its shard. The durable pin store keeps one grain-state blob per shard with an entry per routed leaf consumer, and Orleans writes grain state whole, so persisting one leaf's frontier rewrote every neighbour on that shard - O(consumers per shard), 1.33 MB on an observed 7,841-file tree. LeafCursorReporter.FlushDurableMaterialiserFrontierAsync routed that routine flush through the write-through birth path, so every leaf activation and graceful deactivation forced a full awaited blob rewrite serialized through one non-reentrant grain. Under activation collection with roughly 9,500 leaves this backed the pin grain's non-reentrancy queue up to 165 deep with a single write taking 42.9 s, past the 30 s response timeout; the durable pin floor then never advanced, WAL GC could not trim, cold leaves replayed past MaxLeafReplayEntries, and callers timed out. The retention flush now goes through the coalesced report path, so the merge stays immediate and only the durable write is debounced - safe because the pin is a monotonic-max retention floor, and a durable pin that lags the in-memory frontier only ever retains more WAL. The pin grain additionally amortises its coalesced flush against its own measured cost, deferring a timer tick unless nine times the last write's duration has elapsed since it completed, which bounds the grain's write duty cycle to one tenth and leaves the rest for draining its queue; it is self-tuning, needs no configuration, and never defers an explicit durability point. Only the birth block-pin seed keeps write-through, where the barrier is genuinely load-bearing. (Orleans.Lattice 9.6.0)

  • The replication health check's sustained-degraded window no longer documents the opposite of what it enforces. LatticeReplicationHealthCheckOptions.UnhealthyAfter claimed that TimeSpan.Zero would treat every degraded sample as unhealthy on the next probe. It does the reverse: any non-positive value disables sustained-degraded escalation entirely, so an operator reaching for the strictest possible gating silently got the loosest. The XML doc, the prose page, and a test whose name asserted the opposite of its own assertion now all agree with the code, and the negative, infinite and small-positive cases are pinned by tests. No behaviour or public member changes. (Orleans.Lattice.Replication 9.6.0)

  • A page fill that finds nothing live no longer walks a shard's entire leaf chain in one call, and repocontext_list_repos no longer re-counts every embedding on every call. Two independent defects that compounded into a 605-second GetSortedEntriesBatchAsync timeout, and a partition head-of-line blocked behind it, on a repository mid-back-fill. (1) The shard root's leaf-walk budget - the leaf cap (MaxLeavesPerScanPage, default 64) and the wall-clock deadline (MaxScanPageDuration, default 5 seconds) that bound every read page fill - was consulted as ShouldYield(resultsCollected) and returned false outright whenever nothing had been collected yet. That was a guard against emitting an empty page the caller could not advance past, but it disarmed both bounds for the case that needs them most: a sterile run of leaves, where every row is moved-away, tombstoned, TTL-expired, or rejected by the predicate or the slot filter. A scan crossing thousands of such leaves - routine during a back-fill, a bulk delete, or a slot-filtered read - held the non-reentrant shard root for the whole chain and queued every other request to that shard behind it. The budget is now a pure work bound, and the ability to act on it is what each call site owns: the six page-fill methods return a leaf-boundary resume key on the page (EntriesPage.ResumeFromKey / KeysPage.ResumeFromKey, both additive) and accept it back, so a sterile page yields with a position the caller can genuinely continue from. Where no such position exists the walk still refuses to yield, so a bounded page is never one the caller cannot advance past. The wall-clock half of the budget now starts when the grain call begins rather than when it reaches its leaf loop, so the time already spent preparing the grain and traversing down to the start leaf counts against the page - a request that had already held the shard for minutes on a cold activation is no longer granted a further full MaxScanPageDuration of leaf walking on top of it. The deadline is sampled between leaf reads, so it bounds how many individually slow leaves one page fill chains together rather than preempting a leaf read already in flight. The boundary is direction-aware because the semantics differ: forward it is the visited leaf's exclusive high key, which is an inclusive lower bound for the resumed walk and therefore cannot be returned as an exclusive continuation token, while in reverse it is the leaf's inclusive low key, which is already exclusive as an upper bound. Resuming is strictly monotone in both directions, so a resumed page cannot revisit the leaf it resumed from. The same fix closes a latent hole on the caller side: the entry and key cursors treated an empty page as the end of the walk even when it claimed more, silently truncating any result set whose page happened to come back empty. (2) repocontext_list_repos reports embeddedVectorCount per repository, and computing it exactly means walking the whole vector-membership tree - the largest in the store. It was memoized against a cache generation that any membership write advances, so during an active ingest, which is exactly when an operator lists repositories to watch progress, the memo never hit and every call re-scanned tens of thousands of entries synchronously on the request thread. The read no longer performs that walk: it serves the last completed figure, states its currency through a new embeddedVectorCountPending flag, and schedules at most one out-of-band refresh per repository. A repository nothing has counted yet omits the count entirely rather than reporting 0, because "not measured" and "no vectors landed" are different answers and an operator acts differently on each. An exact count remains available, and priced, through the writer's explicit scan. The approximate-index sweep, which reads only repository ids, no longer builds a full summary per repository to discard everything but the id. (#1992) (Orleans.Lattice 9.6.0, Orleans.Lattice.Api.Mcp.RepoContext)

  • A read page fill now has a hard end-to-end ceiling, so a stalled prologue or a single leaf read that never returns can no longer hold a shard indefinitely. #1992 moved the page budget's clock to the start of the grain call, but the budget is cooperative: MaxScanPageDuration is sampled between leaf reads, so it can only stop a walk where the walk can name a resume position. Two shapes never reach such a point, and that residual limit was recorded at the time as deliberately out of scope. A prologue or descent that parks - preparing the activation, resolving options, or traversing down to the start leaf - never enters the leaf loop, so nothing samples the budget at all; and a leaf read already in flight cannot be preempted, so one read that never returns is unbounded no matter what the budget says. Either shape holds the deliberately non-reentrant shard root for as long as the underlying call takes and head-of-line blocks every other request to that shard: the reported incident was a single GetSortedEntriesBatchAsync holding a shard for 576.8 seconds against a 5 second budget, 115 times over. A new option, MaxScanPageStallDuration, is the hard ceiling that covers both. It defaults to null, meaning derive it: the effective ceiling is the silo's configured Orleans ResponseTimeout minus a five second headroom, floored at MaxScanPageDuration. That relationship is the load-bearing one - a ceiling at or above the response timeout is dead configuration, because the caller's own RPC deadline expires first, so the caller sees an anonymous Orleans timeout rather than the typed fault and, critically, nothing releases the shard. Deriving keeps that ordering true for any configured timeout instead of only for the Orleans default, so a deployment that tightens ResponseTimeout does not silently lose the guard; an explicit value overrides it, and Timeout.InfiniteTimeSpan disables it. When it fires the call is abandoned and faulted with a typed, retriable ScanPageStalledException (a TimeoutException), which releases the shard so its queue drains and lets the caller retry from its last continuation token; abandoning a parked await is the same shape the activation-readiness and shadow-forward deadlines already use, and the deadline is plumbed with a CancellationTokenSource rather than Task.WaitAsync(TimeSpan), which this codebase has observed failing to fire. MaxScanPageDuration is unchanged and remains the graceful bound that yields a resumable page - the new ceiling only fires where the graceful one structurally cannot, and validation requires it to sit strictly above it. Every fault names the phase it stalled in (prologue, descent or leaf-walk), so the next occurrence is self-diagnosing rather than a bare timeout, and a new counter, orleans.lattice.shard_root.scan_page.stalls, is dimensioned by that phase and charted on the existing shard-root wedge-guard panel. Closing the prologue gap also removes a grain call from each of the fifteen page-fill entry points: all three bounds are plain per-tree options rather than structural pins, so they resolve synchronously from the options monitor instead of through the registry round trip every site previously awaited - which is precisely what lets the budget be armed as the literal first statement of the call. Each entry point is therefore now a synchronous wrapper over an async core, and a new structural test asserts that of all fifteen, so a future edit cannot silently reintroduce an await in front of the clock. That test also found four further defects, all of them the same #1992 defect surviving in the shard root partials that sweep missed: the operator projection rebuild and the materialiser-lag walk in ProjectionAdmin, and the shard diagnostics page and the leaf byte-footprint refresh in Diagnostics, each built its budget after an await and so had no start clock at all, leaving its entire prologue unbounded - the lag walk's worst of all, since that prologue awaits a WAL head offset per partition before it ever reaches a leaf. All four are fixed here with the rest, and all fifteen entry points are now covered by the structural test. The per-call phase probe and its cancellation source are pooled and the guard adds no state machine when it is disabled or the page completes synchronously, so the ceiling costs nothing on a healthy read. (#2002) (Orleans.Lattice 9.6.0, Orleans.Lattice.Dashboards 9.6.0)

  • Four read-only leaf-chain walks in the shard root no longer hold the shard for the whole chain, and two uncapped tree descents can no longer spin forever. The sweep that bounded the shard's traversal walks (#1955, #1957, #1971) audited ShardRootGrain.Traversal.cs, so four walks living in the Diagnostics and ProjectionAdmin partials were never reached: a deep diagnostics scrape, a materialiser-lag query, an operator-forced storage-usage re-anchor, and an operator projection rebuild each walked every leaf inside one call on a non-reentrant grain, so on a wide shard a single dashboard scrape or admin verb queued every other request to that shard behind it for the length of the chain. All four are now work-bounded on the same MaxLeavesPerScanPage / MaxScanPageDuration budget as the rest of the shard, resuming by key rather than by leaf grain id so a leaf split or consolidated away between batches is re-descended rather than read as a clean end of chain. Three of the four route through the same BoundedLeafWalk helper the background coordinators adopted in #1973, reached through a new in-shard entry point that takes an already-resolved start leaf, because a shard resolving its own start leaf through IShardRootGrain would self-call and deadlock; sharing that helper also makes these walks immune by construction to the disarmed-budget hole a sterile run can otherwise hit (#1992). The projection rebuild is the one deliberate exception and says so at the site: it deactivates each leaf as it goes, so it must read the sibling pointer and the resume key before the rebuild rather than after, which is the one ordering the shared helper cannot express - reading them afterwards would force an immediate WAL replay purely to obtain a cursor and turn the deliberately lazy rebuild into an inline one. Every answer is bit-for-bit what the single-call walk produced. The WAL heads a lag query measures against are captured on the first batch only, so a tree committing mid-walk cannot inflate the reported lag; the storage-usage totals are re-anchored only on the batch that completes the walk, and only from the whole-chain figure, so the shard never advertises a fraction of its own footprint to a concurrent reader; and the diagnostics report's O(1) shard-wide fields (depth, hotness, lifecycle flags) are computed on the first batch only rather than re-derived per batch for an identical answer. Each legacy verb is retained as a wire-compatible shim that drives the bounded protocol to completion in one call, so a caller from an older build is unaffected. Separately, the two leftmost-path descents behind the diagnostics and footprint walks were written while (!childrenAreLeaves) rather than the bare while (true) that audit swept for, so both were still uncapped: a corrupt or cyclic leftmost-child pointer spun the non-reentrant shard root forever, and both now fail with a typed exception at the same MaxTreeDescentLevels cap every other descent uses. One walk is deliberately left unbounded and is instead instrumented through the existing slow-atomic-walk warning: the transaction-terminal chain collection must observe one terminal HLC across the whole chain, and yielding mid-walk would tear the saga it commits. (#1972) (Orleans.Lattice 9.6.0)

  • Two distinct caller identities that alias under a bare-delimiter join no longer coalesce onto one shared metrics sampling loop. SharedMetricsSampler.BuildIdentityComponent renders the visibility-determining identity of a caller - its subject id, transitively-expanded group closure, and claim bag - into the signature that decides when two dashboard subscribers may share a single identity-filtered sampling loop. It joined the group ids with a bare , and each claim as key=value without length-prefixing any field, so two distinct identities could collapse to the same string: groups ["a,b"] and ["a","b"] both rendered a,b, and claims {"a=b":"c"} and {"a":"b=c"} both rendered a=b=c. A wrongly shared loop serves one subject the first subscriber's identity-filtered view, so an aliased pair breaks the "identical signature implies identical read access" guarantee (#971) - the same delimiter-aliasing class the #1952 projection-version fix closed. Every subject id, group id, and claim key and value is now length-prefixed, making the signature injective so no delimiter a field happens to contain can shift a field boundary. The signature is ephemeral in-memory coalescing state, so no persisted data or rebuild is affected. (Orleans.Lattice.Api.State 9.6.0)

  • A sliding-expiration blob cache entry that degrades to "never expires" no longer reports a spurious slid deadline. BlobCacheEntryExpiration.Slide computes the next sliding deadline for an entry, but inverted its guard for the no-effective-expiry case: a null effective expiry is the documented never-expires state that the sibling IsExpired and the FromMetadata reader already honour (a partially written or corrupt blob whose sliding window parsed but whose absolute expiry did not degrades to never-expires), yet Slide returned the freshly computed candidate deadline instead of null there, turning a never-expiring entry into one that expires at the slid time. The guard now returns the candidate only when it strictly extends a set effective expiry, and null whenever the entry never expires, matching the never-expires semantics its own siblings enforce. (Orleans.Lattice.Caching.AzureBlob 9.6.0)

  • A redefined view whose filter differs only in where a colon falls can no longer reuse a stale projection version and skip its rebuild. The view maintainer gates a rebuild on the projection version, and the three built-in projections (PredicateLatticeViewProjection, LatticeFoldProjection, AggregationLatticeViewProjection) derived that version by joining each predicate node's caller-controlled member path and constant with a bare : delimiter. A value containing : could shift a field boundary so that two structurally different filters serialized identically and hashed to the same version, leaving a redefined view serving results built for the old filter (the same aliasing class as the #1952 operation-id fix). Both variable-length fields are now length-prefixed, making the node encoding injective. Because this changes the projection-version hash for every filtered view, the first drain after upgrading recomputes the version and performs a one-time rebuild - the designed response to a version change - after which the version is stable again. (Orleans.Lattice 9.6.0)

  • A misconfigured clamp bound no longer lets a state-API request exceed its configured maximum. The four query clamp helpers in LatticeStateQuery (ClampHistoryLimit, ClampHistoryPreviewBudget, ClampPageSize, ClampScanPreviewBudget) fell back to the raw configured default whenever the requested value was below one, without also capping that default at the configured maximum - and nothing validates LatticeApiStateOptions, exactly as the sibling LatticeDataApi clamp path already notes. A default configured above its own maximum therefore let a defaulted request return more history rows, a larger preview budget, or a larger page than the maximum was meant to cap, while the same helpers still capped an explicitly requested value correctly. Each below-one fallback now returns Math.Min(default, maximum), so a defaulted request can never exceed the maximum; a correctly ordered configuration (default at or below maximum) is unaffected. (Orleans.Lattice.Api.State 9.6.0)

  • An effectively-unbounded backup retention window no longer faults the whole retention pass. BackupSchedulerGrain.ComputeKeepSet computed its age cutoff as now - RetentionMaxAge, but the schedule-options validator rejects only a non-positive window, so a very large RetentionMaxAge (up to TimeSpan.MaxValue) is permitted and underflowed DateTimeOffset, throwing ArgumentOutOfRangeException and failing the retention pass after an otherwise successful capture. A "retain for practically forever" window means keep every manifest, so the cutoff now saturates at DateTimeOffset.MinValue instead of faulting, retaining all manifests as intended. (Orleans.Lattice.Backup 9.6.0)

  • A backup health-monitor shutdown race no longer faults the host on the way out. BackupHealthMonitorActivationService retries a transient startup failure - the silo not yet being ready to serve its grain call - and caught that failure with catch (Exception ex) when (!stoppingToken.IsCancellationRequested). An exception filter is evaluated at throw time, so a stop request arriving between the grain call failing and the filter running made the filter false and left the exception uncaught. It escaped ExecuteAsync and faulted the BackgroundService task, and because BackgroundServiceExceptionBehavior defaults to StopHost, a benign "silo not ready" raised on the way out could take the whole host down - a shutdown-time race turning a retryable startup condition into a host stop. The failure is now caught unconditionally with the cancellation check moved inside the handler: once stopping has been requested the call was going to be abandoned anyway, so whatever it threw is not a fault. A regression test drives the interleaving deterministically (cancel, then throw) and fails against the old handler. The existing cancellation test was also unsound - it signalled its completion source before throwing, so the case sometimes exercised the throw-racing-cancellation path instead of the delay-cancellation path it is about - and now signals in a finally, with the other path covered by its own test. (#1959) (Orleans.Lattice.Backup 9.6.0)

  • Two more MCP tool-argument enum parsers no longer accept a numeric ordinal or a comma-combined name that folds onto a defined member. The backup tool's scopeKind and restore mode parsers (BackupToolMappings.ParseScopeKind and ToRestoreMode) matched their value with Enum.TryParse plus an Enum.IsDefined guard - the same guard the #1941 hardening added to three sibling parsers - but that guard rejects only an out-of-range ordinal. Both still accepted an in-range numeric string ("1" binds to the defined BackupScopeKind.Prefix and LatticeRestoreMode.ShadowCutover) and a comma-combined name, which Enum.TryParse treats as a bitwise OR: "WholeTree,Prefix" = 0|1 = 1 folds onto Prefix and "InPlace,ShadowCutover" = 0|1 = 1 folds onto ShadowCutover, so a wire caller could select a backup scope or restore mode it never named, each violating the parser's documented "reject unrecognised value" contract. These were the two members the #1941 sweep missed. Each now adds the ordinal round-trip name-equality check (string.Equals(parsed.ToString(), value, StringComparison.OrdinalIgnoreCase)) the sibling parsers use, which rejects both the numeric and the combined forms while still parsing a genuine name in any case. (Orleans.Lattice.Api.Mcp 9.6.0)

  • A misconfigured range-read page-size bound no longer faults every bounded range read. LatticeDataApi.ReadRangeAsync clamped its page size from above with Math.Min(size, MaxRangePageSize) but, unlike the sibling range-delete path that floors its step at one (Math.Max(1, RangeDeleteStepSize)), applied no floor of its own - and nothing validates LatticeApiDataOptions. A non-positive MaxRangePageSize (or DefaultRangePageSize) therefore collapsed every resolved page size to zero, which the cursor grain rejects with ArgumentOutOfRangeException, so a single misconfigured option turned every bounded range read into a fault instead of a bounded page. The clamp now floors at one, mirroring the range-delete step, so a non-positive bound yields a one-entry page that still paginates rather than a fault. (Orleans.Lattice.Api.Data 9.6.0)

  • Seventeen broken in-page documentation links now land where they say they do. Cross-references into the core API reference, the metrics and chaos-test documents, the tree-storage guide, and three replication documents pointed at anchors that do not exist, so following one arrived on the correct page at the wrong position - on github.com as much as in any rendered site. The failures were the usual heading-slug traps: an anchor written for a shortened heading (#resize for "Resize and reshard"), and headings whose punctuation leaves a doubled hyphen in the slug ("Saga / coordinator / lifecycle" becomes #saga--coordinator--lifecycle). Every replacement was taken from the anchor the renderer actually generates rather than derived by hand. The new documentation pull-request check fails closed on any broken relative link or in-page anchor, so this class of rot cannot silently return. (Orleans.Lattice 9.6.0, Orleans.Lattice.Replication 9.6.0)

  • Emptiness probes and the remaining shard count paths no longer walk a whole leaf chain in one call. AnyAsync, CountWithMovedAwayAsync, and CountForSlotsAsync each walked every leaf in the shard inside a single non-reentrant call, head-of-line-blocking every other request to that shard for the duration. Each now has a work-bounded form that returns a resume position, with the original retained as a wire-compatible wrapper, and the real callers - the tree emptiness probe and the two per-slot count fan-outs - drive the bounded form so the bound actually engages. AnyAsync short-circuits at the first non-empty leaf, so its bound only ever engages on the empty or fully-tombstoned shard that was the unbounded case; a negative answer is believed only once the shard reports no resume position, so a bounded batch can never be mistaken for a genuinely empty shard. The moved-away slot sets are unioned across batches rather than replaced, because a slot observed only in an earlier batch is as real as one observed in the last. CountForSlotsAsync also gains the past-range exit, paid only when a leaf yields no in-range keys. Resumption is by key throughout, never by leaf grain id, since Orleans grains are virtual and a leaf reclaimed between batches would otherwise activate empty, report no sibling, and silently truncate the walk. (#1971) (Orleans.Lattice 9.6.0)

  • Counting a shard no longer walks its whole leaf chain in one call, and a narrow range no longer costs a whole-shard walk. CountAsync on a shard walked every leaf inside a single non-reentrant call, so it head-of-line-blocked every other request to that shard for the duration - and, uniquely among the range-scoped reads, it had no past-range exit at all, so counting a narrow range still walked to the end of the chain. The shard now exposes a work-bounded batch that counts a bounded number of leaves and returns a resume position, and LatticeGrain drives those batches to completion, so ILattice.CountAsync remains one logical call that simply makes several short shard calls instead of one long one. Resumption is by key rather than by leaf grain id, because Orleans grains are virtual and a leaf reclaimed between batches would otherwise activate empty, report no sibling, and silently end the walk. A resume position is emitted only when a next leaf actually exists: unlike a repeated range delete, which is idempotent, re-counting a leaf would inflate the answer and never terminate. Atomic visibility is unchanged, because it comes from the saga-decision snapshot pinned for the lifetime of the enumeration rather than from the granularity of any one shard call, and the multi-batch loop stays inside a single LatticeGrain invocation so that pin survives it. The past-range probe is paid only when a leaf contributes nothing, so a productive leaf keeps its single-call cost. (#1971) (Orleans.Lattice 9.6.0)

  • Tombstone compaction can no longer be silently truncated by a stale resume cursor. TombstoneCompactionGrain walks a shard in batches and persists its resume position as a leaf grain id. Orleans grains are virtual, so a cursor naming a leaf whose state no longer exists does not fail - it activates a fresh, empty grain that reports no sibling, which the walk reads as a clean end of shard. It would then report the shard fully compacted with the remainder never visited: no exception, no metric, and no log line to distinguish it from genuine completion. This was not reachable in practice, because the leaf chain is grow-only (splits and bulk-load grafts insert, nothing unlinks) and leaf state is cleared only when a tree is purged, which by contract runs against an already-offline tree - but that safety was an emergent property of the rest of the design rather than anything the compaction code asserted, so a later change that reclaimed leaves would have reintroduced it silently. The resume path now validates the cursor and falls back to re-walking the shard from its leftmost leaf when it does not resolve, which is safe because per-leaf compaction is idempotent, and is the same trade the sibling dirty-set resume path already made. A probe that throws is treated as live rather than dead, so a transient fault is left to the existing per-leaf retry instead of restarting the whole shard. (#1970) (Orleans.Lattice 9.6.0)

  • The replication config report now names every tree that actually replicates, not just the ones enabled at runtime. A replication-enabled host decides whether to ship a tree from two sources: the runtime configuration tree an operator writes through enable and disable, and the static replicated-tree map supplied as deployment configuration, which acts as a fallback floor the runtime tree layers on top of. The report described only the first, so an estate configured entirely through deployment configuration - which is exactly what the reference architecture's own -ReplicationTrees flag produces - answered "no trees" in every region while demonstrably converging writes across three of them and refusing writes that disagreed with its declared merge mode. This is the one tool an operator reaches for to answer "which trees replicate here?", and on a correctly configured estate it answered "none". Both sources are now reconciled under exactly the precedence the commit path applies, so the report describes what the host does rather than what one of its two inputs says: a divergent runtime mode still fails closed and is never resolved by a static declaration, an enabled unambiguous runtime mode still wins, and otherwise the static declaration is the floor that keeps the tree shipping. Each entry carries a new Source (Runtime, Static, or RuntimeAndStatic) naming which one is in force, which also surfaces a trap that was previously invisible: because the static map is a floor, a tree reported Static keeps shipping after a runtime disable and is turned off by editing deployment configuration instead. Nothing on the shipping path changed, and the addition is wire-compatible - an entry from a peer that predates the field reads as Runtime, the only source the report used to describe. (#1940) (Orleans.Lattice.Replication 9.6.0, Orleans.Lattice.Api.Abstractions 9.6.0, Orleans.Lattice.Api.Replication 9.6.0, Orleans.Lattice.Api.Replication.Grpc 9.6.0, Orleans.Lattice.Api.Mcp 9.6.0)

  • Installing the moved-away seal during a shard split no longer holds the shard for the sum of its per-leaf writes. When an adaptive split or a consolidation migrates virtual slots, the source shard records the change on every leaf in its chain, and it did so one leaf at a time. ShardRootGrain is non-reentrant and every read path reaches a leaf through it, so the shard was unavailable to every other caller - point reads, writes, further splits, health probes - for the total of those writes, on a shard already hot enough to have triggered the split. The walk still happens inside a single grain turn, which is what makes the installation unobservable and is load-bearing: a half-installed seal is wrong in the dangerous direction, because an already-sealed leaf reports a key the source shard still owns as missing. What changed is that the per-leaf writes, which have no ordering relationship with one another, now issue with bounded concurrency instead of serially, while chain discovery stays sequential. Measured over a real cluster reshard at 121 leaves per shard, the walk fell from roughly 190 ms to roughly 60 ms per shard against in-memory storage, where a per-leaf write is nearly free; against a real store, where each is a network round trip, the gap widens. (#1960) (Orleans.Lattice 9.6.0)

  • A leaf grain shutting down no longer overruns its deactivation deadline and floods the log. BPlusLeafGrain.OnDeactivateAsync performs storage I/O, and three of its four awaits accepted no cancellation token - one passing CancellationToken.None to a reporter that accepts one, and the deactivation-time snapshot capture hard-coding it on the blob write, the most expensive operation on the path - so none of them could be interrupted when Orleans' deactivation deadline fired. The hook already swallowed storage failures, which made it look defended, but an uninterruptible await means the grain never returns: Orleans reports the overrun as a TaskCanceledException raised in its own frame, which the grain's own exception handling can never observe. The effect arrives as a burst rather than a trickle, because at the end of a WAL replay several thousand leaves go idle together, contend for the same storage provider, and slow each other past the deadline - observed on a single-silo deployment as roughly 1,100 error-level lines in a single minute, accounting for over 99% of all errors it logged. All three now honour the token and abandon promptly, which the existing handler absorbs, and an abandoned capture is swallowed without logging so one flood is not traded for another. None is correctness-critical on this path: the digest is staleness-tolerant and republished on the next mutation, the durable frontier is re-reported on the next activation, an abandoned snapshot simply leaves coverage unadvanced so the write-ahead log is retained rather than trimmed ahead of it, and the log remains the durability boundary throughout. (#1965) (Orleans.Lattice 9.6.0)

  • Three MCP tool-argument enum parsers no longer accept a numeric ordinal or a comma-combined name that folds onto a defined member. ReplicationToolMappings.ToMergeMode, the repository-context Scan handler's scope argument, and the repository-context Remember handler's kind argument each matched their value with Enum.TryParse plus an Enum.IsDefined guard - the hardening #1941 applied - but that guard only rejects an out-of-range ordinal. It still accepted an in-range numeric string ("11" binds to the defined RwSet) and a comma-combined name, which Enum.TryParse treats as a bitwise OR: "OrSet,GSet" = 1|10 = 11 folds onto the distinct defined LatticeMergeMode.RwSet, "Packages,Symbols" folds onto RepoContextScanScope.Memory, and "Decision,Note" folds onto MemoryKind.Memory - so a wire caller could enable a tree under a merge mode, scan a scope, or file a durable memory entry under a kind it never named, each violating the parser's documented "reject unrecognised value" contract. Each now adds the ordinal round-trip name-equality check (string.Equals(parsed.ToString(), value, StringComparison.OrdinalIgnoreCase)) that CrdtProvenanceDecoderRegistry.TryGet already uses, which rejects both the numeric and the combined forms while still parsing a genuine name in any case. (Orleans.Lattice.Api.Mcp 9.6.0, Orleans.Lattice.Api.Mcp.RepoContext)

  • A single shard range-scan call can no longer hold a shard for minutes, blocking every other request to it. The six paged range-scan loops in ShardRootGrain were bounded by their output - the number of results collected - and walked the leaf sibling chain until the page filled. A leaf can cost a grain call, plus a snapshot rehydration or WAL replay when cold, and still contribute nothing to the page: its entries may all be filtered as moved-away by an adaptive split, tombstoned and still inside TombstoneGracePeriod, TTL-expired, or rejected by a pushed-down predicate. The worst case was therefore O(leaves in range) inside one call, and because shard reads are deliberately non-reentrant - the alternative reintroduced a chaos-reshard mid-saga invariant violation and was reverted - that call head-of-line-blocks every other request to the shard for its whole duration. On a live deployment one such call held a shard for roughly six minutes, queueing requests behind it for 159-360s and emitting 14,363 Orleans long-request warnings. Each page fill is now also bounded by the work it does, via the new MaxLeavesPerScanPage (64 leaves) and a MaxScanPageDuration safety net for individually slow cold leaves, returning a partial page and releasing the shard so other traffic interleaves. The bound is applied only once the page holds at least one result, so a caller can always derive its next continuation token and make forward progress - an empty page reporting more available would strand a caller that resumes from the last returned key - which keeps every existing caller correct with no wire-format change. Atomic visibility is unaffected: it comes from the saga-decision snapshot pinned for the lifetime of the enumeration, not from the granularity of an individual shard call, so an extra page boundary observes the identical view. Alongside it, five bare while (true) tree descents now honour the MaxTreeDescentLevels cap the write descent already used, so a cyclic routing pointer surfaces as a typed exception rather than an unbounded loop. (#1955) (Orleans.Lattice 9.6.0)

  • A wide range delete no longer holds a shard for its whole duration. ShardRootGrain's range delete walked its entire leaf chain inside one non-reentrant call, so deleting a wide range blocked every other request to that shard - point reads, writes, splits and health probes alike - until the walk finished. Each shard's walk is now work-bounded by the same MaxLeavesPerScanPage budget the read paths use: it tombstones a bounded number of leaves, returns the key to resume from, and releases the shard so other traffic interleaves, with the tree-level call driving each shard to completion. Resumption is by key rather than by leaf grain id, because Orleans grains are virtual and a leaf reclaimed between two batches would activate empty with no siblings, silently ending the walk and leaving part of the range undeleted. This weakens no documented guarantee: ILattice.DeleteRangeAsync already fanned out to every physical shard in parallel with no cross-shard atomicity, and its documented visibility is per-key only - "a concurrent saga may be observed as committed for some keys and pending for others" - so the whole-shard atomicity of the old walk was an implementation artifact rather than a contract. The end-state guarantee, that every key in the range is tombstoned, is unchanged. Each batch publishes its own DeleteRange observer notification carrying that batch's matched-key closure; their union is the closure the single notification previously carried, so replication apply still reproduces the tombstone set without re-evaluating the predicate. (#1956) (Orleans.Lattice 9.6.0)

  • A shard held by one of the remaining atomic leaf-chain walks now says which walk is holding it. Four leaf-chain walks cannot be work-bounded, because for each the fact that no other message runs on the shard between the first and last leaf is the invariant the surrounding protocol depends on: marking moved-away slots runs immediately before a splitting shard enters Reject phase so no read crosses the Swap boundary observing an unmarked leaf (a half-installed seal is wrong in the dangerous direction - an already-sealed leaf reports a key the source still owns as missing), unmarking them during a consolidation is the mirror image against the donor freeze, a snapshot-cursor baseline reads its captured WAL head only after every leaf has frozen so the baseline describes a single instant, and shard purge runs on an already-offline tree and is destructive so cannot resume once a sibling pointer is cleared. Each now records that reasoning in place, so a later change cannot apply the work bound uniformly without first reading why it must not, and each logs a warning naming the operation and its leaf count when it holds the shard past a short threshold - Orleans' own long-request warnings name the blocked messages and say nothing about the blocker. For the two where the stall is worth removing rather than explaining, the real fix is a redesign rather than a budget and is tracked separately (#1960, #1961). (#1956) (Orleans.Lattice 9.6.0)

  • The CRDT provenance decoder registry no longer resolves a decoder for a numeric or combined shape string. CrdtProvenanceDecoderRegistry.TryGet(string?) matched the shape tag with a bare Enum.TryParse<LatticeMergeMode>, which also accepts a numeric ordinal string and a comma-separated combination, so "3" resolved the ordinal-3 VersionVector decoder and "OrSet,GSet" folded to the defined RwSet ordinal and resolved a different decoder again - both violating the documented contract that a shape tag is the mode's Enum.ToString() form (e.g. "OrSet") and that an unrecognised shape yields false. The shipped decode path only ever supplies an enum name or null (through ResolveCrdtShape), so no product caller reached the defect, but the public method now guards with Enum.IsDefined and an ordinal round-trip name-equality check - the same Enum.IsDefined hardening #1941 applied to the MCP argument parsers, extended with the round-trip check that also rejects an in-range ordinal like "3". (Orleans.Lattice 9.6.0)

  • The replication accepted-secret change token no longer aliases distinct secret sets that differ only around a NUL character. StableSecretSetHash.Compute delimited entries with a single 0x00 separator byte, which is indistinguishable from a 0x00 that occurs inside an entry - both an embedded '\0' and the high byte of every ASCII code unit hash as 0x00 - so two distinct ordered secret sets could produce the identical byte stream and the same Version token (for example {"a\0\0c"} and {"a", "\0", "c"}, or {"\0"} and three empty entries). A secret source bound from configuration can reach these shapes, and an aliased token lets a consumer skip a rebuild that a genuine secret-set change should trigger. Each entry is now length-prefixed so its extent frames its content in the digest, mirroring the sibling ReplicationContentHash that was corrected the same way; the token is an in-memory change token only and never travels on the wire, so the algorithm change is safe. (Orleans.Lattice.Replication 9.6.0)

  • The tenant quota report now names the burst ceiling admission control actually enforces. TenantQuotaUsageMapping.BurstCeiling computed the burst-adjusted ceiling as ceiling / 100 * burstPercent (divide before multiply), which floors the allowance to zero for any ceiling below 100 and understates any ceiling that is not a multiple of 100 - so a tenant with a 50% burst over a ceiling of 10 was reported as admitting 10 while the evaluator admits 15, and a 10% burst over 250 was reported as 270 while the evaluator admits 275. The tenancy engine's TenantQuotaEvaluator had already been corrected to multiply through a 128-bit intermediate and clamp to long.MaxValue; the read-only report mapping had drifted from it. The mapping now mirrors the evaluator exactly, so every surface that consumes the report (Explorer, MCP, and the gRPC binding) names the same ceiling the engine enforces. (Orleans.Lattice.Api.TenantAdmin 9.6.0)

  • A ulong transform constant above long.MaxValue is no longer silently corrupted. LatticeValueTransformTranslator captured a ulong literal via Integer(unchecked((long)ul)), wrapping any value above long.MaxValue to a negative integer (ulong.MaxValue became -1) and lowering to a projection constant that no longer matched the intended value. It now captures such a value as a Double when it exceeds long.MaxValue and an exact integer otherwise, matching the contract the sibling LatticePredicateTranslator already enforces. (Orleans.Lattice.Schema 9.6.0)

  • The WAL rebalance recommendation now names the real hot partition on the default storage account. StoragePressureCollector normalises a null partition provider key to the default catalogue key when it attributes retained bytes, so the hottest account it selects carries the normalised key; its partition search, however, compared the raw (still null) partition key against that normalised key, never matched, and collapsed the advice to the empty-tree, partition-zero fallback. The search now applies the same normalisation, so a recommendation for the default account names the actual hot tree and partition instead of "" / 0. (Orleans.Lattice.Scaling 9.6.0)

  • The MCP tool surface now rejects malformed tool arguments instead of silently accepting them. Three independent argument-validation defects shared one root cause: a value was trusted without confirming it named a real, in-range option. First, every facade tool wrapped by CredentialStampingTool silently ignored an unknown argument name, so a misspelled or unsupported argument (treeid for treeId, say) was dropped and the call proceeded as if it had never been supplied, masking the caller's mistake; the wrapper now computes the accepted argument names from the tool's own input schema and throws an McpException naming the offenders. Second, the backup tools' ToRestoreMode and scope parsers used Enum.TryParse without Enum.IsDefined, so a numeric enum string such as "99" parsed to an undefined enum value rather than raising the documented ArgumentException. Third, the repository-context Scan and Remember handlers had the same Enum.TryParse-without-Enum.IsDefined gap on their scope and kind arguments, so an out-of-range numeric ordinal bypassed the documented McpException. All three now guard with Enum.IsDefined (or an explicit allow-set), matching the existing ReplicationToolMappings.ToMergeMode pattern. (#1941) (Orleans.Lattice.Api.Mcp 9.6.0, Orleans.Lattice.Api.Mcp.RepoContext)