Instrument catalog: Leaf-level (sourced from BPlusLeafGrain)

This page is part of the documentation for Orleans.Lattice 9.9.0 (release line 9.9), built 2026-10-04. It is also published as markdown, with every table and list, at instrument-catalog-2.md, and llms.txt lists every page.

Part of Instrument catalog, in Metrics.

Leaf-level (sourced from BPlusLeafGrain)

Name	Kind	Unit	Description
orleans.lattice.leaf.write.duration	Histogram<double>	ms	Duration of a leaf grain's persisted state-row write (IPersistentState.WriteStateAsync) - i.e. storage-provider write latency - recorded with no kind tag. Those writes come from the leaf's topology and bookkeeping paths: sibling, parent, key-range, tree-id and shard-index updates, splits, moved-away slot marks, successor reclaim, snapshot load-hint banking, and projection checkpoint flushes and rebuilds. The same instrument also times three WAL appends, each tagged kind: compact (the append a tombstone compaction issues), merge (the append for a batched last-writer-wins merge into the leaf - for example replication apply, a tree merge, a snapshot copy, or a shard split or consolidation migration - and for the entries a leaf split transfers into its new sibling) and backstop (the cross-migration backstop append on a transaction terminal), so filter on kind rather than reading the unfiltered distribution as one population - the bundled Overview and CommitPath panels aggregate over it, and a query for the state-row writes alone selects kind="". Tagged tree.
orleans.lattice.leaf.commit.duration	Histogram<double>	ms	Per-step latency on the leaf commit path. Tagged tree and step=wal (commit-log-writer append), step=apply (the in-memory merge, plus any relocation of out-of-span rows or leaf split the write triggers; the merge itself persists no grain state), step=observer (IMutationObserver fan-out), or step=digest, which hands the write's projection-digest change to the parent internal node: with the default DigestCoalescingWindowMs (5 ms) it schedules, or joins, a publish the leaf sends when the window elapses, outside the commit, so the step includes the awaited cross-grain publish itself only when the window is 0. When the write leaves the digest unchanged, the leaf has no parent, or digest maintenance is off, the step instead covers the leaf's best-effort byte-footprint publish to its shard root. Emitted from every foreground commit-path code path (single-key SetAsync / DeleteAsync, the per-leaf batched SetManyAsync and conditional-batch commits, per-leaf DeleteRangeAsync, and saga prepare writes, which ride the same set paths) so operators can attribute latency between durability, projection, observer overhead and the digest hand-off independently. A per-leaf DeleteRangeAsync records no observer step: its observer publish is made once per shard, by the shard root after the leaf-chain walk, rather than per leaf. The structural digest publishes of a projection-checkpoint flush, a saga terminal, a tombstone-reap compaction or a cross-shard merge run outside these paths and are not recorded here, but a leaf split that a write triggers runs inside that write's apply step, its own inline digest publish included.
orleans.lattice.leaf.commit.in_flight	Histogram<int>	{commit}	Concurrent foreground commits in flight on a single leaf at the moment a new commit enters the commit path. Tagged tree only (no shard tag - leaves are routed per-key so the same leaf grain can be addressed by multiple shards under online reshard). The leaf's mutator surface is [AlwaysInterleave], so a commit that arrives while another is still awaiting its WAL append enters the commit path rather than queueing behind the turn: a flat-zero series means commits never overlapped on one leaf activation, and a sustained tail is that overlap - saga-driven prepare arrivals or producer-side fan-in landing on the same leaf.
orleans.lattice.leaf.scan.duration	Histogram<double>	ms	Duration of leaf-level range scans. Tagged tree and operation=keys (a keys scan) or operation=entries (an entries scan).
orleans.lattice.leaf.compaction.duration	Histogram<double>	ms	Duration of one leaf's tombstone-compaction pass, clocked from the start of the grain call - the activation replay barrier included - to the end of its reap loop, so it measures the same quantity the pass's work budget bounds. A visit that returns early because the leaf is unchanged since its last complete compaction (counted outcome=noop), and a call that throws before its reap loop ends, record no sample. Every pass that truncated records a sample at or above BackgroundDrainMaxDuration, but the converse does not hold: the budget is checked only at intervals during the scan once an entry has been condemned, and after each removal while condemned entries remain, so a pass that condemned nothing (a cold activation's replay barrier alone can spend the budget) or whose budget ran out after its last check finishes, and records a sample above the budget as well. outcome=partial on orleans.lattice.compaction.leaves.visited is the exact count of truncated passes; a non-positive BackgroundDrainMaxDuration disables the bound, and then no pass truncates. Tagged tree, plus trigger (as on orleans.lattice.compaction.pass.duration) when at least one compaction policy knob (MinTombstoneRatioForCompaction or MaxLeafEntriesBeforeForcedCompaction) is non-default; with every knob at its default the tag is omitted.
orleans.lattice.leaf.tombstones.reaped	Counter<long>	{tombstone}	Tombstones (from explicit DeleteAsync / DeleteRangeAsync) permanently removed by compaction. Tagged tree, plus trigger under the same condition as orleans.lattice.leaf.compaction.duration.
orleans.lattice.leaf.tombstones.created	Counter<long>	{tombstone}	Tombstones newly written by DeleteAsync (1) or DeleteRangeAsync (N). Tagged tree.
orleans.lattice.leaf.tombstones.expired	Counter<long>	{tombstone}	Live entries reaped by compaction because their per-entry TTL (set via SetAsync(key, value, TimeSpan)) elapsed past the configured grace period. Separate from reaped so operators can distinguish TTL churn from explicit-delete throughput. Tagged tree, plus trigger under the same condition as orleans.lattice.leaf.compaction.duration.
orleans.lattice.compaction.pass.duration	Histogram<double>	ms	End-to-end wall-clock duration of one tombstone-compaction pass over the tree's shards, recorded once when the pass completes and measured from when the current coordinator activation started or resumed it, so a pass resumed after its coordinator reactivated reports only the resumed part. Tagged tree, trigger=reminder (baseline reminder-driven sweep), trigger=ratio (policy trigger fired by MinTombstoneRatioForCompaction), trigger=size (policy trigger fired by MaxLeafEntriesBeforeForcedCompaction), or trigger=operator (out-of-cycle pass requested via ILattice.CompactShardAsync).
orleans.lattice.compaction.leaves.visited	Counter<long>	{leaf}	Per-leaf compaction outcome. Tagged tree, outcome=reaped (at least one tombstone or TTL-expired entry physically removed and the leaf finished) or outcome=noop (nothing was removed: the leaf was unchanged since its last complete compaction, or every tombstone and expired entry it held was still inside the grace window) or outcome=partial (the leaf reaped what its bounded turn allowed and stopped on its work budget with condemned entries still outstanding, so its dirty mark is retained and a later pass re-nominates it - a sustained rate here means compaction is reclaiming ground more slowly than the tree is condemning it, which is a capacity signal rather than a fault) or outcome=skipped (the coordinator's call to the leaf threw - any exception, a request timeout included; on the dirty-set fast path the fault is absorbed once the leaf's dirty mark is retained above the pass watermark and the walk continues, so this arm, recorded once per blocking leaf per pass, is the only signal that a specific leaf is wedged; on the legacy chain walk, or when that mark cannot be retained, the fault fails the batch and the shard's retry-then-skip handling (orleans.lattice.compaction.shard.retries, orleans.lattice.compaction.shard.skipped) takes over, and a retry that faults again records the leaf again), path=walk (legacy chain walk) or path=dirty-set (dirty-leaves fast path) when a path is in scope, and - when at least one compaction policy knob (MinTombstoneRatioForCompaction or MaxLeafEntriesBeforeForcedCompaction) is non-default, on reminder and operator passes as well as policy-triggered ones - trigger=reminder\|ratio\|size\|operator.
orleans.lattice.compaction.shard.retries	Counter<long>	{retry}	Compaction-coordinator transient retries during a tombstone-compaction pass. Tagged tree. A non-zero rate that does not lead to skips is healthy resilience; a sustained rate is worth investigating.
orleans.lattice.compaction.shard.skipped	Counter<long>	{shard}	Shards the coordinator gave up on after exhausting retries. Any non-zero rate is alert-worthy - the affected shard's tombstones will not be reaped until the next pass. Tagged tree.
orleans.lattice.compaction.shard.dirty_leaves	Histogram<int>	{leaf}	Per-shard dirty-leaf snapshot size at the moment the compaction coordinator enters a shard. Tagged tree. A flat-zero series indicates the dirty-leaves fast path saw nothing to do and the coordinator fell back to the legacy chain walk for that shard.
orleans.lattice.leaf.tombstone.ratio	Histogram<double>	1	A leaf's tombstone-to-total-entry ratio, sampled at the entry of every tombstone-compaction pass over that leaf (including passes that reap nothing). Tagged tree and tenant only: the leaf's identity is deliberately not a tag, so every leaf of a tree records into one series and the family holds at most one series per tree, independent of how many leaves the store holds (issue #2518 - a per-leaf tag previously gave it one series per leaf grain). The distribution across a tree's leaves is what the quantiles report; the p95 is the headroom signal for the MinTombstoneRatioForCompaction threshold.
orleans.lattice.leaf.splits	Counter<long>	{split}	Leaf-node splits triggered by a leaf exceeding its key-count bound (MaxLeafKeys) or its byte bound (MaxLeafBytes). Tagged tree.
orleans.lattice.leaf.split.completion.in_flight	ObservableGauge<long>	{completion}	Leaf-split completions currently suspended inside CompleteSplitAsync, per tree (issue #2967). The division outcome counters (orleans.lattice.leaf.split_attempts: divided, faulted, and the declining arms) are a partition of terminated divisions, so a division still executing the completion body at scrape time is a member of none of them - and because the catch that records faulted is unconditional, a division carrying neither divided nor faulted did not throw but is genuinely in flight. That un-terminated state had no instrument before this gauge, so a division wedged forever inside the completion read as healthy concurrency and nothing separated busy from stuck. Covers both the forward path (SplitAsync) and the recovery path (CompleteRecoverySplitUnderGateAsync), because both reach the completion through the same seam; it is orthogonal to the recovery outcome metering of issue #2860. Read it with orleans.lattice.leaf.split.completion.oldest_age: a non-zero count is only actionable once the oldest age is climbing. There is no per-tree pre-mint - a tree with nothing in flight emits no series, so an absent series is the healthy steady state rather than a build that lacks the instrument.
orleans.lattice.leaf.split.completion.oldest_age	ObservableGauge<double>	s	Age in seconds of the oldest leaf-split completion currently suspended inside CompleteSplitAsync, per tree, or no series for a tree with none in flight (issue #2967). This is the signal that separates a division that is merely momentarily in flight (age near zero, falling as scrapes advance) from one that is stuck (age climbing without bound), which the bare in-flight count cannot do. It is deliberately a measurement, not a threshold: the code encodes no age at which a completion is declared abandoned, because the honest answer to "how long is too long" cannot be chosen without observing real completion durations first, so the gauge supplies the observation and leaves the judgement to an operator or an alert reading real values. A registry-backed observable gauge rather than a completion-duration histogram, and the distinction is load-bearing: a histogram only records on completion, so a division that never completes - the exact wedge this exists to surface - would never appear in one. Its declared unit is s, so a host exporting through .AddPrometheusExporter() publishes it with a _seconds suffix, while the repository-context container's own exposition publishes the bare name; the bundled panel queries the _seconds family, matching orleans.lattice.backup.inventory.oldest_age (see How an instrument name becomes a PromQL series name).
orleans.lattice.leaf.bisect_refusals	Counter<long>	{refusal}	Leaf splits that could not place their pivot from the snapshot frame and fell back to the ordered key view. Tagged tree, reason (no_snapshot_attached, too_few_rows, frame_key_unreadable, no_key_sorts_below_pivot) and detach_seam - the cache surface that had already released the frame: none, clear, keys_accessor, enumerate_rows_accessor, frame_decode_fallback, underlying_rows_accessor or range_hydration_completed. What the fallback costs is decided by whether a frame is still attached at the refusal, and reason says which (issue #2856). reason=no_snapshot_attached is reported exactly when no frame is attached, so every row is already resident and the fallback materialises nothing: with detach_seam=none no frame was ever attached (a leaf replayed from the WAL); with any other seam a frame was attached and the named surface had already materialised or discarded it - earlier, typically on the read path - so the cost was paid there, not by the split, and the seam is where to look. too_few_rows is cheap (the attached frame holds under two rows). frame_key_unreadable and no_key_sorts_below_pivot are the expensive refusals: a frame is still attached, so the fallback itself materialises the whole remainder of the leaf and detaches the frame, resident and unsheddable for the life of the activation, on precisely the oversized leaf that can least afford it. A sustained non-zero rate carrying a seam other than none is operator-actionable and names the read-path seam that forfeited the frame; any non-zero rate of the two expensive reasons (frame_key_unreadable and no_key_sorts_below_pivot) is a division paying for the whole leaf. Two detach_seam values are never produced by a deployed process, so their absence is expected and carries no information: underlying_rows_accessor, whose only producer is reached solely through a deliberately retained test-only accessor, and range_hydration_completed, which no surface has assigned since issue #2843 and which survives only so the enum ordinals do not shift. Scoped to those two by name (issue #2865); this is not a claim that every other value is reachable on every deployment.
orleans.lattice.leaf.byte.overflow	Counter<long>	{leaf}	Leaves found to exceed MaxLeafBytes, the byte bound that complements the key-count bound MaxLeafKeys. Tagged tree and outcome: split (the leaf was divided to bring it back under the bound) or irreducible (the leaf holds a single entry larger than the bound, so dividing it cannot terminate and it is left intact). A sustained irreducible rate is alert-worthy: a leaf whose snapshot cannot be captured pins the whole tree WAL trim floor at zero, because the floor is a minimum over leaves. See Write-Ahead Log. Pre-minted at zero for both outcomes whenever a leaf reaches the byte-bound check on the snapshot-capture path, carrying the exact tag set a real emission carries, so an absent series means the capture seam was never reached rather than that no leaf overflowed (issue #2756). That ambiguity is not hypothetical: the absence of this series on a live deployment was read as evidence the capture-seam pre-split had not landed, when the same absence was equally consistent with the pre-split having landed and found nothing to divide. A counter that is only ever incremented cannot separate "not happening" from "not deployed" from "nothing to do", which makes it unfit as a stop criterion however good its evidence once it fires. Because the series is minted, a flat zero here is a measured zero, and the bundled panel omits the or vector(0) fallback its neighbours use.
orleans.lattice.leaf.split_attempts	Counter<long>	{attempt}	Leaf divisions sought on an over-capacity leaf. Tagged tree and outcome: divided (the division ran to completion), gate_contended (another turn held the split gate, so this turn returned without evaluating the leaf - no division was attempted and none was refused), already_under_capacity (the in-gate re-check found the leaf back under threshold, because a concurrent turn had already divided it), faulted (the division began and threw before it could complete, additionally tagged failure_class: unaffordable when it could not be paid for in memory, timeout when it exceeded a deadline, other for anything else - the two named classes have opposite remedies, a smaller leaf or bigger hydration budget against examining the storage path under it), no_admissible_pivot (the leaf held no row strictly inside its own declared [LowKeyInclusive, HighKeyExclusive) range, so there was no key it was entitled to divide at and the division was declined rather than taken at an out-of-span pivot - issue #3117), and recovered (a division whose intent was already durable - stranded by an earlier fault, a deactivation, or a crash - was completed later, either by the recovery path before the next write was admitted or by an over-capacity check that resumed the half-finished division instead of starting a fresh one; issue #2860). Exists to make orleans.lattice.leaf.bisect_refusals interpretable: a zero refusal count carries two opposite meanings - no division ever forfeited its fast path, or no division was ever attempted - and only the attempt count separates them. That distinction is the whole question on an oversized leaf, because a leaf sitting far above MaxLeafBytes with zero refusals and zero attempts is not a leaf whose divisions are succeeding, it is a leaf nothing is trying to divide, which is a different defect with a different fix. Pre-minted at zero for all six outcomes - and, on faulted, for each failure_class arm separately, since the class tag is part of the series identity - whenever a leaf reaches the byte-bound check on the snapshot-capture path, through the same recording helper a real emission uses so the tag shape matches by construction, so a flat zero here is a measured zero and an absent series means the capture seam was never reached (issue #2756). The bundled panel therefore omits the or vector(0) fallback its neighbours use. Sustained gate_contended with no divided is one pathological shape: divisions are being sought and none completes. Any non-zero faulted is the other, but it must be read against orleans.lattice.leaf.splits rather than on its own. That counter increments only after the split intent is persisted, so splits - divided is the number of divisions that stranded a leaf holding a half-finished division for the recovery path to re-enter on the next write, whereas a faulted with no matching splits increment threw before the intent was persisted and stranded nothing. The subtraction is the reading, not the fault count. A recovery that completes lands on recovered - never on divided, and never on a second orleans.lattice.leaf.splits increment, since that counter already counted the division when its intent was persisted and a grain reactivates for idle collection with the process alive - so splits - divided - recovered is the divisions still stranded over the same window. A recovery can complete a division initiated before a restart, so on a freshly restarted process that difference can read negative. Before the recovered arm existed a recovered completion recorded nothing here (issue #2860), so in the restart regime, where recoveries are the only split activity, the instrument read as though no division had been sought. The distinction is not academic: the hydration admission gate declines in advance, so failure_class=unaffordable is both the arm least likely to have wedged anything and the arm an operator is most likely to meet first on an oversized leaf. Before the faulted arm existed such a division recorded nothing at all here (issue #2845), so the instrument read identically to a tree on which no division had ever been sought - the exact collapse it exists to prevent.
orleans.lattice.leaf.digest.publishes	Counter<long>	{publish}	Leaf-side projection-digest publish decisions, partitioned by path. Tagged tree and path, where path is one of: coalesced_scheduled (a fresh one-shot timer was registered - first dirtying mutation inside a new coalescing window), coalesced_skipped (a dirtying mutation arrived while a coalesced publish was already pending, so the cross-grain hop was deferred onto the existing window - this is the publishes saved surface that justifies the coalescing default), coalesced_fired (the coalescing timer tick issued the cross-grain OnChildDigestPublishedAsync RPC - one per window per leaf unless an inline publish or graceful flush cancelled the timer first), inline (the leaf issued the cross-grain publish synchronously - either because DigestCoalescingWindowMs is zero, the timer registration failed in a test harness, or a structural caller routed through PublishDigestUpwardInlineAsync), or deactivation_flush (the leaf's graceful OnDeactivateAsync drained a pending coalesced publish before the activation tore down). The headline coalescing invariant - N writes inside one window produce one cross-grain hop - translates to coalesced_scheduled + coalesced_fired per window regardless of write count, with coalesced_skipped absorbing the remaining N - 1 dirtying mutations. A regression to the c2-xxix shape (resolver dropping DigestCoalescingWindowMs on the floor) shows up as zero coalesced_* increments and inline growing in lockstep with writes - the propagation-guard regression gate catches the property-level drop, this counter catches the behaviour-level drop.
orleans.lattice.leaf.activation_replays	Counter<long>	{replay}	Activation-time leaf-materialiser replays started, counted once per activation that begins a replay. Tagged tree and activation_temperature: cold when the activation had neither a snapshot rehydrate nor a populated entry cache and therefore replays the whole readable WAL window, warm when it resumed above one of those anchors and replays only the tail. Both arms are the same counter, so any single scrape yields a coherent cold:warm ratio - two independently scraped counters would not. An activation with no tree id bound takes no replay permit and is counted on neither arm, which excludes both arms identically and so cannot bias the ratio. The same two cumulative totals are also sampled into a rate-limited Information log line (once per tree per minute) for hosts that expose no metrics scrape (issue #2148).
orleans.lattice.leaf.activation_replays_over_budget	Counter<long>	{replay}	Activation-time leaf replays whose own post-range-filter applied-entry count exceeded MaxLeafReplayEntries while the WAL still covered the whole needed window. Since issue #2149 the count is the leaf's exact applied-entry total, taken during the replay; it was previously the classifier's partition-wide WAL gap, which spans every leaf pinned to that partition and so overstated a single leaf's work by up to the partition's leaf fan-out (measured at ~1,350 on one deployment). The counter is therefore now in the same units as the budget it is named after. These converge because the replay flushes its checkpoint incrementally, so every attempt makes durable progress even when torn down early, and a sustained rate is a capacity signal rather than a fault. That reading holds only while the persisted checkpoint is advancing: a sustained rate at a frozen checkpoint is a replay livelock (issue #1819) and IS a fault. Tagged tree and partition (the WAL partition ordinal, bounded by WalPartitions). The counter is deliberately not tagged by leaf, because the leaf count is unbounded - so it measures the rate only and cannot by itself separate many leaves each replaying once from one leaf replaying forever. Make that call from the accompanying warning log, which names the leaf and its persisted checkpoint; comparing checkpoints across warnings is only valid when they name the same leaf and partition (issue #2023). That log is a deliberately bounded sample and this counter remains the exact census: repeat warnings back off per leaf and are capped per tree, because one warning per occurrence made this line 46% of one deployment's container log (issue #2100). Withheld repeats are reported as a summary line while that tree keeps replaying; that summary is carried by a later occurrence, so a tree's final withheld tally goes unreported and this counter is the exact census for that case. A leaf partition reporting over budget for the first time is never withheld, so a newly appearing condition still surfaces promptly. A leaf that re-enters replay from an unchanged checkpoint is reported separately as a stalled-replay fault, so that shape no longer has to be recovered by hand from the cost warnings (issue #2149).
orleans.lattice.wal.replay.permit_adaptations	Counter<long>	{permit}	Adaptations to the per-silo WAL replay concurrency gate under memory pressure (issue #2781). Tagged outcome: withheld (a permit the replay held was not returned to the gate) or restored (a replay completed cleanly with the heap relieved, so one previously withheld permit was returned). The withheld arm is additionally tagged trigger (issue #2883), naming which of the two mechanisms withheld: occupancy for the proactive one and fault for the reactive one. Before that dimension existed both wrote the same untagged series, so withheld = N was a sum no scrape could attribute - and that ambiguity produced a wrong published conclusion when acceptance run 12's withheld = 6 was read as evidence that the reactive trigger fires, a reading equally consistent with it firing zero times. Summing over trigger recovers the historical untagged total, so comparisons against runs that predate the tag remain valid; that is why the fix was a dimension rather than a second counter. The restored arm deliberately carries no trigger, and a per-trigger level is therefore not derivable. Withheld permits are fungible - the accounting is one process-wide count, not a per-trigger ledger - so a restore cannot know which mechanism withheld the permit it hands back, and withheld{trigger="fault"} - restored is meaningless. Read the trigger split as which mechanism is doing the work, never as how much each is currently holding. Two triggers withhold (issue #2862): proactively, a replay that finishes while managed heap occupancy is at or above 75% of the GC hard limit withholds its permit whether or not it faulted; reactively, a replay that fails for memory pressure withholds its permit whatever the occupancy. A permit is restored only when a replay completes cleanly and occupancy has receded below 60%, so the two figures form a hysteresis band the gate settles inside rather than a single edge it chatters across. The occupancy denominator is the GC hard limit (GC.GetGCMemoryInfo().TotalAvailableMemoryBytes, floored by the cgroup memory limit), not the container grant: under a container memory limit .NET applies its default GCHeapHardLimitPercent to the grant, so a 12 GiB grant yields a 9 GiB managed ceiling and a threshold written against the grant would sit 33% above the limit that actually throws. Read the difference withheld - restored as the number of permits currently out of circulation, making the effective ceiling configured - (withheld - restored); read the rates to see how hard the gate is oscillating. The gate admits WalMaterialiserMaxConcurrentReplays, defaulting to the lesser of Environment.ProcessorCount and the enforced container CPU grant (issue #2816), cold WAL replays at once, and a tree whose entries are large makes every such replay hold large buffers - so on a constrained host the admitted set can exhaust the heap, at which point the replays fail slowly while holding permits, other leaves time out queued for a permit and never replay at all, no snapshot is banked, the durable materialiser pin never advances, and WAL garbage collection reclaims nothing, which makes the next replay window larger still. This counter measures the mechanism that breaks that loop. Deliberately not tagged by tree: the gate is process-wide, so a per-tree tag would imply a per-tree ceiling that does not exist and would invite a reader to sum arms sharing one resource. Backpressure never withholds the last permit, so the effective ceiling has a floor of one and the mechanism cannot latch. It is also incapable of raising the gate above the configured ceiling for any input, because it works by declining to return a permit already taken rather than by resizing - a dynamic floor beneath the operator's setting is not an override of it (issues #2278/#2279). Both arms are primed at zero when the gate is sized - and since issue #2883 that means all three primed series, each trigger value on the withheld arm as well as the restored arm, because priming the counter as a whole but not each trigger would make "this trigger never fired" and "this build predates the trigger split" identical on a scrape. A flat zero means "measured, never engaged" while an absent series means the build does not carry the mechanism.
orleans.lattice.wal.replay.permits_withheld	ObservableGauge<int>	{permit}	Replay permits currently withheld from the per-silo WAL replay concurrency gate by memory backpressure, so the effective ceiling is the configured ceiling minus this figure (issue #2784). Tagged with the platform tenant label and nothing else, matching its sibling counter: the gate is a process-wide static, so every tree on the silo draws from the same pool and a per-tree series would repeat one figure under many names and invite a reader to sum them. It exists because the two arms of orleans.lattice.wal.replay.permit_adaptations cannot show what a restart destroyed. The level is derivable from them as withheld - restored, but both are process-lifetime counters: when a process dies with adaptation in force they reset to zero together, so the derived difference silently becomes zero too and nothing in the series marks the discontinuity - "the gate had adapted and the restart threw that away" and "the gate never adapted" render identically. What this gauge adds is therefore the boundary, not a new number: the last sample before the gap is the level that was discarded, and the step to zero across the gap is the discard. That reading cannot be produced by the successor process, which has no access to its predecessor's static, so it has to be published continuously by the process holding the state rather than emitted once by the one that inherits none of it. It is not a substitute for the counters: a gauge is sampled, so a withhold and its matching restore landing between two scrapes are invisible here and visible there, and the counters remain the record of how often the mechanism fired. This series answers only "how much headroom is being declined right now". Published unconditionally once the gate is sized, so a flat zero is a measured zero - the mechanism is present and has never engaged - while an absent series means the build does not carry it.
orleans.lattice.wal.replay.permit_ceiling	ObservableGauge<int>	{permit}	Ceiling the per-silo WAL replay concurrency gate was sized to, or 0 before the first activation sizes it (issue #3047). This is the denominator the rest of the permit family was missing. One permit withheld from a ceiling of sixteen is noise; one withheld from a ceiling of two is half this silo's replay throughput; on a scrape those two readings are identical, and until this series existed nothing in the estate could tell them apart. The figure is not recoverable after the fact: a SemaphoreSlim does not expose its own maximum, and the gate is sized once and never re-created or topped up, so the only way to publish it is to capture it at sizing time and report it continuously. Tagged with the platform tenant label and nothing else, matching the rest of the family, because the gate is a process-wide static and a per-tree series would repeat one figure under many names. A 0 here is an unambiguous not yet sized sentinel rather than a degenerate reading: the sizing resolver returns the configured value when it is positive and the derived default otherwise, and both are at least one, so a sized gate cannot publish zero.
orleans.lattice.wal.replay.permits_available	ObservableGauge<int>	{permit}	Permits currently available on the per-silo WAL replay concurrency gate - the headroom a new activation arriving now would find (issue #3047). Never read alone. A zero here is ambiguous between a gate that has not been sized and a gate that is fully saturated, and only orleans.lattice.wal.replay.permit_ceiling separates the two; saturation is the reading this instrument exists for, which is why the pair landed together and is documented as a pair rather than as two independent series. It also does not measure the queue: the underlying count saturates at zero and reports the same figure whether one activation or a thousand are blocked behind it, so a saturated gate looks the same here however deep the backlog is. orleans.lattice.wal.replay.permits_queued is the instrument for that. Tagged with the platform tenant label only, as the gate is process-wide.
orleans.lattice.wal.replay.permits_queued	ObservableGauge<int>	{activation}	Activations that have entered the wait for a permit on the per-silo WAL replay concurrency gate and not yet left it (issue #3047). This counts arrivals in flight, not activations blocked (issue #3290): the count is incremented before the wait, so an activation that acquires without ever blocking is included for its acquire window, and the blocked set is a subset of it, exceeded by at most orleans.lattice.wal.replay.permit_ceiling. This is the only series in the estate that can observe a gate admitting nothing, and the reason it is worth a third instrument is structural rather than a matter of resolution. Every other permit series records at a terminal outcome - orleans.lattice.wal.replay.permit_queue_wait records on acquisition and on cancellation, and the activation-replay counters record once a permit is held - so all of them sit downstream of the wait. An activation that never acquires and is never cancelled therefore records nothing anywhere: not its wait, not its replay start. A permanently saturated gate consequently renders byte-identically to a gate no tree has ever asked for a permit from, which is precisely the ambiguity that left 47% of a deployed estate's stranded-leaf population uninterpretable. Read the three together: the ceiling says how wide the door is, availability says whether it is open, and this says how many have arrived at it and not yet got through. Tagged with the platform tenant label only, as the gate is process-wide.
orleans.lattice.wal.replay.permit_queue_wait	Histogram<double>	ms	Time an activation spent queued on the per-silo WAL replay concurrency gate, recorded once per wait as it ends (issue #2873). Tagged tree, tenant, and outcome: acquired (the wait ended in a permit) or canceled (the activation was cancelled while still queued). It exists to discriminate a reading that orleans.lattice.leaf.activation.failures{reason="canceled_awaiting_permit"} cannot make on its own: that counter is honest about where an activation was cancelled and silent about why. The Orleans request deadline spans the whole grain call, so an activation that burned its budget upstream arrives here already doomed and is cancelled within seconds, and the count is identical whether the gate was saturated or idle. Only the duration separates them, and it separates them by orders of magnitude: genuine permit starvation shows a canceled arm whose wait approaches the request budget, whereas upstream budget exhaustion shows a canceled arm at or near zero, because the activation was already out of time when it arrived. Read the two arms together - a large acquired wait alongside a large canceled wait is a saturated gate serving a queue, while a near-zero canceled wait says the gate is not the constraint no matter how high that counter climbs. A lifetime _sum / _count mean is NOT a present-state discriminator, and reading it as one has already produced a misdiagnosis on this epic. The sum and count are cumulative over the process lifetime, so a pool that saturated once and has long since drained still reports the mean of that episode indefinitely - a twelve-second mean read as a current wait against a pool that was empty at the time of reading. Use this histogram to characterise the shape of waits that have completed; to answer whether the gate is saturated now, read orleans.lattice.wal.replay.permits_queued, orleans.lattice.wal.replay.permit_waits_in_flight, and orleans.lattice.wal.replay.permit_wait.oldest_age, which are levels and carry no history. Never conclude health or absence from a single sample of any of them. Read it as _sum / _count, not as a quantile. The repository-context container's own exposition renders a Histogram<T> as a Prometheus summary carrying _sum and _count and no _bucket, so histogram_quantile over this series returns nothing; the usual or vector(0) repair would substitute a literal zero that reads exactly like a genuine near-zero wait, which is one of the two conclusions this instrument exists to distinguish, so it must not be used here. The mean is sufficient for that discrimination because the two regimes differ by orders of magnitude, not by percentile shape. Since issue #3044 this histogram declares explicit InstrumentAdvice bucket boundaries, and that does NOT retire the paragraph above. InstrumentAdvice is a .NET-level hint that only an exporter which reads it can honour, and the exposition this container runs honours none: a full scrape of it contains zero _bucket lines, types all 98 of its histogram families as summary, and carries no quantile= labels and none of the OpenTelemetry exporter fingerprints (target_info, otel_scope_name). The advice is therefore necessary and not sufficient - it makes buckets available to an exporter that honours them, such as the OpenTelemetry AddPrometheusExporter used by samples/McpTelemetry, and changes nothing under this adapter. Keep reading the mean until a scrape actually shows _bucket series for this family. Switching to histogram_quantile on the strength of boundaries existing in source returns nothing and invites exactly the or vector(0) repair the paragraph above forbids, so the boundary is part of the reading rather than a footnote to it. Tagged by tree, unlike its sibling orleans.lattice.wal.replay.permit_adaptations, and the difference is not an inconsistency: that counter measures a process-wide ceiling, which has no per-tree value, whereas a wait is one caller's own experience and is attributable to the tree that waited. Scoping a reading to a single tree is also what the runs 10/11 scoring defect could not do. Deliberately not primed at zero, unlike its sibling: priming a histogram means fabricating a 0 ms sample, which is not a neutral "measured, never engaged" marker but a datum reading as "the gate was instant" - it would bias the instrument toward one of the two conclusions it exists to distinguish. An absent series is therefore UNINTERPRETABLE rather than innocent and must be diagnosed, not read as a pass: corroborate against orleans.lattice.wal.replay.permit_adaptations (which is primed) and the activation counters to tell "no build" from "no waits".
orleans.lattice.wal.replay.permit_hold	Histogram<double>	ms	Time a replay held a permit on the per-silo WAL replay concurrency gate, from acquisition to the end of the replay, recorded once per hold for an activation replay and for a WAL GC starvation drive alike (issue #3921). Tagged tree and tenant. This is the measurement that separates the gate's two regimes, which have opposite remedies: when replays are CPU-bound, raising WalMaterialiserMaxConcurrentReplays finishes more replays per second and hold time stays flat; when they are bound on a grain store that serialises its writes, the service rate on orleans.lattice.wal.replay.permits_served stays flat and hold time grows, and the admission gate's no_progress arm fires harder. Declares explicit bucket boundaries from 1 ms to 30 minutes. Not zero-primed, for the reason given for orleans.lattice.wal.replay.permit_queue_wait: a fabricated 0 ms hold would bias the distribution toward "the store is fast".
orleans.lattice.wal.replay.permits_served	Counter<long>	{permit}	Permit holds that ended on the per-silo WAL replay concurrency gate, whether the permit was then returned or withheld by memory backpressure (issue #3921). Its rate is the gate's service rate in replays per second, and by Little's law the mean in-flight replay count is that rate multiplied by the mean of orleans.lattice.wal.replay.permit_hold. Tagged with the platform tenant label only, since the gate is process-wide. Zero-primed when the gate is sized, so a sized gate whose service count has stopped moving while activations are queued reads as a measured flat line, which is the regime the no_progress refusal arm reports.
orleans.lattice.wal.replay.permit_waits_in_flight	ObservableGauge<long>	{activation}	Activations currently queued on the per-silo WAL replay concurrency gate (issue #3044). Tagged tree and the tenant derived from it. It exists because orleans.lattice.wal.replay.permit_queue_wait is structurally blind to a wait that never ends: both of that histogram's arms are terminal - acquired is recorded after the semaphore is entered, canceled from the catch around the wait - so an activation parked on a saturated gate and never admitted records on neither, and the instrument falls silent in exactly the condition it was built to measure. No amount of zero-priming reaches that gap, which is the part worth keeping: a primed canceled arm reading zero asserts only that no cancellation completed, and the missing observable here is a level, not a terminal event. That generalises - priming is free for an instrument whose empty state is a level, and wrong for one whose empty state is a shape, which is why this gauge is primed and its sibling histogram deliberately is not. Primed at zero per tree once that tree has queued at least once in this process, so a flat zero here is a measured zero rather than an absent build. That priming has a boundary which is part of the reading and is not severable from it: it covers trees this process has observed queueing, not every tree in the estate, so a zero means "has queued before, is not queued now" and never "cannot queue" - and a tree that has never queued has no series at all, an absence which is correct and carries no information about that tree. Tagged by tree, like permit_queue_wait and unlike permit_adaptations, for the same reason: a ceiling is process-wide and has no per-tree value, whereas a wait is one caller's own experience and is attributable to the tree that waited. Useless on its own - a count of three cannot separate three healthy 200 ms waits from one activation parked for twelve minutes - so read it with orleans.lattice.wal.replay.permit_wait.oldest_age, which supplies the half it cannot.
orleans.lattice.wal.replay.permit_wait.oldest_age	ObservableGauge<double>	s	Age of the oldest activation currently queued on the per-silo WAL replay concurrency gate (issue #3044). Tagged tree and the tenant derived from it. This is the half of the pair that separates busy from stuck: a steady permit_waits_in_flight whose oldest age oscillates near zero is a gate serving its queue promptly, while the same count with this age climbing without bound is an activation that will never be admitted. That second shape closes the loop described under permit_adaptations - a leaf that cannot obtain a permit cannot replay, cannot bank a snapshot, cannot advance the durable materialiser pin, so WAL garbage collection finds the pin unusable and reclaims nothing, which makes the next replay window larger still. No threshold is encoded here deliberately; "stuck" is read off the slope against real data, because an honest threshold differs per host and per WAL size, and a fixed one would be a number that looked authoritative while meaning nothing. Deliberately NOT primed at zero, unlike its sibling count, and the asymmetry is the design rather than an oversight. The age of the oldest waiter when there is no waiter is undefined, not zero: a fabricated zero would publish the healthiest possible value for the emptiest possible state, making an idle gate indistinguishable from one whose waits are all instant - which is precisely one of the two conclusions this pair exists to let a reader choose between. An absent series here is therefore correct and means nothing is currently queued; that is safe to read only because the primed count beside it does separate a measured zero from a build that does not carry the mechanism, so the pair is interpretable where neither member is alone.
orleans.lattice.wal.replay.slice_narrowings	Counter<long>	{narrowing}	Activation-time replay slice-width narrowings forced by memory pressure on a commit-log read (issue #2867). Tagged tree, partition, tenant. This is the second of the two factors that set peak replay memory. Peak memory is the product of how many replays run at once and how much each one buffers. The first factor is WalMaterialiserMaxConcurrentReplays - an option, surfaced on the container tuning overlay as REPOCONTEXT_MAX_CONCURRENT_REPLAYS, and already measured from both sides by orleans.lattice.wal.replay.permit_adaptations and orleans.lattice.wal.replay.permit_queue_wait. The second is the per-replay slice width: WalReplaySliceBudget (issue #2898) sets the width a replay starts from, and the only thing that moves it below that at run time is the reactive narrowing this counter records - a read refused for memory pressure is retried at a quarter of the current width. Without this counter that run-time half of the factor is invisible, because the narrowing is otherwise reported by a warning log alone and a log is not a series. What it discriminates: a managed OutOfMemoryException during a mass reactivation is consistent with two stories whose remedies point in opposite directions - either the narrowing engaged and was not enough, so the width is genuinely too coarse for the host, or it never engaged at all because the allocation that failed was not the slice read the clause guards, in which case lowering the width would change nothing. Read it against orleans.lattice.leaf.activation.failures on the same tree: failures climbing while this stays at zero is the second story. Tagged by tree deliberately, unlike the process-wide permit_adaptations: a width is a per-replay property and a replay belongs to the tree whose leaf is activating, so this genuinely has a per-tree value - and that tag is what lets one tree be shown buffering harder than its siblings under identical cycling. Zero-primed per (tree, partition) when a partition replay begins, because the healthy steady state is never to narrow, so an unprimed counter would leave the common case indistinguishable from a build with no narrowing at all.
orleans.lattice.wal.replay.starvation_drive_abandonments	Counter<long>	(none)	WAL GC starvation drives abandoned after exhausting their budget, recorded on the grain side as the drive gives up (issue #3065). Tagged tree and the universal derived tenant dimension, which the leaf grain derives from the same tree id the scheduler does, so on a tree that has not been resized, restored or remediated the zero the scheduler primes and the one the grain records share a single series identity; on such an aliased tree the scheduler primes under the physical copy's id while the grain records under the logical id (see The tree dimension across aliasing). This is the sole discriminator between a bounded-but-slow drive and a drive that would have parked forever, and that is the whole reason it exists. Bounding the drive necessarily destroys the diagnostic that found the defect: before issue #3065 a parked drive held one of two per-silo replay permits indefinitely, and the scheduler's own arms rendered that as attempted climbing, undelivered climbing just behind it (the touch times out at the Orleans response-timeout default long before the drive's own budget elapses), and drove_already_driving collecting the successors that bounced off the in-flight latch. The measured signature on two independently frozen containers was attempted 81 = undelivered 78 + completed 3, with all three completions returning AlreadyDriving. A drive that is merely slow now produces that same shape, byte for byte, because the touch still times out at the same deadline and successors still bounce off the same latch - so none of the scheduler's arms can separate the two any more and only this counter can. An operator reading a frozen-looking reactivation panel must read this series before concluding anything: advancing means drives are hitting the budget, and the paired warning log says why: it reports how the budget divided between admission to the replay gate and the replay itself, and how far the replay moved the leaf's checkpoint, so a slow host-supplied commit-log read or an oversized replay gap is visible as such (issue #3479), whereas flat-at-zero means the drives are completing and the shape is ordinary sweep contention. Primed at zero beside attempted on the scheduler, not at the grain's drive entry point. Priming at the drive would make an absent series mean "no drive has ever run" as well as "this build is not deployed", which is exactly the ambiguity the priming exists to remove; co-primed with attempted the reading is unambiguous, because attempted greater than zero with this series present is a real measured zero, while attempted greater than zero with this series absent proves the build carrying the fix never landed. That makes the prime a free deployment proof as well as a zero-disambiguator. One arm covers both abandonment shapes - a drive that timed out queued for a permit, and one that timed out deep inside the replay - because they are the same verdict on the sweep; they call for different operator responses, so they are separated on the paired warning log, which splits the drive's time into admission and replay (issue #3479), and the first also records on orleans.lattice.wal.replay.permit_queue_wait's canceled arm while the second does not. The drive's elapsed time is carried on the paired warning log rather than as a histogram, because an abandoned drive's duration is approximately the budget by construction and would carry almost no information as a series.
orleans.lattice.leaf.activation_stalled_replays	Counter<long>	{replay}	Activation-time leaf replays that re-entered from a persisted checkpoint that had not advanced since the same leaf partition's previous replay on this silo, meaning the previous activation banked no durable forward progress at all (issue #2285). This is the fault arm and is a different condition from orleans.lattice.leaf.activation_replays_over_budget, which is a capacity signal: an over-budget replay converges, just slowly, whereas a non-advancing one has not converged at all. Before this counter the fault was reported only by a warning log throttled to one line per (tree, leaf, partition) per minute, so the rate an operator could observe was the throttle's and not the condition's; the counter is incremented outside that throttle and is the exact census, with the log a bounded sample of it. Read it as a rate over time rather than as a level: the condition is transient whenever an activation is torn down mid-replay, so a burst of cancellations or timeouts produces a cluster and then stops, and it is persistent only when the same leaf partition keeps reporting across many minutes. The two are indistinguishable in a single sample and were conflated in issue #2285, where a 70-second burst of 47 occurrences was read as a permanent convergence defect. Alert on the condition continuing, not on its appearance. Tagged tree and partition; deliberately not tagged by leaf, because leaf count is unbounded - the accompanying warning names the leaf.
orleans.lattice.leaf.activation_cursor_publish_failures	Counter<long>	{failure}	Activation-time eager cursor-publish failures, emitted when the post-replay cursor publish fails during a leaf activation. Tagged tree.
orleans.lattice.leaf.deactivation.checkpoint_delta	Histogram<long>	{offset}	Projection-checkpoint offsets a leaf activation banked durably while gracefully deactivating: the sum across WAL partitions of the persisted checkpoint on leaving OnDeactivateAsync minus the same sum on entering it (issue #2280). Tagged tree, deactivation_reason and activation_temperature; deliberately not tagged by leaf, because the live-leaf population is unbounded - per-leaf detail goes to the accompanying Debug line, which also names the entries applied through the projection seam. It measures the persisted mark on purpose: the in-memory accessor returns max(persisted, pending), so an offset advanced but not yet flushed is already present on entry and differencing it would report zero for every deactivation at every rate of occurrence. This is a LOWER BOUND on deactivations and must not be quoted as a census. Crash, process kill and silo failure bypass the hook by design, and an activation that THREW never reaches it at all - Orleans does not run OnDeactivateAsync when OnActivateAsync throws, measured with a positive control on Orleans 10.2.2 - so that population is carried by orleans.lattice.leaf.activation.failures instead and the two must be read together. A zero on the cold arm is expected and is not by itself a fault: a cold replay restarts from the -1 sentinel and the checkpoint is strictly monotonic, so nothing can be banked until the scanned-through offset passes the mark the activation started above. Cold progress below that mark is not discarded, it is unrepresentable, which makes an unchanged cold checkpoint arithmetically forced rather than symptomatic. The offsets are scanned-through, not applied-through (issue #2270): replay advances the checkpoint over entries it skips as another leaf's work, deliberately, because the WAL retention floor is the minimum of these offsets. A non-zero delta is durable forward progress through the log - the quantity issue #2280 is about - and is not a count of mutations this leaf applied; the accompanying log line carries that separately as EntriesApplied, counted as each entry is applied to the leaf's projection.
orleans.lattice.leaf.deactivation.barrier.failures	Counter<long>	{failure}	Graceful-deactivation durability barriers that faulted, tagged tree and reason: digest_publish, checkpoint_flush, snapshot_capture, frontier_pin (issue #3366). Since issue #3393 the snapshot_capture and frontier_pin arms also count a barrier that skipped because the deactivation deadline had already torn the activation down: a barrier that could not do its work is counted under its own reason whether it threw or declined, and only the paired warning log tells the two apart. Each barrier is contained independently, so a fault in one no longer cancels the barriers after it - previously the four ran as one unguarded sequence and the first fault silently skipped the rest, which is why a digest publish failing could cost a checkpoint flush. Since issue #3393 they run as checkpoint_flush, snapshot_capture, frontier_pin, then digest_publish last, because the digest publish is the slow, staleness-tolerant step and running it first spent the deadline the durable barriers needed; when the final checkpoint flush deferred its own inline digest publish, that publish takes the last slot instead and a failure of it is counted as inline_digest_publish on orleans.lattice.leaf.checkpoint.flush.tail.failures, not here. The arms are not equally serious and must not be alerted on as one series. digest_publish is benign in isolation: the digest is staleness-tolerant and the next mutation republishes it. checkpoint_flush above zero is a durability fault - the activation's checkpoint progress was not banked, so the next activation replays from an older offset and any progress the WAL no longer carries is gone. snapshot_capture and frontier_pin sit between the two: neither loses data on its own, but both leave the WAL retention floor held below where it should be. Read it against orleans.lattice.leaf.deactivation.checkpoint_delta, which shares the same population and the same blind spot. A zero is NOT evidence of health, and on some trees it is structurally unreachable. This counter is only written from inside the deactivation hook, so a tree whose leaves never deactivate - a hot tree read continuously, with no residency sheds - emits no series at all while being exactly the population most at risk, because nothing else captures its snapshots either. Establish that a tree runs the deactivation path (a live orleans.lattice.leaf.deactivation.checkpoint_delta series) before reading this counter's silence as anything. Crash, process kill and silo failure bypass the hook by design, so like checkpoint_delta this is a lower bound.
orleans.lattice.leaf.deactivation.barrier.duration	Histogram<double>	ms	Wall-clock duration of each graceful-deactivation barrier, tagged tree and reason with the same four barrier values as orleans.lattice.leaf.deactivation.barrier.failures (issue #3628). The failure counter says which barrier did not complete; this says where a drain spends its time, so a rise in per-activation drain cost can be attributed to one barrier rather than guessed at. One sample per barrier that ran, whatever its outcome: a barrier that faulted or skipped after consuming the deadline spent that time all the same. digest_publish also covers the teardown persist's deferred inline digest publish, which is the same upward publish. Read it as an interval mean per reason, delta _sum over delta _count. The per-barrier means summed approximate one activation's serial teardown cost; Orleans deactivates concurrently, so a drain's wall-clock is less than that sum multiplied by the resident count. Crash, process kill and silo failure bypass the hook, so like its siblings this covers graceful deactivations only.
orleans.lattice.leaf.deactivation.barrier.elided	Counter<long>	{barrier}	Graceful-deactivation barriers skipped because the pin store had already acknowledged, in the same deactivation, a pin dominating everything the barrier would publish, tagged tree and reason (today only frontier_pin) (issue #3643). The teardown persist tail publishes the frontier pin and the frontier_pin barrier then resolves the same batch again; when every partition of that batch is dominated on both axes (frontier HLC and checkpoint offset) by what the pin store acknowledged for the tail, the redundant pin-store round trip is skipped. Acknowledged means every shard write for the batch landed: a faulted or shed shard write does not count, and a deactivation whose tail did not run always publishes. The frontier_pin series of orleans.lattice.leaf.deactivation.barrier.duration is still recorded for an elided barrier, so the ratio of this rate to that histogram's count rate is the fraction of drains whose pin publish was redundant. Not zero-primed, matching its sibling barrier instruments: absent means no barrier has elided yet, not that the instrument is missing.
orleans.lattice.leaf.checkpoint.flush.tail.failures	Counter<long>	{failure}	Post-flush notification steps that threw after the checkpoint flush itself succeeded, tagged tree and reason: cursor_report, inline_digest_publish, snapshot_recheck (issue #3393). Read it beside orleans.lattice.leaf.deactivation.barrier.failures, not instead of it, because the two measure different halves of one named thing: that counter covers the flush body, the durable write, and stands down once the write lands, while the tail runs its post-write notifications after it, each contained separately. That containment is deliberate and correct - under issue #2220 a failed notification tearing down the activation produced a replay loop - and this counter does not change it; what it changes is that the fault is countable. The consequence for reading the barrier counter is the whole point: a checkpoint_flush count of zero there does NOT establish that the checkpoint path is healthy. On the drain captured under issue #3393 the barrier read zero for checkpoint_flush while 1,699 dirty digests died inside this tail, so barrier counts are a floor on the damage rather than a total. The arms are not interchangeable and must not be alerted on as one series. inline_digest_publish is the arm seen in production and means a leaf's parent was never told its digest changed, so the upward digest chain is stale until the next mutation re-drives it - on a quiescing tree, possibly never. snapshot_recheck means the post-flush re-evaluation of snapshot need did not run (on the ordinary tail the arm also covers the coverage-lag timer arming that precedes it), so a block pin can retain the tree's shared WAL, making it a candidate contributor to unbounded WAL growth (issue #3094). cursor_report carries a specific warning: each per-partition reporter call is also wrapped in its own try/catch inside the cursor report, so an ordinary report failure is logged there and never reaches this counter; only what surrounds those calls can - the grain-state and option reads, consumer-id construction, and the durable materialiser pin publish that follows the report. Since issue #3599 the same arm also counts a failed pin republish after a snapshot-recheck capture the store kept and, on a graceful deactivation's final flush, a failed awaited pin publish before the recheck or after a kept capture. A zero on cursor_report is therefore not evidence that cursor reporting is healthy, because the per-partition reports are double-contained and their ordinary failures are invisible here by construction. Not zero-primed, matching its barrier sibling: a series appears on the first fault, so an absent line does not establish that the build lacks the instrument. Confirm deployment from the image digest, never from the presence of this series.
orleans.lattice.leaf.activation.failures	Counter<long>	{activation}	Leaf activations that ended by throwing out of OnActivateAsync rather than coming online (issue #2280). Tagged tree, activation_temperature and reason: canceled when a replay already in progress was cancelled, canceled_awaiting_permit when the cancellation arrived while the activation was still queued for the replay concurrency permit and no replay had begun, canceled_resolving_options when it arrived while the activation was still resolving its tree options and had not yet requested a permit, canceled_rehydrating_snapshot when it arrived during the snapshot rehydrate, refused_replay_admission when the activation was turned away before it ever joined the permit queue because the admitted-waiter bound of issue #3284 was already reached, and faulted for any other failure. refused_replay_admission and canceled_awaiting_permit are a pair and must be read as one. The second means an activation was admitted, waited, and lost; the first means it was never admitted, because the queue was already deeper than it could outlast. Before admission control existed the second population did not, so the whole backlog surfaced under canceled_awaiting_permit and a queue nobody was bounding was indistinguishable from a gate that was merely busy. A rise in refused_replay_admission is the bound working, not a fault: it is the cheap failure, since a refusal is immediate and the caller retries, whereas an admitted-but-doomed waiter holds an activation for its whole request deadline and then enqueues its replacement. The cancellation arms are kept apart deliberately, and issue #2770 added the last two after the original pair proved too coarse to act on: canceled_awaiting_permit was selected by replayPermit is null, which is true for a cancellation in the tree-options resolve and in the permit queue alike, so one arm was unreachable and a cold-start stall on the shared registry singleton was reported for six deployments as replay-gate saturation (issue #2768). Read the arms as distinct causes with distinct remedies: a rise in canceled_resolving_options means activations are serialised behind a shared dependency and the replay gate may be entirely idle; a rise in canceled_awaiting_permit means the gate itself is saturated (issues #2279, #2256); a rise in canceled means a single replay is too long to finish inside the deadline (issue #2411). Do not infer gate saturation from a cancellation that never reached the gate. This counter exists because orleans.lattice.leaf.deactivation.checkpoint_delta is structurally blind to this population: a failed activation never runs the deactivation hook, so without it a cancelled cold replay would read as zero at every rate of occurrence including the highest, an absence indistinguishable from health. The same blindness hid how far a failed replay got, so since issue #2411 every counted failure is also paired with an unthrottled Information line naming the WAL entries the activation had applied through the projection seam, with its temperature, failure arm and admission phase; read those lines as a distribution filtered to cold activations that entered replay, because one cancelled while queued for a permit or resolving tree options reports zero by construction. Still a lower bound, though a tighter one than it was: the permit acquisition and, since issue #2770, the snapshot rehydrate both sit inside the guarded region, so the queue wait and the rehydrate are now covered where a cancellation during the rehydrate previously escaped uncounted entirely - it did not even appear in the split as a zero. A failure raised before that region is entered, during the coherence reset, is still not counted, and an outright process kill reaches no observation site anywhere.
orleans.lattice.leaf.activation.cold_replay_loop	Counter<long>	{activation}	Cold leaf-activation cancellations at or past the escalation threshold (3) of a consecutive cold-cancellation streak on this silo, meaning the same leaf's cold activation has been cancelled at least three times in a row with no successful activation in between (issue #2280). The streak counts a cold cancellation wherever it landed - mid-replay, queued for a replay permit, or still resolving tree options or rehydrating its snapshot - and the paired warning splits the run by those phases; every cancellation at or past the threshold counts, so a streak that runs to five contributes three increments. Tagged tree only - the leaf's identity is on the paired warning, never a tag. It exists because every other signal for this condition is an aggregate over leaves: orleans.lattice.leaf.activation.failures and the cold-replay log line both sum across the leaf population, and a sum cannot separate one leaf cancelled five times, which is a self-reinforcing loop and a defect, from five leaves cancelled once each, which is a cost. Because the streak is tracked per leaf, this counter can. The streak is consecutive and resets on the leaf's first successful activation, warm or cold, which is load-bearing and not an implementation detail: a cumulative count against a fixed threshold is a function of process age rather than leaf health, so at the measured rate it would eventually fire on a healthy silo no matter how high the threshold was set. Calibrated from issue #2278's corrected field measurement - 79 runtime cancellations over roughly 40 minutes, all bplusleaf, across 67 distinct leaves, of which 55 were cancelled once, 12 twice, and none more than twice - which reads as leaves occasionally repeating rather than leaves trapped. A threshold of 2 would fire on 12 of those 67 leaves in a normal window and would be pure noise, and a warning that always fires gets muted, taking the real signal with it; 3 is one above the observed maximum and is also robust to the log never having recorded whether the 12 twice-cancelled leaves succeeded in between, since neither reading of that ambiguity reaches 3. Do not recalibrate from the figures originally published on issue #2278 (27 cancelled more than once, one four times): they were impossible on their own arithmetic, because 79 occurrences over 67 distinct leaves leaves a surplus of only 12. A paired structured Warning line names the pathology and the leaf and is throttled to one per leaf per minute; this counter is incremented outside that throttle and is the exact census, with the log a bounded sample of it. A cancellation that lands mid-replay banks the prefix it had already re-read as snapshot coverage, so successive attempts are expected to advance; the warning therefore names the three shapes that still reach the threshold - a replay cut before its first slice boundary, a banked claim refused because it would regress a partition's existing coverage, or too little progress per attempt to converge - and notes that a cancellation queued for a permit or still resolving options or rehydrating never entered replay and banks nothing. Observation only - nothing here bounds the readable replay window, checkpoints incrementally, or relaxes the snapshot-capture gate, which together are the bounding design tracked as issue #2411.
orleans.lattice.leaf.replay_barrier_outcomes	Counter<long>	{replay}	Terminal states of a leaf's deferred WAL replay (issue #2871), tagged tree and outcome: completed when the replay applied fully and the barrier is satisfied for the rest of the activation, faulted when it threw, and canceled when it was cut short by deactivation or by an operation that discards the state it was rebuilding (ClearGrainStateAsync, RebuildProjectionFromWalAsync). This series exists because #2871 moved the replay OFF the activation critical path, and that trade replaces a loud failure with a quiet one: before, a failed replay destroyed the activation and was counted on orleans.lattice.leaf.activation.failures; now the activation comes online and stays healthy-looking while its data operations fail, so a fault here does not appear on that counter and the two must be read together rather than one substituted for the other. All three outcome arms share one instrument and each is zero-primed per tree when a barrier is armed, so a flat zero on faulted is a measured zero and not an unpublished series - the distinction the priming exists to preserve, because no leaf on this tree ever armed a replay and replays were armed and none failed are different facts that an absent series cannot tell apart. canceled is expected in bounded quantity and is not on its own a defect; a sustained non-zero faulted is, and its symptom is data-operation failure on a live leaf. A replay that faults or is cancelled disarms itself, so the next data operation or WAL GC touch re-arms a fresh one: the count is of replay attempts, not of leaves, and one leaf may contribute several.
orleans.lattice.leaf.deferred_terminals_dropped_at_cap	Counter<long>	{terminal}	Durable ledger records refused, not terminals lost (issue #3190). Pass 1 retains each TxCommit, TxAbort or DeleteRange in deferredTerminals before attempting the capped UnresolvedReplayWork ledger write. A refusal falls back to the in-memory checkpoint clamp; pass 2 still applies the terminal and drains its ledger record if one was admitted. Interruption before that drain can require re-reading unbanked work. Also counts refusals when the ledger is disabled. The head-of-window liveness admission is unchanged. Tagged tree, partition and tenant, not leaf; a cumulative count or a sustained aggregate rate cannot establish that one leaf stays full, fails to drain, or holds the WAL floor. The #3190 vector-index observations are consistent with expected saturation in a range-delete-heavy replay: LatticeVectorIndexStore.DeletePrefixAsync normally issues DeleteRangeAsync, and non-last partitions defer those deletes even without sagas. Regression coverage at cap 1 and the shipping cap 1024 shows saturation followed by a fully drained ledger when pass 2 completes; the aggregate field counts do not establish a separate drain defect. Raising the cap may reduce re-reads after interruption but increases persisted row size (notably constrained on Azure Table); any finite cap merely moves the threshold for a long enough WAL window. No default increase is justified by these counts alone. The uncapped prepare-recording arm (leaf.unresolved_prepare_ledger_beyond_cap) is not a proxy for terminal-record refusals. Zero-primed on partition replay entry. The shipped name, unit, tags, emission sites and replay behaviour are unchanged.
orleans.lattice.leaf.span_fail_open_commits	Counter<long>	{key}	Keys a leaf committed locally although its declared [LowKeyInclusive, HighKeyExclusive) span excludes them, because declared-span admission found no neighbouring leaf to forward them to (issue #2125). Tagged tree, reason (no_sibling: the chain pointer on the key's side is null; self_reference: it names the leaf itself) and origin (client_write, merge, or cross_shard_migration), so a deliberate migration graft reads apart from an accidental fall-back. It matters because WAL replay admits by declared span: the leaf holding the row will not reinstate it on its own rebuild, so its survival across a cold restart depends on the checkpoint position of the leaf that does declare it. A leaf that declares no span at all (a single-leaf tree or a bulk-loaded leaf) owns every key and never advances it. Each fail-open also logs a warning naming the leaf and tree, rate-limited to one line per silo every 10 seconds. Not pre-minted, so an absent series is the healthy reading and any value is worth investigating. Observability only: the fail-open behaviour is unchanged. See Span Admission.
orleans.lattice.leaf.snapshot.captures	Counter<long>	{capture}	Leaf-snapshot capture attempts, counted once each at the point the leaf's snapshot-capture core commits to a capture (issue #2696). Tagged tree and outcome: succeeded once the blob has landed and durable coverage has advanced, abandoned when the caller's token was cancelled, failed for anything else. It exists because the capture path was wholly uninstrumented: the advisory capture wrapper catches every exception and only logs, and the sole capture-derived series, orleans.lattice.storage.snapshot_bytes, is fed only after a successful save. So a deployment in which every capture was failing and one in which no capture had ever been attempted exported byte-identical metrics, and those are opposite operational states - a broken storage provider against a correctly idle deployment. Read attempted as the sum across the outcome tag, which keeps that total a within-family comparison that cannot drift from its parts; a bare failure counter would not have fixed the defect, because zero failures is itself ambiguous between "all succeeded" and "none ran". The abandoned split is load-bearing rather than cosmetic: a fleet-wide shutdown abandons thousands of captures by design (issue #1965), and folded into failed that routine event would present as a mass provider outage. Declines are deliberately not counted - no tree id, nothing checkpointed and no live data, a capture already in flight, or no coverage it could claim (see orleans.lattice.leaf.snapshot.capture.declines) - so the total is attempts, not invocations.
orleans.lattice.leaf.snapshot.capture.declines	Counter<long>	{decline}	Leaf-snapshot capture invocations that declined before the attempt boundary, tagged tree and reason: no_tree_id, not_eligible, already_in_flight, no_coverage_claim (issues #2696, #2725). It is a separate instrument rather than a fourth value of the outcome tag on orleans.lattice.leaf.snapshot.captures, and the separation is load-bearing: that counter and orleans.lattice.leaf.snapshot.capture.duration are written at one boundary so they share a population exactly, and a decline is never timed, so folding declines into the attempt counter would make the two families silently differ. It exists because without it the capture instruments reproduce, one gate higher, the ambiguity they were added to remove - a deployment in which every capture is declined and one in which the capture path is never reached at all would both leave the attempt counter at zero. A declined capture is a third state, distinct from a failed capture and from an idle deployment, and it is the most likely way a self-heal silently never runs. The four reasons call for different responses: already_in_flight dominating is a contention signal, captures being requested faster than the shared snapshot storage provider retires them; not_eligible dominating is the starved-leaf signal, a leaf with nothing checkpointed and no live data that will never cover itself; no_tree_id above zero is a bug, capture invoked on a leaf never attached to a tree; no_coverage_claim is the unclaimable-rows signal (issue #2725), a leaf holding live rows that has never checkpointed, so any blob it wrote would claim no coverage and be refused by the load gate - benign, since WAL replay covers such a leaf completely, but it names the population whose retained WAL cannot shrink until it checkpoints. A no_tree_id decline carries no tree tag, because the leaf has no tree identity at that point, but it still carries the derived tenant label with the reserved _platform_ value - read it as a global count and do not expect it under a per-tree filter.
orleans.lattice.leaf.snapshot.driver.declines	Counter<long>	{decline}	Leaf-snapshot capture drivers that declined to drive a capture, tagged tree and reason (issue #3185). Deliberately a separate instrument from orleans.lattice.leaf.snapshot.capture.declines, and the separation is load-bearing in exactly the way that counter's own separation from orleans.lattice.leaf.snapshot.captures is: those declines are raised inside CaptureSnapshotCoreAsync, so declines and attempts together partition its invocations exactly and stay co-populated with orleans.lattice.leaf.snapshot.capture.duration, whereas the declines counted here happen in a caller that never reached that method. Folding them in would count a strict superset and silently corrupt the decline-versus-attempt ratio the other counter exists to report. It exists because the capture drivers were the last unlit segment of the snapshot-coverage path: the zero-coverage repair reports six outcomes and CaptureSnapshotCoreAsync reports four declines, but the graceful-deactivation hook and the periodic recheck returned in complete silence on every gate. Ten arms: deactivate_unproven_coverage_stale (the #1535 no-loss gate declined while some partition's checkpoint had outrun its durable coverage, so this leaf leaves behind a pin frozen below its own checkpoint), deactivate_unproven_coverage_current (the same gate declining harmlessly, coverage already current everywhere), deactivate_unproven_unclassified (the classification itself threw - expected to be zero outside a test harness, and kept separate so the two informative arms never absorb an unknown), recheck_cadence_not_reached, recheck_capture_in_flight and recheck_coverage_current for the three periodic-recheck gates, recheck_no_durable_checkpoint (issue #3300) and recheck_checkpoint_stalled (issue #3389) when the coverage-lag timer instead routes a leaf holding rows to the starvation drive, because no partition has ever checkpointed or because its checkpoint has stopped advancing, and recheck_drive_refused and recheck_drive_deferred (issue #3575), which qualify those two routing arms rather than adding to them. Each qualifier is recorded on the same tick as one of the routing arms - refused when the routed drive was refused admission to the per-silo WAL replay gate before replaying anything, deferred when it was skipped because the leaf is backing off after a refusal - so the reasons on this instrument do not partition ticks. The timer's drives never take the last free slot of the gate's GC share, which is kept for the WAL GC sweep, and while that share is a single slot they yield it to a refused sweep drive; a refused leaf backs off by a count of drive opportunities that doubles with consecutive refusals up to four to seven, jittered per leaf, and a drive that is admitted ends the backoff. Before issue #3575 the refusal escaped the timer callback as an unhandled exception, logged twice by the runtime and counted nowhere. Reading it: deactivate_unproven_coverage_stale is the minting rate of the frozen floor-holder population and is the arm that matters. The decline itself is correct - a leaf that neither advanced a checkpoint over cache-resident applies nor cold-rebuilt its cache from the WAL start cannot honestly stamp coverage - so what is being measured is the rate, not a fault. Read it against the WAL GC reactivation drive's completion rate (orleans.lattice.wal.gc.blocked_leaf_reactivations): sustained above it, the frozen population grows without bound and no drive-side tuning can converge, because the mismatch is asymptotic rather than a matter of constants; below it, the drive is merely slow and tuning applies. Until this instrument existed only the accumulated consequence of that minting was observable and never its rate, which is why a flow presented as a static population. Like orleans.lattice.leaf.snapshot.capture.declines it is not zero-primed, so a series appears on first decline and absence does not establish that a build lacks the instrument - the usual "presence of a series is the deployment fact" reading does not apply to this family. A decline on a leaf with no tree identity carries no tree tag, for the same reason no_tree_id does not.
orleans.lattice.leaf.snapshot.capture.duration	Histogram<double>	ms	Wall-clock duration of a leaf-snapshot capture attempt, recorded at the same boundary and with the same tree/outcome tags as orleans.lattice.leaf.snapshot.captures, so the two share a population exactly: every counted attempt is timed and every timed attempt is counted (issue #2696). Timing starts at the single-flight boundary rather than at method entry, which is a correctness requirement and not a detail: the decline gates return in microseconds and are taken far more often than a capture runs, so timing them would let near-zero no-ops dominate the sample count and report "captures are fast" precisely when none are happening. Capture is awaited inline inside OnActivateAsync, and Orleans delivers no request to a grain until activation completes, so this duration is paid by every caller of that leaf - it is activation latency, not background work. All per-leaf snapshot grains write through one shared storage provider with no cross-leaf admission control (issue #2696), so a mass capture event is the shape a latency cliff takes. The Prometheus exporter renders this as an explicit-bucket histogram, exactly as it does every other ms-unit histogram on this meter, so histogram_quantile over _bucket is available and is what the bundled CommitPath panel charts. A delta of _sum divided by the delta of _count gives an interval mean as a companion reading; do not read either cumulative total as a level.
orleans.lattice.leaf.snapshot.capture.concurrency_peak	ObservableGauge<int>	{capture}	Greatest number of leaf-snapshot captures observed executing concurrently on this silo since process start, as a monotone non-decreasing high-water mark (issue #2696). It measures the quantity every other capture instrument is blind to by construction: the single-flight guard those are written around is an instance field on one activation, so it stops a leaf capturing twice at once and says nothing about how many different leaves are capturing against the one shared snapshot storage provider. A thousand leaves each passing their own guard is the fan-out issue #2696 describes, and it registers on no per-leaf series. The shape is chosen because both obvious alternatives fail silently on this deployment. An instantaneous gauge of the current depth is sampled only when the endpoint is scraped, so the transient spike that makes the metric worth having is exactly what falls between two scrapes - the objection PR #2723 raised when it deferred this metric, and it is correct. A Histogram<T> recorded at each entry is event-sampled and so cannot miss the spike, but the repository-context container's own exposition renders every histogram as _sum/_count with no buckets and no quantiles, leaving only the mean, and a mean dilutes a spike to nothing: one capture at depth 64 among ten thousand at depth 0 reads as 0.0064. A monotone peak is immune to both, because a value that never falls is reported by the scrape that sees it and by every scrape after it, so scrape timing stops being a risk to manage and becomes irrelevant. The cost is that it is a high-water mark for the life of the process and does not decay; that is the intended reading, and a restart resets it. Registered at host build from AddLattice rather than on first capture, so a silo that has never captured still publishes the series and a scraped 0 means measured none rather than no detector. Carries the constant platform tenant sentinel and no tree tag: the peak is a maximum across every leaf on the silo, so it spans trees and tenants, and the contended resource - one storage provider per silo - is itself silo-wide. Splitting it per tree would report several smaller numbers, none of which is the depth the provider actually saw. Pair it with orleans.lattice.leaf.snapshot.capture.concurrent_entries for the per-tree and frequency halves of the question: the peak says how bad it got, the counter says how often it happens, and a peak alone cannot tell one transient spike at startup from sustained contention.
orleans.lattice.leaf.snapshot.capture.concurrent_entries	Counter<long>	{capture}	Leaf-snapshot captures that crossed the attempt boundary while at least one other capture was already in flight on the same silo, tagged tree (issue #2696). It is the frequency half of the cross-leaf concurrency reading whose magnitude half is orleans.lattice.leaf.snapshot.capture.concurrency_peak, and the pairing is load-bearing rather than decorative: a high-water mark alone cannot distinguish a single transient depth of 3 during startup from a silo sitting 3 deep continuously for an hour, and those are opposite operational states. Note it is not a duplicate of the already_in_flight reason on orleans.lattice.leaf.snapshot.capture.declines: that reason counts a capture the per-leaf guard turned away on a leaf already capturing, whereas this counts a capture that was admitted while a different leaf was capturing. The two populations are disjoint - one never reaches the attempt boundary, the other is past it - and only this one describes real concurrent load on the shared provider. Emitted on every attempt, adding 0 for an uncontended one, so each tree gets a zero-primed series from its first capture; the zero-add is deliberate and must not be optimised away, because without it a tree with no contention is indistinguishable from a tree whose captures never ran.
orleans.lattice.leaf.snapshot.load_failures	Counter<long>	{load}	Snapshot loads that failed, distinct from genuine snapshot absence (issues #2364, #2404). Tagged tree, reason, tenant. The three reasons are resource_exhausted (an observable OutOfMemoryException), contiguity_exhausted (that OOM while hydration was admitted as the sole occupant), and unclassified (no OOM observable in the exception chain). Both observed-memory arms decline activation instead of escalating to a more expensive WAL replay. For contiguity_exhausted, divide the leaf or lower MaxLeafBytes; increasing the memory grant does not repair a contiguous-allocation failure. unclassified retains the existing cold-replay fallback but does not rule out memory pressure: a snapshot-grain activation failure can hide the original OOM. Correlate storage/activation logs and memory headroom rather than inferring a storage-provider fault. Read alongside activation failures and orleans.lattice.leaf.snapshot.hydration_admissions; a load failure alone does not establish whether activation succeeded. unclassified replaces the historical faulted value; update reason-filtered alerts to include both during rollout. The public SnapshotLoadFailureFaulted field is retained for source compatibility and now exports unclassified. This is a vocabulary correction, not a claim of live production OOM attribution.
orleans.lattice.leaf.snapshot.hydration_admissions	Counter<long>	{hydration}	Activation-time leaf-snapshot hydrations that passed the per-silo admission gate (issue #2765). Tagged tree and outcome: immediate when the claim fitted the budget straight away, queued when it had to wait behind it, and sole_occupancy when the contiguity rule serialised it (issue #2844). These three strings are the literal tag values the gate emits, from LatticeMetrics.SnapshotHydrationAdmittedImmediately, SnapshotHydrationQueued and SnapshotHydrationSoleOccupancy; document them exactly as emitted and never derive them from the constant's field name, because a query or alert matches on the value, so a plausible near-miss spelled from the field name instead of the value matches no series at all while looking correct (issue #2854). The gate exists because nothing bounded how many leaves materialised their persisted state at once on a cold start, and the cost of one such materialisation is several multiples of the stored frame, by an amount that depends on which read path the payload takes (issue #2858). A blob written by a current build carries the Orleans binary LGB1 marker and costs about 2x the frame: the provider's column read plus the decoded rows. A legacy JSON blob written before issue #2516 still decodes as JSON, because reads route on the stored payload rather than on the type, and costs roughly 10-13x the frame: the base64 document is itself 4/3 of the frame, and reading it back allocates a UTF-16 System.String of the whole document, Newtonsoft's growing character buffer, and the decoded rows on top. The gate prices every claim at 5x the stored frame whichever path it will take, because it sizes the claim before the row is read. That factor covers the binary path with headroom and deliberately under-prices the legacy one, whose excess the budget absorbs: an all-legacy cold start, fully admitted, still peaks at about a third of the heap hard limit. Do not lower it to the binary figure while legacy blobs remain, and do not raise it to the legacy figure, which would halve admission for every current blob to protect a population that shrinks as leaves re-snapshot. The aggregate in the original incident crossed the .NET heap hard limit, which the runtime sizes from the container's cgroup, so the process threw a managed OutOfMemoryException, exited 0 and was never reported as OOM-killed - a restart loop that read for a long time as an unexplained clean exit. The budget is derived from that same heap hard limit and is deliberately not configurable, because a bound an operator has to set cannot rescue a process that is already restart-looping. Admission is first-in-first-out, and a claim larger than the entire budget is admitted as soon as it is the only claimant - without that rule an oversized leaf could never activate, so it could never be divided back under bound, and the gate would turn a crash loop into a permanent stall. Read a climbing queued rate during a cold start as the gate working: hydrations are being serialised instead of exhausting the heap together, and it should decay as leaf division shrinks the corpus. A queued rate that never decays across many restarts means division is not progressing; correlate with the leaf byte-overflow and capture instruments, not with container memory headroom. All three outcome arms are pre-minted at zero the first time a tree is seen in the process, so a flat zero is a measured zero and an absent series means the build does not carry the gate. The sole_occupancy arm reports a different predicate and is read differently. The budget above is denominated in total bytes, and the allocation that actually failed in issue #2844 is a single contiguous one, which must be met by an unbroken run on the large object heap and which no amount of aggregate headroom guarantees. How large that run is depends on which read path the payload takes, and the gate cannot know that before it sizes the claim: a blob written by the Orleans binary serializer returns as a byte[] of roughly the stored frame, while a legacy JSON blob written before issue #2516 returns as a contiguous UTF-16 System.String costing 8/3 of the frame (base64 inflates it by 4/3, then each character costs 2 bytes). Reads route on the stored payload rather than on the type, so both paths stay live and the gate assumes the worse of the two. A claim whose stored frame exceeds the default leaf size bound is therefore admitted only as sole occupant and excludes every other claim while it is held; that ratio applies equally to the claim and to the ceiling, so it cancels out of the comparison and governs only the contiguous figure the exception reports. That ceiling is deliberately absolute rather than derived from the memory grant, because every other bound here - this budget, the resident working-set budget, the replay gate - rises with the grant and admits more concurrent work, while contiguous feasibility does not improve with it at all. Read a sustained sole_occupancy rate as a signal to divide leaves or lower MaxLeafBytes, and specifically not to provision more memory, which would make it more frequent. All three outcome arms are pre-minted at zero.
orleans.lattice.leaf.snapshot.segment_reads	Counter<long>	{read}	Individual snapshot segment frames read during a segmented hydration (issue #2914). Tagged tree and outcome: loaded when the frame read back and validated, missing when it was absent or failed validation, failed when the read threw. It exists because a snapshot whose encoded frame exceeds LeafSnapshotSegmentBytes is no longer stored as one row: the row payload is split into row-aligned segments, each a standalone frame in its own grain-state row, and hydration folds them one at a time. The split is what makes the bound real. The contiguous allocation that failed in issue #2844 is materialised by the storage provider's column read, before any lattice code runs, so bounding it requires bounding the column, and the only way to bound a column is to stop putting the whole payload in one. missing and failed are deliberately separate arms because a torn durable snapshot and a transient storage fault call for different responses: missing means the manifest references a segment that is not there, which is a durability question, while failed is an I/O question. Either one fails the hydration closed - the cache is emptied and the rehydrate declines, so the leaf replays its whole readable WAL rather than activating with a snapshot that silently holds fewer rows than its coverage claims, which is the one shape that would let the coverage-gated WAL GC trim the last durable copy of a prefix. All three outcome arms are pre-minted at zero the first time a segmented hydration is observed for a tree in the process, so a flat zero is a measured zero (an absent series means no snapshot for that tree has yet exceeded the segment window, or the build lacks the instrument); without that priming an uninstrumented branch and a never-taken branch export byte-identical metadata.
orleans.lattice.leaf.snapshot.segmented_hydrations	Counter<long>	{hydration}	Activation-time leaf-snapshot hydrations that folded a segmented snapshot to completion (issue #2914). Tagged tree. Read it as the population for which the contiguity bound was actually exercised: an ordinary leaf whose snapshot fits inside LeafSnapshotSegmentBytes is stored and hydrated inline and is never counted here, so this counter names exactly the leaves that would previously have demanded a single contiguous array of the whole payload. It is counted only on a complete fold, so it is not a proxy for attempts; a hydration that declined partway is visible on the missing or failed arms of orleans.lattice.leaf.snapshot.segment_reads and not here, which keeps "how many segmented snapshots hydrated successfully" and "how many segment reads went wrong" from sharing a population they do not share. Read it alongside orleans.lattice.leaf.snapshot.segment_peak_bytes, which reports whether the bound held, and against orleans.lattice.leaf.snapshot.hydration_admissions, whose sole_occupancy arm should fall as segmentation takes effect, since a segmented snapshot no longer presents an oversized contiguous claim to the gate. Pre-minted at zero per tree.
orleans.lattice.leaf.snapshot.segment_peak_bytes	ObservableGauge<long>	By	High-water mark of the largest single contiguous segment frame this process has materialised while hydrating a segmented snapshot, per tree (issue #2914). This is the instrument that says whether the bound held: it must stay at or below the configured LeafSnapshotSegmentBytes window however large the underlying snapshot grows, so a value above the window is the direct signal that segmentation is not bounding what it claims to bound. It is a monotonic high-water gauge rather than a histogram, and that choice is load-bearing on the repository-context container's own exposition, which renders every histogram as a summary carrying only _sum and _count, with no buckets and no quantiles, so the only reading available from one there would be a mean - and a mean is exactly the wrong statistic for a peak, diluting the single large allocation that matters into an average of many small ones until it disappears. Because the value never decreases within a process lifetime, read it as "the worst this host has seen since it started", not as a current level; it resets on restart, which is the intended behaviour for a high-water mark. Pre-minted at zero per tree, but only when the tree's first segmented hydration is observed, so the series is absent until a snapshot for that tree has exceeded the segment window, and a flat zero means the segmented path was entered but no segment frame has yet read back.
orleans.lattice.leaf.residency.sheds	Counter<long>	{activation}	Leaf activations deactivated by the per-silo resident leaf working set so the hydrated population stays under its derived byte budget (issue #2767). Tagged tree and kind: banked when the shed leaf had a snapshot to reload from, unbanked when it did not. It exists because orleans.lattice.leaf.snapshot.hydration_admissions bounds only how many hydrations run concurrently, and the heap cost of a hydrated leaf is resident, not transient: LeafSnapshotHydrationSource retains the whole encoded frame for the activation's lifetime, so completed hydrations accumulate at roughly one frame each and a forced blocking compacting gen2 collection reclaims none of it while the activations live. Concurrency-bounding therefore controls the peak of each hydration and nothing at all about the steady state, which is why the heap hard limit was still reached with that gate provably binding. Shedding is a graceful deactivation, so the leaf captures a snapshot on the way out and an unbanked shed converts into a banked candidate for next time - the reason the class preference drains rather than recirculates. Read a steady banked rate as the bound working at its cheap end. A sustained unbanked rate is the one to act on: it means the working set has run out of cheap candidates and is paying whole-window replays to stay under budget, so the answer is snapshot coverage, not this bound. Both arms are pre-minted at zero the first time a tree registers, so a flat zero is a measured zero and an absent series means the build does not carry the bound.
orleans.lattice.leaf.residency.budget_bytes	ObservableGauge<long>	By	The byte budget the per-silo resident leaf working set is actually enforcing (issue #2788). Carries the platform tenant sentinel (tenant="_platform_") and no other tag: the working set is per-silo and spans every tree, so it has no tenant to derive and a tree tag would be a category error, and the silo dimension arrives from the exporter's resource attributes. The sentinel is emitted rather than omitted so that an unattributable measurement is distinguishable from a site that simply failed to attribute one. Read it against the container's memory grant. This gauge exists because orleans.lattice.leaf.residency.sheds reads zero in two situations with opposite remedies - the bound is populated and correctly under budget, or the bound cannot engage at all - and no amount of priming separates them, because priming makes a counter's silence readable while a counter observes an action and this question is about state. A budget of the same order as the whole container grant is the second case: the process reaches its heap limit and dies before the threshold can ever be crossed, so the bound is present, plausible-looking, and doing nothing. That was a real defect, not a hypothetical: the budget was originally derived from GCMemoryInfo.TotalAvailableMemoryBytes alone on the belief that it reports zero when no heap hard limit is configured, whereas it reports host physical memory, so a large host running a small container derived a budget larger than the grant. It is now the smaller of the heap hard limit and the cgroup memory limit, divided by HeapBudgetDivisor (4) and floored at 64 MiB. A flat line at exactly 1073741824 is the both-ceilings-unknown fallback and should be investigated as a detection failure rather than read as a tuned value.
orleans.lattice.leaf.residency.resident_bytes	ObservableGauge<long>	By	Bytes currently accounted to live, un-shed leaf registrations in the per-silo resident leaf working set (issue #2788). Carries the platform tenant sentinel, for the same reason as the budget gauge. Read as a ratio against orleans.lattice.leaf.residency.budget_bytes: that ratio is the headroom, and it is the only reading that separates an estate comfortably under bound from one about to begin shedding. Resident pinned near the budget with a steady shed rate is the bound working. Resident at a small fraction of the budget while the process is exhausting memory does not mean leaf residency is not the cost; it means the budget is too large to be a bound, so compare the budget line against the container grant. Resident flat at zero while leaves are demonstrably activating is a wiring defect rather than quiescence, and orleans.lattice.leaf.residency.registrations is what tells the two apart.
orleans.lattice.leaf.residency.registrations	ObservableGauge<long>	{registration}	Leaf registrations currently held by the per-silo resident leaf working set (issue #2788). Carries the platform tenant sentinel, for the same reason as the budget gauge. This is the instrument that settles whether a zero shed count is quiescence or a dead ledger. An empty ledger and a populated ledger correctly under budget are indistinguishable on the shed counter, because a counter reports events and both of those are the absence of an event; this gauge reports the ledger's occupancy directly. A non-zero count with zero sheds is the bound populated and under budget, which is healthy and needs no action. A count flat at zero while the silo is activating leaves means the registration path is not running and the bound is inert. All three residency gauges are registered eagerly from AddLattice rather than on first registration, so a present-and-zero reading is a measured zero and an absent series means the build does not carry them.
orleans.lattice.leaf.snapshot.coverage_repairs	Counter<long>	{evaluation}	Zero-coverage repair evaluations driven by the leaf's zero-coverage snapshot repair (issues #2692, #2940). Tagged tree and outcome with seven arms in two classes, and the distinction is load-bearing rather than taxonomic. Six are terminal outcomes, and those six partition every invocation of the repair - the denominator is named deliberately, because it includes the two invocations that decline before any repair work happens, so "invocations that reach the repair" would be the wrong denominator and would leave the declines outside the partition: repaired (a capture left every checkpointed partition covered), unsatisfied (a capture ran and a checkpointed partition is still uncovered - including a capture that threw or timed out, because the advisory capture wrapper swallows every exception, so a leaf whose snapshot store is failing is counted here rather than going silent), exhausted (the attempt budget was spent with a partition still uncovered), capture_in_flight (declined, a snapshot capture was already running on this leaf), no_checkpointed_uncovered_partition (declined, no checkpointed partition lacks coverage - the healthy majority, recorded on every activation and every checkpoint persist of a leaf that is not in the repairable population, so its magnitude tracks write volume rather than severity; it is not comparable with the blocked arm of orleans.lattice.wal.gc.blocking_pin_state, whose population is disjoint from this one by construction, but it does intersect that instrument's floor-holder arm by design, because a pin classified checkpointed_uncovered there is driven into this very repair - so that classification standing against a climbing no_checkpointed_uncovered_partition for the same leaf is a direct contradiction rather than two unrelated readings, issue #3168) and backing_off (suppressed, an earlier exhaustion armed a re-arm backoff that has not yet elapsed). The seventh, rearmed, is a lifecycle transition and NOT a terminal outcome: it records that a spent attempt budget was reset after its backoff, and the invocation that records it then proceeds to the repair and goes on to record repaired or unsatisfied as well. Those two are named rather than counted deliberately: by the time an invocation records rearmed it has passed both entry guards and both the abandon and backoff branches, so four of the six terminal arms are unreachable from there, and "one of the six" would be true only in the weak sense of not excluding them. It therefore co-occurs with a terminal arm rather than excluding one, exactly as rearmed does on the sibling blocked_leaf_reactivations_total, whose documentation separates its lifecycle events from the terminal outcomes that partition for the same reason. Do not read the seven as a partition; read the six as one, and rearmed as an event alongside them. A leaf that holds a checkpointed partition with no durable snapshot coverage publishes the (Zero, -1) block pin, and because that leaf's Zero-HLC block pin disables the cursor trim in every partition it registers one in, one such leaf disables cursor-based WAL trimming for its entire tree - which is how a single-silo deployment reached tens of gigabytes of retained WAL. Attribute that to the right axis. The disabling happens on the frontier axis, in the WAL GC durable-materialiser floor's pin <= HybridLogicalClock.Zero branch, which blocks the cursor branch for the pin's own partition; a leaf seeds a Zero pin in every partition at birth, and that is what carries a single leaf to the whole tree. The offset floor does the opposite of folding: the offset-floor computation skips a -1 reporter and lets only real checkpoints (offset >= 0) constrain the floor, specifically so that a -1 cannot collapse it and wedge the trim tree-wide - a skipped reporter is deliberately left out of the covered set and protected by its cursor or its block pin instead (issue #3172). The distinction is load-bearing rather than pedantic: attributing the fold to the offset floor makes an offset-based release read as impossible when it is in fact the designed path. The two rejection arms are the discriminator, and they are why this instrument was widened. Until issue #2940 the guard exit and the attempted-but-unsatisfied exit both recorded nothing, so "the repair never ran" and "the repair ran and failed" were byte-identical silence while calling for opposite responses: no_checkpointed_uncovered_partition says the repair is structurally unable to address the fault and routes the remedy to the pin/guard seam, whereas unsatisfied says the repair is reaching the fault and failing and routes it to the capture seam. exhausted and rearmed are a pair, and the gap between them is the reading that matters. The attempt budget was documented as a per-activation ceiling, which is self-limiting only while activations turn over - false for a leaf held permanently active by read traffic, where it was in effect a per-process ceiling that, once spent, was never reset at all. It is now abandoned on exhaustion and re-armed in place after a backoff that DOUBLES per abandonment to a ceiling, so an exhausted count with no matching rearmed is a leaf still abandoned, whereas the two advancing in step is a leaf retrying on an ever-slower schedule - the intended shape for a leaf whose captures keep failing, since the underlying fault is a capture timeout rather than a transient one and a flat backoff would have it paying a timing-out capture at constant cost for ever. All seven outcome arms are zero-primed per tree the first time a tree is seen in this process, so an absent series means the repair path never ran for that tree, or the running binary predates the instrument - it does not mean healthy - and a zero is a measured zero. Since issue #3195 this instrument also has a cadence floor: the coverage-lag timer drives the repair once per LeafSnapshotMaxCoverageLagSeconds (default 300s) per ACTIVE leaf, above the cadence gate and the debounce, so an arm is recorded regardless of write traffic - which is what makes a zero here interpretable at all, unlike a purely traffic-driven counter, provided the observation window is at least the cadence (the tick phase is jittered uniformly across the period) and the option has not been set to 0 or below, which disables the timer and removes the floor. That guarantee was previously conditional on the tree having already emitted repaired, which is precisely the trees not under diagnosis. The priming walk iterates the single declared arm set LatticeMetrics.CoverageRepairArms, pinned by a reflection test to every CoverageRepair* arm declared on the class, so a new arm declared the way every existing arm is declared is forced into the primed set. Be exact about the residual gap rather than importing the sibling's stronger claim: a recording site that inlines a tag pair instead of declaring a static would evade both the reflection test and the priming walk, so "an arm added later CANNOT ship unarmed" is true of the sibling and NOT true here. This is a weaker construction than that sibling blocked_leaf_reactivations_total, which walks an enum through a switch that throws on an unmapped member, because these arms are tag-value statics rather than enum members - the guarantee is enforced by a test rather than by the compiler. One boundary now, and the history of the other is worth keeping because it is the reason this instrument is trusted. A re-arming invocation records rearmed in addition to its terminal arm, so summing all seven over-counts on exactly the leaves this repair is working hardest on. Sum the six terminal arms if you want an invocation tally, and read rearmed separately as an event count. That sum is now EXACT rather than a lower bound: every path below the two entry guards records exactly one terminal arm, which the fixtures pin as an equality (41 evaluations, 41 increments) rather than as an inequality, so a new silent return added below the guards fails the build instead of quietly reappearing as an under-count. Until backing_off existed the sum WAS a lower bound, and the documented reason was wrong as well as incomplete: the shortfall was attributed to the once-per-activation exhaustion dedup latch, but the suppressed invocations never reached that latch at all - they took the backoff-suppression branch, which recorded nothing. That branch is the one that needed an arm, and on an eight-hour backoff against a 300-second recheck cadence it was hiding roughly ninety-six invocations behind a single exhausted. The dedup latch is in fact unreachable in the present flow (its one call site is guarded by a null re-arm deadline that the same branch immediately sets, and only the re-arm clears it, in the same step); it is retained as a defence because that reachability argument is a property of the call-site guards rather than of the latch. Read repaired as the heal rate while a stranded deployment drains, expected to fall to zero once every leaf is covered; read a sustained exhausted or unsatisfied rate as a snapshot-store or capture-seam fault; read a tree whose only non-zero arm is no_checkpointed_uncovered_partition, while its WAL is still pinned, as evidence the pin has a cause this repair does not reach.
orleans.lattice.leaf.unresolved_prepare_ledger_beyond_cap	Counter<long>	{prepare}	Resident unresolved saga prepares recorded into the leaf's durable replay-work ledger once that ledger holds at least MaxDurableUnresolvedReplayWork entries (issue #2183). The threshold is inclusive (issue #2756), so the prepare that fills the ledger to the cap is counted as well as every prepare recorded past it: the cap is the resting value at which the ledger starts refusing deferred terminals, so that is the value the signal must see. A resident prepare must never be dropped - dropping one pins the incremental flush ceiling at (prepare - 1) permanently and the leaf banks zero forward progress, which is the frozen-checkpoint livelock this counter's fix removes (a latent defect proven by a two-arm unit control, not the deployed-leaf freeze tracked as issue #2220) - so past the cap the prepare is recorded unconditionally and the row is allowed to grow for as long as a saga leaves a prepare unresolved (registry status InFlight: the residual population after issue #2190's self-terminalisation, whose orphan source is tracked as issue #2304). Persist risk: Azure Table Storage rejects oversized writes at its ~960 KB grain-state limit (see Storage provider per-row limits), bounding persisted row growth while leaving the previous row intact. Read risk: the larger SQLite limit on the default local profile permits growth that can exhaust memory or the read budget during activation, before grain-level repair can run. A successful persist is not proof of a safe activation read; write amplification is not the only cost. Alert on every profile, including local. It is observability only - nothing here caps or drops a prepare, because a behavioural cap would reintroduce the exact drop-and-freeze defect issue #2183 removes. A paired warning fires once per activation when the crossing first occurs. Tagged tree and partition (the WAL partition ordinal, bounded by WalPartitions).
orleans.lattice.materialiser.drain_lag	Histogram<double>	ms	Materialiser drain lag, in milliseconds, recorded by the WAL saturation sampler on every tick (default 200 ms) while the drain-lag input is enabled, as the WAL head wall-clock timestamp minus the slowest eligible in-memory WAL consumer cursor - leaf materialisers and tree-wide consumers alike - clamped at zero. Recorded for every checked tree, not only over-threshold ones. A sustained breach of WalSaturationMaterialiserLagThreshold holds the tree at Throttled and never escalates to Saturated. Measures the in-memory cursor, so it does not observe the durable materialiser pin store that sets the WAL retention floor (issue #2015); orleans.lattice.materialiser.pin.durable_write_latency is the input that does. The frontier is a minimum over the consumers WalDrainLagConsumerFreshness admits: a consumer whose last report is older than that window is excluded (issue #2446), and so is a leaf materialiser whose position has not advanced within it and predates it, because a leaf's cursor covers only its own key range and a leaf whose range received no writes is caught up however far its cursor trails the tree-wide head (issue #3131). A leaf that is draining, or a tree-wide tailer that has stalled, still sets the reading on its own, and the aggregate carries no dimension naming which consumer contributed it, so a large value is a prompt to identify the contributor - the sampler logs up to three eligible holders on the crossing and periodically during a standing breach (see holder observations) - rather than evidence that projection is falling behind ingest (issue #2433). Tagged tree.
orleans.lattice.materialiser.lagging_consumers	Histogram<int>	{consumer}	Count of WAL cursor consumers whose own reported cursor trails the WAL head wall clock by more than WalSaturationMaterialiserLagThreshold, recorded by the WAL saturation sampler for a tree whose aggregate drain lag is already over that same threshold on the same tick. Counts exactly the consumers the aggregate was computed over: consumers that have never reported a cursor (HLC zero), cold consumers, and position-stale leaf materialisers are excluded by the same predicate as the min(cursor) meet (issues #2446, #3131). It exists because orleans.lattice.materialiser.drain_lag above is a minimum and therefore reads identically for one legitimately dormant consumer holding the frontier down and for many consumers genuinely falling behind, two conditions whose correct responses are opposite (issue #2444). It makes that aggregate triageable, not diagnosable: it separates those two shapes but still does not name the contributing consumer, which is named out of band by the holder observations. Sampled only for trees already found over threshold, so a healthy estate adds no per-tick cost and emits nothing. Tagged tree.
orleans.lattice.materialiser.pin.durable_writes	Counter<long>	{write}	Durable writes to the leaf-materialiser pin store: one increment per persist that issued at least one durable write, so a persist that throws records nothing and, once a shard's pins span several slots, one persist can rewrite more than one slot. Tagged tree and outcome: birth (the awaited seed that makes a newly born leaf's block pin durable before its data becomes reachable) or coalesced (every other persist - chiefly the coalesced flush that makes merged pin reports durable, plus a pin removal, the tree-deletion clear, a re-layout and the deactivation flush). It counts writes, not advancement, and the two are not proportional: pins are bucketed, so one advancing consumer rewrites its whole bucket, and a merge that moves only the HLC frontier rewrites the pin at the same checkpoint offset. Offset advancement is the quantity that lets the WAL GC offset floor move, so read orleans.lattice.materialiser.pin.advances for that (issue #2694), not this.
orleans.lattice.materialiser.pin.advances	Counter<long>	{report}	Merged leaf-materialiser pin reports partitioned by which of the two axes advanced, emitted once per merged report. Tagged tree and outcome: both (the HLC frontier and the durable checkpoint offset moved together), offset_only (the offset moved and the frontier did not), frontier_only (the frontier moved and the offset did not), or none (a report at or behind the existing pin on both axes). The four arms are the complete truth table of the two axes, so exactly one is recorded per report and each marginal is recoverable by summing: offset advanced is both + offset_only, frontier advanced is both + frontier_only. They were three first-match arms until issue #3163: offset absorbed every merge that also advanced the frontier, so it meant "the offset advanced, the frontier is unknown", and an absent frontier_only did not show a flat frontier - only one that never advanced alone. Read both + offset_only when asking "why is retained WAL not being reclaimed?", and read a healthy offset_only rate with both flat as a consumer advancing in offset space alone, which cannot release a WAL entry because the offset floor only lowers a trim point the HLC clauses already authorised. It exists because durable_writes above cannot separate any of these (issue #2694). Every arm is zero-primed once per pin-shard activation, so all four series exist for a tree with a live pin grain and a flat arm is a measured zero rather than an absence.
orleans.lattice.materialiser.pin.durable_write_latency	Histogram<double>	ms	Caller-observed duration of a durable leaf-materialiser pin write, recorded by LeafCursorReporter around each pin grain call that reached WalSaturationMaterialiserPinLatencyThreshold or faulted. The only materialiser instrument that observes the durable retention floor rather than in-memory progress, so it is what makes a stalled pin store visible (issue #2015). Measured at the call site, so it includes time queued ahead of the shard's non-reentrant activation. Emitted only when the threshold option is set; it defaults to null. Tagged tree.
orleans.lattice.materialiser.pin.reports_shed	Counter<long>	{report}	Steady-state per-checkpoint pin reports a leaf's cursor reporter declined to enqueue because the target shard's last durable write showed the store is not keeping up. Only that path is shed; the birth block-pin seed and the deactivation flush are last-chance writes and are always issued. Shedding is safe for durability - a shed report leaves the durable pin staler, which only ever retains more WAL - but it is not safe for boundedness, and the two were conflated until issue #3310. The shed path is the only one that carries an advancing checkpoint offset, so for as long as it is suppressed the WAL GC offset floor cannot restamp and retained WAL grows without bound. A sustained non-zero rate means the pin store is the bottleneck; pair it with durable_write_latency and shed_stall_seconds, and consider raising WalMaterialiserPinBuckets. Tagged tree and pin_shard. pin_shard is not shard: it is the durable-pin routing shard (WalMaterialiserPinShards), a different hash of a different key from the physical WAL partition tag, and both default to 8. A join between them looks well-formed and means nothing. Zero-primed: the WAL GC scheduler emits a single Add(0) for each tree on its first pass, so a tree that has never shed reports an explicit 0 rather than no series at all. That primed series carries tree and tenant but no pin_shard, so it sits beside the per-pin-shard series a real shed records rather than priming them - aggregate over pin_shard to read the two together - and its tree is the registry id the scheduler walks, the physical copy's id included on a resized, restored or remediated tree, while a real shed records under the logical id (see The tree dimension across aliasing). Before that priming the healthy case and a dead reporter were both exported as nothing (issue #2694).
orleans.lattice.materialiser.pin.shed_forced	Counter<long>	{report}	Steady-state pin reports forced through a live shed window because that pin shard had been shedding continuously for longer than WalMaterialiserPinShedCeiling. Structurally zero unless that option is set; it defaults to unset (disarmed), so an unconfigured host reports no forcing at all. Every increment is one deliberate override of the issue #2014 back-pressure, so a sustained rate is an alarm rather than routine: it means the pin store has been unable to keep up for longer than the ceiling and WAL retention was being held hostage to it. Forcing cannot overstate durability - ResolveDurablePinForPartition clamps the reported offset to min(checkpoint, covered) inside the leaf, so a forced report publishes a true value sooner and never a higher one, which is what keeps this fix on the safe side of issue #3300. Tagged tree and pin_shard. pin_shard is not shard: it is the durable-pin routing shard (WalMaterialiserPinShards), a different hash of a different key from the physical WAL partition tag, and both default to 8. A join between them looks well-formed and means nothing.
orleans.lattice.materialiser.pin.shed_stall_seconds	ObservableGauge<long>	s	Age, in seconds, of each pin shard's current unbroken run of shed reports. It resets to zero the moment any report gets through, and the series is omitted entirely for a shard that is not currently shedding, so an idle host publishes nothing rather than a wall of zeros. This is the instrument that separates a healthy burst of shedding from a latch: reports_shed climbs at the same rate under both, whereas a rising shed_stall_seconds means the window has never once closed, and so the durable pin offset - and with it the WAL GC offset floor - has not restamped for that long (issue #3310). It is emitted whether or not WalMaterialiserPinShedCeiling is armed, which is what keeps the default-disarmed configuration observable instead of silent. Tagged tree and pin_shard. pin_shard is not shard: it is the durable-pin routing shard (WalMaterialiserPinShards), a different hash of a different key from the physical WAL partition tag, and both default to 8. A join between them looks well-formed and means nothing.

Drain-lag holder observations

The WAL saturation sampler attributes an over-threshold orleans.lattice.materialiser.drain_lag reading through structured Warning logs, never consumer-id metric tags. It emits an observation immediately on the crossing tick and repeats while the tree stays strictly over WalSaturationMaterialiserLagThreshold, no more often than WalDrainLagHolderLogInterval (10 minutes by default; null means edge-only). Recovery clears the interval. This is per-tree, per-silo diagnostic state, not a durable assignment of blame; a later snapshot may name different consumers.

Each observation contains up to three lines, naming the three eligible consumers with the lowest HLC cursors in ascending order (ties retain snapshot order). All lines share a generated ObservationId and the sampler tick's ObservedAtUtc. Each carries TreeId, DrainLagSeconds (the aggregate), LaggingConsumers (the full eligible over-threshold count, not capped at three), HolderRank (1 is the minimum), HolderCount, ConsumerId, Cursor, ReportAgeSeconds and PositionAgeSeconds. Group by ObservationId rather than counting lines as separate breaches. ConsumerId is operator-facing identity: restrict access to these logs as you would other tree diagnostics. No new admin endpoint or metric instrument is involved.

A recent report with an old position age means a consumer is re-reporting without advancing; a recent position age means it has just advanced, even if its cursor still trails the head. PositionAgeSeconds = null means no advance was observed since registration, or the registry does not track it. Cold consumers, never-reported cursors, and position-stale leaf materialisers are excluded by the same lag-plane predicate as the histogram; tree-wide tailers remain eligible on report freshness alone. Multiple consumer ids can belong to one leaf (one per WAL partition), so three holders are not necessarily three distinct leaves. The snapshot is read after the aggregate: concurrent progress may change the holders or leave none eligible, in which case no holder is invented or logged. Compare successive observations and the lagging-consumer count before inferring stalled work versus active catch-up.

Selection reuses the snapshot already read in the over-threshold branch. Only a due warning selects the top three, using three local nullable value types and no sorting, LINQ or per-tick holder collection. Monotonic rate-limit state is bounded by over-threshold tree cardinality and removed on recovery, disappearance from the sampler or input disablement. Logging arguments allocate only when an observation is emitted; the leaf-materialiser drain loop is unchanged.