Table of Contents

Instrument catalog: WAL compaction (sourced from FileWalShard) to Foreground read envelopes (sourced from LatticeGrain)

Part of Instrument catalog, in Metrics.

WAL compaction (sourced from FileWalShard)

A log-structured WAL provider does not free disk when the garbage collector trims: trimming marks entries dead, and only a compaction

  • a rewrite of the surviving entries into a fresh file - returns the bytes. These two counters are the only signal that the rewrite half is happening at all. Issue #3107 ran undetected precisely because it was absent: a tree trimmed 113,675 entries over 55 minutes while its WAL grew by 1.1 MB, and no series anywhere distinguished that from a healthy drain. Read them against orleans.lattice.wal.entries_trimmed: sustained trimming with a flat wal.compactions means dead bytes are accumulating and the configured thresholds are not being reached.

Read that discriminator per shard, never per tree. All three of those series (wal.entries_trimmed, wal.compactions and wal.compaction.reclaimed_bytes) carry a shard tag, added by issue #3206, because every compaction trigger is shard-local: the ratio test compares one shard's dead bytes against its own payload, so summing to the tree averages one threshold test per shard and reports a dead fraction that no shard actually holds. A tree-scoped read is therefore not a coarser version of the right answer, it is a different and wrong one - an active minority of shards keeps the tree counters advancing while a stranded majority of the bytes never moves, and the healthy shape and the fault shape are identical at tree scope. On the estate that motivated #3206, three of eight shards holding 81% of a 1.6 GB WAL had never compacted while the tree-level counters looked healthy. Group by shard and apply the discriminator to each series.

Name Kind Unit Description
orleans.lattice.wal.compactions Counter<long> {compaction} Compactions completed by the file WAL provider, tagged tree, shard, tenant, and trigger, whose three arms are the complete set of paths that can rewrite a shard: ratio (dead bytes reached CompactionThreshold of the file, the historical trigger), ceiling (dead bytes reached the absolute CompactionMaximumDeadBytes cap added by issue #3107), and reconcile (an unconditional compaction at shard activation). Every arm is zero-primed per shard when that shard first loads, so a flat zero is a measured zero rather than an unpublished series - the distinction that mattered in #3107, where "this WAL has never reclaimed a byte" was indistinguishable from "this deployment does not run the file provider", and that issue #3206 restored at shard scope, where an unprimed shard would leave "this shard has never compacted" indistinguishable from "this shard is not reporting". ceiling is flat by default and that is correct, not a fault: CompactionMaximumDeadBytes defaults to 0 (disabled), because compaction rewrites every live byte to reclaim the dead ones, so an absolute cap far below the live size raises write amplification without bound. A shard whose wal.entries_trimmed never advances at all has a different and prior fault, not a milder one: nothing was trimmed, so nothing was marked dead, so this gate is evaluated and correctly declines at a dead fraction of zero (issue #3258). A shard whose ratio arm never advances while wal.entries_trimmed climbs for that same shard is stranding space at a dead fraction below the threshold, and is the case the ceiling exists to bound.
orleans.lattice.wal.compaction.reclaimed_bytes Counter<long> By Dead bytes released by completed compactions, tagged tree, shard, and tenant, summed at the moment each rewrite finishes. Deliberately a monotonic counter rather than an up/down gauge of current dead bytes: a gauge of that kind needs a compensating write per trim, and issue #2700 established here that a compensating write which can be lost - to a disposed shard, a re-created provider, or a torn activation - ratchets the series permanently, leaving real waste indistinguishable from accumulated drift, which is the exact question the instrument exists to answer. Present-truth occupancy is reported instead by ILattice.GetStorageUsageAsync's WalPhysicalBytes, which is derived per sample and so cannot drift. Zero-primed per shard alongside wal.compactions.
orleans.lattice.wal.compaction.eval.retained_bytes Histogram<long> By Retained (live) payload bytes the shard held when its compaction threshold was evaluated. Tagged tree, shard, and tenant.
orleans.lattice.wal.compaction.eval.dead_bytes Histogram<long> By Dead (trimmed, not yet reclaimed) payload bytes the shard held at the same instant - the numerator of the ratio arm, and the quantity the minimum-dead floor and the absolute ceiling are both tested against. Tagged tree, shard, and tenant.
orleans.lattice.wal.compaction.eval.retained_entries Histogram<long> {entry} Retained (live) entry count at the same instant. Tagged tree, shard, and tenant.
orleans.lattice.wal.compaction.eval.dead_entries Histogram<long> {entry} Dead (trimmed, not yet reclaimed) entry count at the same instant. Distinct from wal.entries_trimmed, which counts trim events monotonically: this is the present backlog awaiting a rewrite, and it returns to zero each time one completes. Tagged tree, shard, and tenant.
orleans.lattice.wal.recovery.torn_tail_bytes Counter<long> By Bytes truncated from a WAL shard log by activation-time recovery because they were never sealed by a commit record. Tagged tree, shard, and tenant. Not a fault on its own: after a crash a non-zero reading is expected and benign, whereas after a drain that completed the expected reading is exactly zero, so a non-zero reading there reports acknowledged writes that did not survive the restart (issue #3366). Read it against the recorded exit rather than in isolation. Zero-primed per shard, which is what makes a flat zero a measured zero rather than an unpublished series - the distinction that mattered here, because recovery truncates the very evidence the value is derived from, so a measurement not taken at activation cannot be reconstructed afterwards.
orleans.lattice.wal.recovery.torn_tail_records Counter<long> {record} Complete data records discarded from a WAL shard log by activation-time recovery because no commit record ever sealed them. Tagged tree, shard, and tenant. Read with orleans.lattice.wal.recovery.torn_tail_bytes, which it disambiguates: bytes above zero with a record count of zero is a single interrupted write, the ordinary shape of a kill mid-append, whereas a non-zero record count is a run of complete records that were written and never committed. Zero-primed per shard alongside the byte counter.

Reconstructing the compaction gate from outside the process

The four wal.compaction.eval.* samples above are the complete input set of the shard-local compaction gate, recorded once per evaluation, before any arm is tested, and armed from the shard's true post-recovery state the first time it loads. Two separate ambiguities make them load-bearing rather than supplementary, and neither is closed by the outcome counters.

A flat outcome counter has two opposite causes. A shard whose compactions series never advances may be evaluated on every sweep and correctly declining, or may never be reaching the evaluation at all. Those have opposite remedies, and nothing in the outcome counters distinguishes them. The eval.* samples do: they are emitted by the evaluation itself, so their presence is proof it ran and their values are exactly what it tested.

Issue #3207 closed the second of those causes for any shard the collector visits. Until then the gate was read in exactly one place, at the end of the provider's trim, so a shard whose scan stopped on its first entry - the shape a held tree-wide offset floor produces - was never evaluated at all, at any dead ratio, for as long as the floor stood. The collector now evaluates a shard it released nothing from, so on a swept tree an absent eval.* sample indicts the sweep reaching that shard rather than the gate, and a present sample alongside a flat compactions series is genuine threshold-bound declining.

A dead ratio derived from published byte totals is only an upper bound. eval.retained_bytes and eval.dead_bytes - like every WAL byte figure this library publishes - count payload length, whereas the file stores framed records: 17 bytes per data record, 17 per trim marker, and 13 per commit record (FileWalRecordFormat). Subtracting a retained total from a physical total therefore leaves dead payload plus all of that framing, and since (D + V) / (L + D + V) > D / (L + D) for any framing total V > 0, the derived ratio always overstates the one the gate actually tests. The correction is not uniformly negligible: at a ~4 KB mean payload it is well under a percent, but at ~256 B it is several percent, which is enough to move a shard sitting near the default CompactionThreshold from one side of it to the other, and so to change whether the correct remedy is to reach the evaluation or to lower the floor.

The entry counts are what close that gap. retained_bytes / retained_entries gives the mean live payload and dead_bytes / dead_entries the mean dead payload, so the framing term becomes computable and the gate's own ratio - dead_bytes / (retained_bytes + dead_bytes) - is recoverable exactly rather than bounded. Before issue #3206 no published series carried a WAL entry count except wal.entries_trimmed, so mean payload was not derivable from telemetry at all.

Read every one of them per shard. Summing across shards averages one independent threshold test per shard and reports a dead fraction that no shard holds, which is the whole defect issue #3206 was filed for.

Per-tree admission control

Opt-in, fail-open per-tree quota enforcement. These per-tree instruments are on the orleans.lattice meter and tagged tree; the two counters (admission.rejected, admission.would_reject) and the admission.utilization gauge additionally carry a low-cardinality dimension tag whose value is keys or bytes. The observable gauges are observed lazily from the same cached, TTL-coalesced per-tree aggregate that backs the storage-usage gauges, so they cost nothing on the write path; they populate once the per-tree storage-usage aggregator has sampled the tree (which happens on any GetStorageUsageAsync call, whenever StorageUsageDeepPollInterval is enabled, and automatically whenever a cap or advisory ceiling is configured, because the write guard refreshes the aggregate at most once per StorageUsageCacheTtl). A tree's admission series stops being reported once its sample has not been refreshed for 60 s, so a silo that no longer hosts the tree's aggregator does not keep reporting it.

Name Kind Unit Description
orleans.lattice.admission.live_keys ObservableGauge<long> {key} Current live (non-tombstone) key count for the tree - the figure compared against LatticeOptions.MaxLiveKeys. Best-effort / eventually-consistent (assembled from each shard root's incrementally-maintained live-key total, re-anchored on a deep refresh); a time-expired entry not yet reaped by compaction counts as live until the next re-anchor. Tagged tree.
orleans.lattice.admission.estimated_bytes ObservableGauge<long> By Current estimated storage footprint for the tree - the figure compared against LatticeOptions.MaxEstimatedBytes. It is the total a deep storage-usage report computes, the figure storage.total_bytes carries after a deep publish: the WAL's physical occupancy (dead bytes a log-structured provider has not yet compacted included) + snapshot blobs + leaf/shard-root state, so dead WAL bytes count toward the cap. The WAL-only polls that refresh storage.total_bytes between deep publishes do not move it. Tagged tree.
orleans.lattice.admission.over_advisory ObservableGauge<long> 1 1 when the tree is currently at or above an advisory ceiling (AdmissionAdvisoryLiveKeys or AdmissionAdvisoryBytes), else 0. Emitted only for a tree that has set at least one advisory ceiling. Non-enforcing: no write is rejected. Tagged tree.
orleans.lattice.admission.would_reject Counter<long> {write} Incremented once per checked write call - one of the calls that check the enforcing caps, listed under MaxEstimatedBytes, a batch counting once - made while the tree is at or above an advisory ceiling, once for each dimension it is over: the dry-run blast radius of a candidate cap. No write is rejected, and no other write call is counted. Tagged tree and dimension (keys or bytes).
orleans.lattice.admission.utilization ObservableGauge<double> 1 Current / ceiling utilisation per dimension. The denominator prefers the enforcing cap (MaxLiveKeys / MaxEstimatedBytes) and falls back to the advisory ceiling when only that is set; a series is emitted only for a dimension whose ceiling is configured. At or above 1 the denominator ceiling is reached - the enforcing cap is being hit where one is configured, and only the advisory ceiling where it is the sole ceiling. Tagged tree and dimension (keys or bytes).
orleans.lattice.admission.rejected Counter<long> {write} Incremented once per checked write call actually rejected by an enforced cap (one LatticeQuotaExceededException thrown, however many entries the call carried). Tagged tree and dimension (keys or bytes).

Semantics. Admission control is strictly opt-in: every cap and ceiling defaults to null (unbounded), and a tree with none set pays nothing and never takes a grain hop on the write path. The cap is enforced against the cached per-tree aggregate, never a per-write fan-out, so it is best-effort and approximate: every call admitted before a refreshed sample lands - including the whole of a batch admitted on one check - can carry the tree past a cap, and each new activation of the tree's grain fails open (never rejects) until its own first sample lands. Only the write calls listed under MaxEstimatedBytes are checked against a cap or ceiling, together with the cross-tree SetManyAtomicAsync extension, which reads the aggregate when it admits the batch; the bulk-load calls and MergeAsync are neither refused by a cap nor counted. Replication and atomic-write-saga apply paths bypass enforcement, so an incoming replicated write is never rejected. On an enforcing breach the call throws LatticeQuotaExceededException (dimension keys or bytes, with the observed Current and configured Limit), a recoverable back-off signal.

Advisory-first-then-enforce adoption workflow.

  1. Observe. With no cap set, watch admission.live_keys and admission.estimated_bytes (enable StorageUsageDeepPollInterval so the gauges populate without a cap configured) to learn each tree's steady-state footprint.
  2. Set an advisory ceiling. Configure AdmissionAdvisoryLiveKeys and/or AdmissionAdvisoryBytes at a candidate ceiling. This rejects nothing.
  3. Right-size. Watch admission.over_advisory (which trees sit over the candidate ceiling) and the admission.would_reject rate (how many checked write calls a promoted cap would reject, by dimension - a batch counts once) to tune the ceiling until the dry-run blast radius is acceptable.
  4. Promote and enforce. Move the tuned value to MaxLiveKeys and/or MaxEstimatedBytes. Watch admission.utilization (how close each tree runs to its cap) and the admission.rejected rate (checked calls actually refused) to confirm enforcement behaves as intended.

Compression dictionary auto-training

These instruments are emitted only when auto-training of the Zstandard compression dictionary is enabled. They expose training cadence and the realised compression of the most recent training probe.

Name Kind Unit Description
orleans.lattice.compress.dictionary.training_runs Counter<long> {run} Auto-training dictionary pass attempts. Tagged outcome (trained, skipped_insufficient_samples, or skipped_cadence).
orleans.lattice.compress.dictionary.trained_bytes_in Counter<long> By No-dictionary (plain Zstd) baseline compressed bytes of the training probe, summed per successful auto-training pass.
orleans.lattice.compress.dictionary.trained_bytes_out Counter<long> By Trained-dictionary compressed bytes of the training probe, summed per successful auto-training pass. Dividing this by trained_bytes_in gives the realised dictionary compression ratio.
orleans.lattice.compress.dictionary.active_version ObservableGauge<long> {version} Currently active auto-trained dictionary id (0 = none trained yet).
orleans.lattice.compress.dictionary.reservoir_fill ObservableGauge<long> 1 Auto-training reservoir occupancy. Tagged kind (samples or bytes).

Cache (sourced from LeafCacheGrain)

Name Kind Unit Description
orleans.lattice.cache.hits Counter<long> {hit} Leaf-cache reads served by a live, cached entry. Tagged tree. Only reads a shard root routes through the leaf cache are counted - the serial point read, ExistsAsync and batched GetManyAsync reads. A point read that the shard root's optimistic read validates (OptimisticShardRootPointReads, on by default) goes to the primary leaf instead and is counted on neither this nor cache.misses, so the hit ratio describes the cache-routed reads, not every point read.
orleans.lattice.cache.misses Counter<long> {miss} Leaf-cache reads that did not find a live cached entry. Tagged tree. Same population as cache.hits: validated optimistic point reads bypass the cache and are not counted.

Foreground write envelopes (sourced from LatticeGrain)

The foreground write paths (SetAsync, SetManyAsync, SetManyWherePredicateAsync) are wrapped at the caller-visible boundary with envelope histograms (and, for SetAsync and SetManyAsync, per-sub-stage histograms) so the end-to-end per-call wall-clock cost can be attributed to its constituent spans without per-call wire-format change. Pair these with the per-leaf leaf.commit.duration and per-WAL wal.* instruments to walk the full path from ILattice entry through Orleans RPC down to the storage provider. Each envelope's timer starts only once the call has passed its entry guards, authorization, write admission and write interception, so a call refused (or short-circuited) there records no envelope or stage sample. SetAsync here means the overload without a TTL: a TTL write (SetAsync(key, value, ttl)) records neither set.duration nor set.stage.duration.

Name Kind Unit Description
orleans.lattice.set.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.SetAsync call. Tagged tree.
orleans.lattice.set.stage.duration Histogram<double> ms Per-sub-stage wall-clock duration of one ILattice.SetAsync call. Tagged tree and stage=gate (the per-call pre-flight that ensures the tree's compaction reminder and hot-shard monitor are registered), shard (shard routing, the cross-grain shard-root RPC envelope and any stale-routing retries, recorded as one span because the three interleave inside the retry loop - this is the dominant cell at the c2-iii operating point), or publish (event-stream dispatch). Unlike set_many.stage.duration, no separate route stage is recorded.
orleans.lattice.set_many.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.SetManyAsync call. Tagged tree.
orleans.lattice.set_many_where_predicate.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one ILattice.SetManyWherePredicateAsync call - the conditional counterpart of set_many.duration, spanning the same gate, routing, per-shard fan-out, and event-publish work (events for the written subset only). Once its timer has started it is recorded whether the call completes or throws; a call refused before that point records nothing (see above). No sub-stage histogram accompanies it. Tagged tree. The envelope to compare the operation="set_many_where_predicate" arm of shard_root.set_many.local_apply.duration and shard_root.set_many.leaf_rpc.duration against.
orleans.lattice.set_many.stage.duration Histogram<double> ms Per-sub-stage wall-clock duration of one ILattice.SetManyAsync call. Tagged tree and stage=gate (the same per-call pre-flight as on set.stage.duration), route (per-entry routing), bucket (per-shard bucket build), fanout (the cross-shard branch fan-out - the dominant cell under saturated traffic; it settles on the first faulted branch, so a failed call measures time-to-first-fault rather than the branch tail), or events (event-stream dispatch).

Foreground read envelopes (sourced from LatticeGrain)

The foreground read paths (GetAsync, GetManyAsync, ExistsAsync, GetWithVersionAsync) are wrapped at the caller-visible boundary with envelope histograms (and, for the point and batched reads, per-sub-stage decomposition) so an ILattice consumer can measure true per-call read latency without falling back to the silo's per-batch ingest envelope. Pair these with the per-leaf leaf.scan.duration (range scans) and per-shard shard.reads counter (count, not latency) for a complete picture of the read pipeline.

Name Kind Unit Description
orleans.lattice.get.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.GetAsync call (includes routing resolution, the shard RPC, and any stale-routing retries). Tagged tree.
orleans.lattice.get.stage.duration Histogram<double> ms Per-sub-stage wall-clock duration of one LatticeGrain.GetAsync call. Tagged tree and stage=route (per-attempt shard resolution) or shard (the per-attempt shard-root read: the optimistic read while OptimisticShardRootPointReads is on and, unless it validates, the serial read that follows it, both inside one sample - see orleans.lattice.shard_root.optimistic_read.outcomes). Under a stale-routing storm a single envelope produces multiple route and shard observations so the histograms attribute the retry cost.
orleans.lattice.shard_root.optimistic_read.outcomes Counter<long> {read} Count of the interleaved optimistic point reads a shard root attempts for ILattice.GetAsync (issue #3474), recorded by the shard root and tagged tree, outcome, and tenant. The caller attempts one only while its own options have OptimisticShardRootPointReads on, so turning the option off leaves every arm at its primed zero rather than moving reads to disabled. outcome=validated is the only arm served without the serial read; every other arm names why the read was handed back to the serial, non-interleaved shard-root read, which holds the shard root's turn for a full leaf round trip: disabled (the caller attempted the read but the shard root's own per-tree options have the flag off, or could not be resolved), busy (a routing mutation or pending prepare / split / graft work; ordinary non-splitting point Sets do not close this gate), gate_closed (tree rejecting, retained redirect, deleted, or the key's slot moved away), routing_cache_miss (the leaf is not resolvable from the cached routing tables), epoch_changed (routing moved while the leaf read was in flight, including a leaf fault raised while it moved), leaf_generation_changed (no warmed stamp when required, different leaf activation/generation, a present proof-path reply without proof, or a raw leaf fault during a point write), and absent (a null reply without a usable ownership proof). Uncontended present replies use the raw fast path. A null reply requires matching leaf activation/generation and synchronous range ownership; old-wire leaves without stamps retry serially when proof is required. Every arm is primed at zero on activation. A high non-validated share predicts per-shard-root read queueing that the get.stage.duration shard stage shows only as latency, because both attempts land inside one sample.
orleans.lattice.get_many.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.GetManyAsync call (includes routing, per-key bucketing, per-shard parallel fan-out, the registry-snapshot double-check, and any stale-routing retries). Tagged tree.
orleans.lattice.get_many.stage.duration Histogram<double> ms Per-sub-stage wall-clock duration of one LatticeGrain.GetManyAsync call. Tagged tree and stage=route (GetRoutingAsync fetch), bucket (per-key shard bucketing), fanout (registry snapshot + cross-shard Task.WhenAll), or merge (topology- and snap2-stability checks + final dictionary materialise). Recorded once per stage per inner snapshot-retry attempt so a snapshot- or topology-retry storm is visible as a multiplicative bump on the same envelope.
orleans.lattice.exists.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.ExistsAsync call. Tagged tree. Lower-traffic than get.duration in typical workloads but published for symmetry so a dashboard tile can confirm probe activity.
orleans.lattice.get_with_version.duration Histogram<double> ms End-to-end caller-visible wall-clock duration of one LatticeGrain.GetWithVersionAsync call. Tagged tree. Lower-traffic than get.duration in typical workloads but published for symmetry so a dashboard tile can confirm versioned-read activity.