Table of Contents

Tag conventions

Part of Metrics.

Every Lattice instrument draws its tags from one consistent, low-cardinality vocabulary. The table lists every tag key the catalog below uses; where a key is shared by many instruments, Applies to names examples and each instrument's row lists its own arms:

Tag key Applies to Value
tree every tree-scoped instrument - a platform-scoped instrument (for example the registry instruments, the WAL replay permit gauges, or orleans.lattice.build.info) carries none; see each instrument below The logical tree id - the id the tree's ILattice grain is addressed by - which grain-side series keep across an alias swap; the WAL garbage collector, the WAL storage providers and a few named series carry the physical copy's id instead: see The tree dimension across aliasing
tenant every instrument Owning tenant derived from the tree id rather than measured: a tenant id, default for a bare legacy tree id, or the reserved _platform_ sentinel for the _lattice_ and sys- namespaces, for a malformed id in the reserved t/ namespace, and for an instrument or arm with no tree dimension. Most descriptions below leave it out of their tag list, but every series carries it, and a hygiene gate over every emission site keeps it that way. See The derived tenant label
shard instruments whose row lists a shard tag, on the shard-root side (for example orleans.lattice.shard.reads and the orleans.lattice.shard_root.scan_page.* family) and on the WAL side (the WAL append pipeline, WAL GC trim, WAL compaction and recovery, and storage-provider commit instruments) An int index with two meanings that are never joined: the tree's physical shard on the shard-root side, and the WAL shard - a mutation's partition, 0 to WalPartitions - 1 - on the WAL side. Distinct from partition and pin_shard below
operation orleans.lattice.leaf.scan.duration, orleans.lattice.registry.call.duration, orleans.lattice.registry.call.in_flight, orleans.lattice.shard_root.set_many.local_apply.duration, orleans.lattice.shard_root.set_many.leaf_rpc.duration keys or entries (leaf.scan.duration); the registry member served - exists, get_entry, get_entries, resolve, get_shard_map, get_all_tree_ids, register, or unregister - on the two registry instruments; set_many (unconditional SetManyAsync) or set_many_where_predicate (conditional SetManyWherePredicateAsync) on the two shard-root batched-write instruments
step orleans.lattice.leaf.commit.duration only wal, apply, observer, or digest
observer orleans.lattice.observer.duration only The IMutationObserver implementation's CLR type name (Type.FullName)
stage orleans.lattice.set.stage.duration, orleans.lattice.set_many.stage.duration, orleans.lattice.get.stage.duration, orleans.lattice.get_many.stage.duration, orleans.lattice.saga.broadcast.shard.stage.duration, orleans.lattice.wal.gc.pass.reach, orleans.lattice.wal.gc.tree.reach Sub-stage name within an envelope, or the region a WAL GC pass reached (the two reach instruments) - see each instrument below
phase the storage-provider commit instruments (orleans.lattice.provider.commit.duration, orleans.lattice.provider.retry.exhausted, orleans.lattice.provider.idempotent_replays, orleans.lattice.provider.phase1.transient_retries); the scan-page stall instruments (orleans.lattice.shard_root.scan_page.stalls, orleans.lattice.scan.stall_resumptions, orleans.lattice.scan.stall_futility_outcomes); orleans.lattice.saga.checkpoint.duration, orleans.lattice.saga.reminder.duration, orleans.lattice.atomic_action.step, orleans.lattice.wal.gc.scheduler.phase_age phase1 (per-batch partition txn) or phase2 (manifest partition txn) on the provider instruments, where idempotent_replays and phase1.transient_retries are always phase1; prologue, descent, leaf-walk or baseline-fold on the scan-page instruments; the call site on the two saga instruments; forward or compensate on atomic_action.step; the scheduler loop's phase on phase_age
status orleans.lattice.provider.retry.attempts, orleans.lattice.provider.retry.exhausted, orleans.lattice.provider.retry.short_circuited; the WAL GC pin-classification and floor instruments (orleans.lattice.wal.gc.blocking_pin_state, orleans.lattice.wal.gc.orphan_pin_sweep, orleans.lattice.wal.gc.floor_holder_classification, orleans.lattice.wal.gc.coverage_unknown_pin_offset, orleans.lattice.wal.gc.never_checkpointed_pin_offset, orleans.lattice.wal.gc.floor_holder_admission, orleans.lattice.wal.gc.floor_holder_offset_admission, orleans.lattice.wal.gc.durable_floor_stall_seconds) Azure Tables HTTP status string on the provider instruments (e.g. 503, 429; 0 for a transport-level failure with no HTTP exchange; unknown when the SDK surfaced no status); a classification arm on the WAL GC instruments - see each instrument below
outcome many instruments - every row below that lists outcome arms, for example orleans.lattice.atomic_write.completed, orleans.lattice.wal.gc.passes, orleans.lattice.leaf.snapshot.captures and orleans.lattice.grain.call.duration Discriminator - see each instrument below
kind orleans.lattice.coordinator.completed, orleans.lattice.coordinator.phase_tick.failures, orleans.lattice.coordinator.phase_tick.consecutive_failures, orleans.lattice.tree.lifecycle, orleans.lattice.events.published, orleans.lattice.leaf.write.duration, orleans.lattice.leaf.residency.sheds, orleans.lattice.shard_root.flush.retries_suspended, orleans.lattice.compress.dictionary.reservoir_fill Discriminator - see each instrument below
trigger orleans.lattice.compaction.pass.duration, orleans.lattice.compaction.leaves.visited, orleans.lattice.leaf.compaction.duration, orleans.lattice.leaf.tombstones.reaped, orleans.lattice.leaf.tombstones.expired, orleans.lattice.wal.compactions, orleans.lattice.wal.replay.permit_adaptations reminder, ratio, size, or operator (the five tombstone-compaction instruments - always on compaction.pass.duration, and on the other four only when a compaction policy knob is non-default); ratio, ceiling, or reconcile (wal.compactions); fault or occupancy (permit_adaptations, withheld arm only)
path orleans.lattice.compaction.leaves.visited, orleans.lattice.leaf.digest.publishes walk or dirty-set (compaction.leaves.visited); coalesced_scheduled, coalesced_skipped, coalesced_fired, inline, or deactivation_flush (leaf.digest.publishes)
reason many instruments - every row below that lists reason arms, for example orleans.lattice.events.dropped, orleans.lattice.leaf.activation.failures, orleans.lattice.wal.gc.trim_stop and orleans.lattice.storage.wal.compression_skipped Discriminator - see each instrument below
config orleans.lattice.config.changed Configuration dimension name: publish_events or history_retention
wal_partitions the WAL append-pipeline histograms (orleans.lattice.wal.append.*, orleans.lattice.wal.shard.dispatch.*) and the saga coordinator's histograms (orleans.lattice.saga.prepare.duration, orleans.lattice.saga.terminal_decision.duration, orleans.lattice.saga.broadcast.duration, orleans.lattice.saga.checkpoint.duration, orleans.lattice.saga.reminder.duration, orleans.lattice.saga.fanout.size, orleans.lattice.saga.perkey.duration) - not the WAL GC, compaction or replay histograms LatticeOptions.WalPartitions at the time of activation (Phase A attribution)
wal_max_pending_batches the WAL append-pipeline histograms (orleans.lattice.wal.append.*, orleans.lattice.wal.shard.dispatch.*) LatticeOptions.WalMaxPendingBatches at the time of activation (Phase A attribution)
pipeline_phase2 orleans.lattice.provider.commit.duration true or false, reflecting AzureTableWalStorageOptions.PipelinePhaseTwoCommits
shard_count orleans.lattice.warmup.duration Per-tree physical-shard-root probe fan-out
grain_type orleans.lattice.grain.call.outstanding_depth, orleans.lattice.grain.call.duration Target Orleans grain type name (e.g. bplusleaf)
method orleans.lattice.registry.caller.duration only ILatticeRegistry member name as declared (e.g. ResolveAsync, UpdateAsync) - the same name Orleans prints in a timeout line; other for a name not on the interface
partition the leaf replay instruments (orleans.lattice.leaf.activation_replays_over_budget, orleans.lattice.leaf.activation_stalled_replays, orleans.lattice.leaf.deferred_terminals_dropped_at_cap, orleans.lattice.leaf.unresolved_prepare_ledger_beyond_cap, orleans.lattice.wal.replay.slice_narrowings), orleans.lattice.wal.gc.blocking_pin_state, orleans.lattice.wal.gc.coverage_unknown_pin_offset, orleans.lattice.wal.gc.never_checkpointed_pin_offset, every orleans.lattice.wal.writer.* instrument, and the partition location tag on orleans.lattice.wal.saturation.transitions WAL partition ordinal, bounded by WalPartitions (plus the reserved value none on the blocking_pin_state reachability priming, and unknown on a WAL GC pin classification whose consumer id names no partition). Kept apart from shard so a future fan-out that decouples the writer partition from the destination WAL shard cannot overload shard
pin_shard orleans.lattice.materialiser.pin.reports_shed, orleans.lattice.materialiser.pin.shed_forced, orleans.lattice.materialiser.pin.shed_stall_seconds Durable leaf-materialiser pin shard: a hash of the consumer id modulo WalMaterialiserPinShards. Never join it to shard - both default to eight values, so the join looks well-formed and means nothing
dimension orleans.lattice.admission.rejected, orleans.lattice.admission.would_reject, orleans.lattice.admission.utilization keys or bytes
decision orleans.lattice.shard.healing.decisions The over-split healing sweep's decision - see that instrument
state orleans.lattice.wal.saturation.transitions, orleans.lattice.view.source_backpressure Saturation regime, lowercased: the new state (healthy, throttled or saturated) on transitions; the observed source regime (throttled or saturated) on source_backpressure
previous_state orleans.lattice.wal.saturation.transitions The regime the tree left, spelt as state
cause orleans.lattice.wal.saturation.transitions, orleans.lattice.wal.gc.scheduler_backoff, orleans.lattice.wal.gc.scheduler_consecutive_faults, orleans.lattice.wal.gc.orphan_pin_sweep (removal-decision arms only), orleans.lattice.wal.gc.drive_orphan_pin_retirement The sampler input a saturation transition was attributed to (transitions); scheduled, faulted or empty (the two scheduler instruments); the evidence a WAL GC pin-removal decision rested on - orphaned, no_durable_state or not_driven (the two orphan-pin instruments, issue #4246)
view every orleans.lattice.view.* instrument Materialised-view name
index every orleans.lattice.tag_index.reconcile.* instrument Tag-index name
activation_temperature orleans.lattice.leaf.activation_replays, orleans.lattice.leaf.activation.failures, orleans.lattice.leaf.deactivation.checkpoint_delta cold or warm
deactivation_reason orleans.lattice.leaf.deactivation.checkpoint_delta The Orleans DeactivationReasonCode name
detach_seam orleans.lattice.leaf.bisect_refusals The cache surface that released a leaf's snapshot frame - see that instrument
failure_class orleans.lattice.leaf.split_attempts (faulted arm only) unaffordable, timeout or other
origin orleans.lattice.leaf.span_fail_open_commits client_write, merge or cross_shard_migration
arm orleans.lattice.saturation.refusals, replay_permit_admission source only Which arm of the replay admission predicate refused (issue #3921): wait_exceeded (the smoothed wait of recently terminated permit waits reached WalReplayPermitMaxQueueWait), no_progress (no permit was released to the queue for WalReplayPermitMaxQueueWait) or gc_share (a background starvation drive - a WAL GC sweep's, or a leaf's own coverage-lag timer's - found the GC share of permits full or no permit immediately free)
source orleans.lattice.saturation.refusals only The admission seam that refused, as the snake_case form of the LatticeSaturationSource it carries: unspecified, wal_admission, atomic_write_saga, snapshot_cursor_open, replay_permit_admission, set_many_fan_out, set_many_envelope or tx_registry_capacity
tree_count orleans.lattice.atomic_write.cross_tree.completed Number of participating trees
version, sha orleans.lattice.build.info only Package version and full 40-character commit sha the process was built from

Leaf grain ids are not emitted as a tag - in a large tree they would produce unbounded tag cardinality. All leaf instruments are aggregated to the tree level.

The tree dimension across aliasing

A resize, a shadow-cutover restore (and its revert) and a schema remediation move a tree's data onto a new physical copy behind a registry alias. The tree tag does not follow the physical copy: every grain that serves a physical copy created to back a logical tree tags its series with that logical tree's id, read once from the copy's DerivedFrom provenance when the grain activates. A dashboard or alert filtered on tree therefore keeps those series through any number of resizes, restores and remediations, with no gap and no second series under the physical id - except the series listed under Series under the physical id below.

  • Retired copies. Maintenance that still runs on a retired or discarded copy keeps the logical owner's tree value it resolved at activation, so it is attributed to the tree the copy belonged to.
  • Independent trees. A tree registered without provenance - an ordinary tree, a standalone snapshot destination, or an entry written before provenance existed - keeps its own id. An administrative alias does not rewrite that identity: the series of a tree aliased onto an independent tree stay under the independent tree's id.
  • Gauges. The snapshot-pin, WAL saturation-state and pin-shed stall gauges fold a logical tree's copies into one series under its id: counts such as snapshot pins are summed, and the WAL saturation state reports the worst state among the copies (the materialiser pin-shed stall age likewise reports the longest run per pin shard). Per-copy state is never merged, so each copy's own WAL saturation state still governs its own writes.
  • Series under the physical id. Some series are tagged with the raw id of the physical copy rather than the resolved logical id, so after a resize, restore or remediation they appear under the copy's id. The WAL garbage collector names the WAL it collects, which is written under the physical copy: orleans.lattice.wal.entries_trimmed, the per-tree orleans.lattice.wal.gc.* instruments, the orleans.lattice.storage.policy.* series it records, and the scheduler's zero-primes of orleans.lattice.wal.replay.starvation_drive_abandonments and orleans.lattice.materialiser.pin.reports_shed (the grain records a real abandonment, and the leaf's cursor reporter a real shed, under the logical id). The WAL storage providers tag their instruments the same way: the storage-provider commit instruments, the orleans.lattice.storage.wal.* compression counters, and the file provider's orleans.lattice.wal.compactions, orleans.lattice.wal.compaction.* and orleans.lattice.wal.recovery.* instruments. On the leaf side, the gauges orleans.lattice.leaf.split.completion.in_flight, orleans.lattice.leaf.split.completion.oldest_age, orleans.lattice.wal.replay.permit_waits_in_flight, orleans.lattice.wal.replay.permit_wait.oldest_age and orleans.lattice.leaf.snapshot.segment_peak_bytes are keyed by the physical id and are not folded, the zero-primes of orleans.lattice.leaf.replay_barrier_outcomes use the physical id while its outcomes use the logical id, and the frozen-baseline replay records orleans.lattice.wal.replay.slice_narrowings under the physical id. orleans.lattice.saturation.refusals carries the refusing seam's own id, which is the physical copy's for a wal_admission or replay_permit_admission refusal, and the client-side orleans.lattice.scan.stall_resumptions and orleans.lattice.scan.stall_futility_outcomes take the physical id the stall exception carries.
  • Storage-usage and admission gauges. orleans.lattice.storage.wal_bytes, snapshot_bytes, leaf_state_bytes, total_bytes, usage_deep_published and policy.over_threshold, and the admission gauges orleans.lattice.admission.live_keys, estimated_bytes, over_advisory and utilization, are published by the tree's storage-usage aggregators under the logical id rather than under a copy's id (issue #4152). ILattice.GetStorageUsageAsync and the admission write guard address the aggregator by the logical id, which measures the live copy. The background storage-usage poller and the cluster roll-up (ILatticeAdmin.GetTotalStorageUsageAsync and RefreshStorageUsageAsync) still walk every registered tree id, the copy's included, and the copy's own aggregator measures the same shards and WAL - but it publishes under the logical id the copy was derived from, not under its own. Where both run on one silo, the copy's publish overwrites the logical tree's series instead of adding a second one. The aggregators are placed independently, though - and so are a tree's WAL-only and deep aggregators, aliased or not - so on a multi-silo cluster more than one silo can export a series under the same tree value, and a cross-silo sum by (tree) then double-counts that tree where max by (tree) does not. The one exception to the logical id is orleans.lattice.storage.policy.over_threshold, which the WAL garbage collector also sets after each byte-pressure evaluation under the id of the WAL it collects (see Series under the physical id above), so that one gauge can still carry a series under the copy's id.
  • Tenant. The derived tenant label is unchanged: a physical copy's id keeps its tree's tenant prefix, so both resolve to the same tenant.
  • System trees. _lattice_ trees are never aliased and keep their own id.

The WAL is still written under the physical tree, and a tree's mutation observers receive the logical id on the same terms.