Table of Contents

Metric-to-panel coverage map

Almost every instrument on the orleans.lattice and orleans.lattice.replication meters is referenced by at least one panel in the bundled dashboards; the exceptions are listed as not charted in the tables below. The drift-guard test in Orleans.Lattice.Dashboards.Tests enforces the inverse direction unconditionally: every metric name a dashboard references must resolve to an instrument declared in source.

The forward direction is enforced only for instruments the guard can observe. The guard discovers live instruments by forcing the type initialisers of LatticeMetrics and LatticeReplicationMetrics and listening for what they publish, plus the instrument-name constants those two classes declare. An instrument created in a field initialiser on some other type - a grain, say - is never constructed at test time, so the guard neither sees it nor demands a panel for it. Such an instrument can ship unpaneled with a green build, which is exactly how the orleans.lattice.tag_index.reconcile.* family below came to be uncharted. When you add an instrument outside those two classes, add its panel by hand - nothing in this guard demands one - although the documentation-coverage fixtures (the MetricsDocCoverageTestsBase subclasses), which scan source for instrument-name literals rather than listening to the meter, still fail the build until it has a row here.

The add-on orleans.lattice.auth and orleans.lattice.membership meters are charted by the bundled Identity & Authorization dashboard. Their coverage is enforced from the owning packages: Orleans.Lattice.Auth.Tests and Orleans.Lattice.Membership.Tests each derive from the shared MeterDashboardCoverageTestsBase, which asserts every instrument on the meter is referenced by that dashboard (and that every token the dashboard references for the meter resolves to a live instrument).

The add-on orleans.lattice.backup meter is charted by the bundled Backup & Restore dashboard. Its coverage is enforced the same way, from Orleans.Lattice.Backup.Tests (deriving from MeterDashboardCoverageTestsBase).

The add-on orleans.lattice.scaling meter is charted by the bundled Autoscaling Signal dashboard. Its coverage is enforced the same way, from Orleans.Lattice.Scaling.Tests (deriving from MeterDashboardCoverageTestsBase).

The add-on orleans.lattice.replication.grpc meter is charted by the bundled Replication Transport (gRPC) dashboard. Its coverage is enforced the same way, from Orleans.Lattice.Replication.Grpc.Tests (deriving from MeterDashboardCoverageTestsBase).

The add-on orleans.lattice.tenancy meter is charted by the bundled Per-Tenant Observability dashboard. Its coverage is enforced the same way, from Orleans.Lattice.Tenancy.Tests (deriving from MeterDashboardCoverageTestsBase).

The orleans.lattice.grainindex.* rows below sit on the core meter but are declared on the grain-index package's own GrainIndexMetrics type, so the core guard above cannot observe them. Their coverage is enforced from Orleans.Lattice.GrainIndex.Tests instead, whose MeterDashboardCoverageTestsBase subclass is narrowed to the orleans.lattice.grainindex.* family and asserts that the Grain Index dashboard charts each of them.

How an instrument name becomes a PromQL series name

The family name depends on which exposition serves the scrape, and the two expositions in this repository disagree about units. The bundled dashboards target a host exporting through .AddPrometheusExporter() - the OpenTelemetry OpenTelemetry.Exporter.Prometheus.AspNetCore package, pinned at 1.15.3-beta.1 by every host in this repository. That exporter maps the declared unit to a word and appends it to the family name, ahead of any type suffix, unless the name already ends with that word: ms becomes _milliseconds, s becomes _seconds, By becomes _bytes and % becomes _percent, while 1 and an annotation unit such as {entry} contribute nothing. A monotonic counter then gains _total; an up-down counter is exported as a gauge and gains nothing.

The repository-context container does not use that exporter. It serves its own Prometheus exposition, which appends no unit segment, records the unit in the HELP text instead, and renders every histogram as a summary carrying only _sum and _count (see the container guide):

# HELP orleans_lattice_storage_policy_bytes_reclaimed_total WAL occupancy bytes freed by GC passes on trees with an advisory byte ceiling, tagged by tree. (unit: By)
# TYPE orleans_lattice_storage_policy_bytes_reclaimed_total counter

That sample is the container's: the trailing (unit: By) is its own HELP formatting, and the instrument's description ends at tagged by tree.. A scrape of 440 container families carried zero appended unit suffixes. Under .AddPrometheusExporter() the same instrument, orleans.lattice.storage.policy.bytes_reclaimed with unit By, is exported as orleans_lattice_storage_policy_bytes_reclaimed_bytes_total, because its name does not end in bytes.

So the family name for an instrument is:

instrument kind .AddPrometheusExporter() repository-context container
counter dotted name with . replaced by _, plus the unit word when the name does not already end with it, plus _total dotted name with . replaced by _, plus _total
up-down counter / gauge / observable gauge dotted name with . replaced by _, plus the unit word when the name does not already end with it dotted name with . replaced by _
histogram dotted name with . replaced by _, plus the unit word when the name does not already end with it, plus _bucket / _count / _sum dotted name with . replaced by _, plus _count / _sum only

An instrument whose name already ends in its unit word (such as orleans.lattice.storage.wal.stored_bytes) therefore yields orleans_lattice_storage_wal_stored_bytes_total on both expositions - the exporter does not double the word. Writing ..._stored_bytes_bytes_total queries a series that cannot exist, and a panel querying a non-existent series renders as an empty graph, which is visually identical to a real zero.

The drift guard accepts only the .AddPrometheusExporter() spelling. AddInstrumentForms in DashboardJsonTests and MeterDashboardCoverageTestsBase derive the single family name that exporter emits from each instrument's declared unit and kind, using the shared PrometheusExporterNaming model of the rules above (issue #3260). No other spelling resolves: neither the container's unsuffixed one, nor a unit word the instrument's unit does not imply. So a By-unit counter whose name does not end in bytes is queried as, for example, orleans_lattice_storage_policy_bytes_reclaimed_bytes_total; an s-unit gauge as orleans_lattice_leaf_split_completion_oldest_age_seconds; the s-unit histogram orleans.lattice.wal.gc.scheduler.wait as orleans_lattice_wal_gc_scheduler_wait_seconds_sum / _count; and the %-unit gauge orleans.lattice.grainindex.backfill.percent_complete as orleans_lattice_grainindex_backfill_percent_complete_percent, because its name does not end in percent. DashboardBucketUnitSuffixTests separately requires every _bucket token to carry the suffix its instrument's unit implies. None of these spellings match anything on the container's exposition, where a histogram also has no _bucket series at all. Issue #3261 records why that split is deliberate.

How to read the Tags column

Each table's Tags column lists every dimension the instrument's series actually carries, including the derived tenant label. That label is present on every instrument on every meter, and what its values mean is described once in The derived tenant label below.

It is spelled out on every row rather than factored out, because the column documents a series identity: a row that omits tenant describes an identity narrower than the real one, and a query built from that row matches nothing. A platform-scoped instrument is not an exception - it carries tenant with the reserved _platform_ value, so the label is still part of its identity. MetricDocTenantDimensionTests.EveryInstrumentRowDocumentsTheTenantDimension enforces this, so a row that leaves tenant out fails the build.

Per-operation vs per-record contract

A throughput-style counter measures either operations or records, and the two diverge sharply on batched and bulk paths. Every such instrument below declares which it is, so a panel title can never imply a semantic the instrument does not deliver:

  • Per-operation ({op}) - one increment per grain call, whatever its payload. A batched or bulk call (SetManyAsync, DeleteRangeAsync, SetManyWherePredicateAsync, BulkLoadAsync, and the internal per-shard batches behind bulk load, replication apply, backup restore, tree snapshots and shard-split migration) counts once regardless of entry count, so a 5000-record import advances the counter by only the number of bulk operations.
  • Per-record ({record}) - one increment per individual entry the operation carried. The same 5000-record import advances the counter by 5000.
  • Per-entry sample - for a histogram, one observation per entry rather than per batch (for example orleans.lattice.replication.apply.duration, which contributes N samples for a batch of N).

orleans.lattice.shard.writes (operations) and orleans.lattice.shard.records_written (records) are the canonical pair: plot both, and their ratio is the effective batch size. Plotting the operation counter alone as "write throughput" under-represents bulk ingestion, which is the defect issue #1648 was raised for.

Contents

orleans.lattice.auth meter

Charted by the Identity & Authorization dashboard.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.auth.decisions counter ({decision}) operation, tree, effect, tenant Authorization Authorization decisions (rate by effect); Decisions by operation (rate)
orleans.lattice.auth.decision.duration histogram (ms) operation, tree, effect, tenant Authorization Decision latency p50/p95/p99 (ms)
orleans.lattice.auth.snapshot.rebuilds counter ({rebuild}) tenant Authorization Compiled snapshot rebuilds (rate)
orleans.lattice.auth.snapshot.epoch observable gauge ({epoch}) tenant Authorization Compiled snapshot epoch (one series per silo process, which the panel separates by the scrape target's instance label)
orleans.lattice.auth.snapshot.age observable gauge (s) tenant Authorization Compiled snapshot age (one series per silo process, which the panel separates by the scrape target's instance label)
orleans.lattice.auth.snapshot.subjects observable gauge ({subject}) tenant Authorization Members with policies configured (one series per silo process, which the panel separates by the scrape target's instance label)

orleans.lattice.membership meter

Charted by the Identity & Authorization dashboard.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.membership.resolution_cache.hits counter ({lookup}) tenant Authorization Subject-resolution cache hit ratio; Subject-resolution cache hits vs misses (rate)
orleans.lattice.membership.resolution_cache.misses counter ({lookup}) tenant Authorization Subject-resolution cache hit ratio; Subject-resolution cache hits vs misses (rate)
orleans.lattice.membership.directory.search.duration histogram (ms) tenant Authorization Identity-directory search latency p50/p95/p99 (ms)
orleans.lattice.membership.directory.search.hits counter ({search}) tenant Authorization Identity-directory search hits vs misses (rate); Identity-directory search hit ratio
orleans.lattice.membership.directory.search.misses counter ({search}) tenant Authorization Identity-directory search hits vs misses (rate); Identity-directory search hit ratio

orleans.lattice.backup meter

Charted by the Backup & Restore dashboard.

That dashboard's scope selector takes its values from orleans.lattice.backup.scope.last_run_status, whose series appears for a scope when its schedule is registered, when a capture of it succeeds, or when a scheduled cycle for it faults. The selector narrows only the panels over scope-tagged instruments (retention and scheduler); the capture, size, processing, restore, incremental-lag and retry instruments carry no scope tag, so their panels apply no scope matcher and chart every scope together.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.backup.captures counter ({backup}) kind, tenant Backup Captures (rate by kind)
orleans.lattice.backup.capture.duration histogram (ms) kind, tenant Backup Capture duration p50/p95/p99
orleans.lattice.backup.bytes histogram (By) kind, tenant Backup Backup size p50/p95 (bytes)
orleans.lattice.backup.artifacts histogram ({artifact}) kind, tenant Backup Artifacts per backup p50/p95
orleans.lattice.backup.entries histogram ({entry}) kind, tenant Backup Entries per backup p50/p95
orleans.lattice.backup.entries_processed counter ({entry}) kind, tenant Backup Processing throughput
orleans.lattice.backup.bytes_processed counter (By) kind, tenant Backup Processing throughput
orleans.lattice.backup.restore.duration histogram (ms) tenant Backup Restore duration p50/p95/p99
orleans.lattice.backup.restore.entries counter ({entry}) tenant Backup Restore entries (rate)
orleans.lattice.backup.incremental.lag_entries histogram ({entry}) tenant Backup Incremental lag entries p50/p95
orleans.lattice.backup.incremental.lag_age histogram (ms) tenant Backup Incremental lag age p50/p95
orleans.lattice.backup.retention.bytes_reclaimed counter (By) scope, tenant Backup Retention reclaimed
orleans.lattice.backup.retention.pruned counter ({backup}) scope, tenant Backup Retention pruned (rate)
orleans.lattice.backup.capture.failures counter ({failure}) kind, phase, reason, tenant Backup Capture failures (by reason)
orleans.lattice.backup.restore.failures counter ({failure}) phase, reason, tenant Backup Restore failures (by reason)
orleans.lattice.backup.capture.retries counter ({retry}) reason, tenant Backup Capture retries / fallbacks
orleans.lattice.backup.scheduler.skipped counter ({run}) scope, tenant Backup Scheduler skipped vs overruns
orleans.lattice.backup.scheduler.overruns counter ({run}) scope, tenant Backup Scheduler skipped vs overruns
orleans.lattice.backup.scheduler.failures counter ({run}) scope, reason, tenant Backup Scheduler capture failures by reason
orleans.lattice.backup.cross_tree_fence.selections counter ({fence}) tree_count, tenant Backup Cross-tree fence selections / drained
orleans.lattice.backup.cross_tree_fence.drained_in_flight counter ({saga}) tenant Backup Cross-tree fence selections / drained
orleans.lattice.backup.cross_tree_fence.retries counter ({retry}) tenant Backup Cross-tree fence retries
orleans.lattice.backup.cross_tree_fence.drain_wait histogram (ms) tenant Backup Cross-tree fence drain wait p50/p95
orleans.lattice.backup.inventory.count observable gauge ({backup}) tenant Backup Tracked backups
orleans.lattice.backup.inventory.chain_depth_max observable gauge ({backup}) tenant Backup Max chain depth
orleans.lattice.backup.catalog.bytes observable gauge (By) tenant Backup Catalog size
orleans.lattice.backup.inventory.oldest_age observable gauge (s) tenant Backup Oldest backup age
orleans.lattice.backup.inventory.newest_age observable gauge (s) tenant Backup Newest backup age
orleans.lattice.backup.scope.last_run_status observable gauge ({status}) scope, tenant Backup Per-scope last-run status
orleans.lattice.backup.scope.last_success_age observable gauge (s) scope, tenant Backup Per-scope seconds since last success

orleans.lattice.scaling meter

All instruments are observable gauges published from the cached ScalingSignal on the silo's sampling timer. Charted by the Autoscaling Signal dashboard; coverage enforced from Orleans.Lattice.Scaling.Tests.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.scaling.scale_value observable gauge ({replica}) tenant Autoscaling Signal Scale value (smoothed vs raw)
orleans.lattice.scaling.raw_scale_value observable gauge ({replica}) tenant Autoscaling Signal Scale value (smoothed vs raw)
orleans.lattice.scaling.compute.activation_pressure observable gauge (1) tenant Autoscaling Signal Compute pressure by dimension
orleans.lattice.scaling.compute.resource_pressure observable gauge (1) tenant Autoscaling Signal Compute pressure by dimension
orleans.lattice.scaling.compute.wal_dispatch_pressure observable gauge (1) tenant Autoscaling Signal Compute pressure by dimension
orleans.lattice.scaling.compute.replicas observable gauge ({replica}) tenant Autoscaling Signal Recommended replicas
orleans.lattice.scaling.storage.accounts_over_threshold observable gauge ({account}) tenant Autoscaling Signal WAL accounts over threshold
orleans.lattice.scaling.storage.rebalance_recommendations observable gauge (1) tenant Autoscaling Signal WAL rebalance recommended

orleans.lattice.replication.grpc meter

The gRPC replication transport's telemetry. Charted by the Replication Transport (gRPC) dashboard; coverage enforced from Orleans.Lattice.Replication.Grpc.Tests.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.replication.grpc.insecure_channel counter ({channel}) peer, transport, tenant Replication Transport (gRPC) Insecure (plaintext) channels constructed; Insecure channel construction rate by peer and transport

orleans.lattice.tenancy meter

Per-tenant usage, quota, burst, and metered-overage telemetry published by the opt-in lattice.tenancy add-on. Charted by the Per-Tenant Observability dashboard; coverage enforced from Orleans.Lattice.Tenancy.Tests.

Every instrument is an observable gauge published on a fixed cadence (TenantObservabilityOptions.PublishInterval, default 30 seconds) from the last landed metering sample, so these are periodic samples rather than live readings. Every series carries a tenant tag: the per-tenant series name the tenant they measure, and the cluster-aggregate tenant count carries the reserved _platform_ value. A quota.* gauge emits a measurement only for a bounded dimension - an unbounded ceiling contributes no series at all, so "no series" reads as "unlimited", not "zero". The overage.* gauges are grow-only converged sums, not instantaneous readings.

MaxOpsPerSecond has no gauge: the rate budget is enforced from silo-local token buckets rather than a published aggregate, so a breach surfaces as an ops-per-second LatticeQuotaExceededException rather than a series.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.tenancy.tenants observable gauge ({tenant}) tenant = _platform_ (cluster aggregate) Per-Tenant Observability Registered tenants
orleans.lattice.tenancy.usage.bytes observable gauge (By) tenant Per-Tenant Observability Stored bytes by tenant
orleans.lattice.tenancy.quota.bytes observable gauge (By) tenant Per-Tenant Observability Stored bytes by tenant (quota overlay)
orleans.lattice.tenancy.usage.keys observable gauge ({key}) tenant Per-Tenant Observability Live keys by tenant
orleans.lattice.tenancy.quota.keys observable gauge ({key}) tenant Per-Tenant Observability Live keys by tenant (quota overlay)
orleans.lattice.tenancy.usage.memory_bytes observable gauge (By) tenant Per-Tenant Observability Resident memory by tenant
orleans.lattice.tenancy.quota.memory_bytes observable gauge (By) tenant Per-Tenant Observability Resident memory by tenant (quota overlay)
orleans.lattice.tenancy.usage.trees observable gauge ({tree}) tenant Per-Tenant Observability Owned trees by tenant
orleans.lattice.tenancy.quota.trees observable gauge ({tree}) tenant Per-Tenant Observability Owned trees by tenant (quota overlay)
orleans.lattice.tenancy.quota.burst_percent observable gauge (%) tenant Per-Tenant Observability Burst headroom by tenant
orleans.lattice.tenancy.overage.bytes observable gauge (By) tenant Per-Tenant Observability Metered byte overage by tenant
orleans.lattice.tenancy.overage.keys observable gauge ({key}) tenant Per-Tenant Observability Metered overage (keys / memory / trees) by tenant
orleans.lattice.tenancy.overage.memory_bytes observable gauge (By) tenant Per-Tenant Observability Metered overage (keys / memory / trees) by tenant
orleans.lattice.tenancy.overage.trees observable gauge ({tree}) tenant Per-Tenant Observability Metered overage (keys / memory / trees) by tenant

orleans.lattice.api.mcp meter

The MCP server host's own telemetry, published by Orleans.Lattice.Api.Mcp (LatticeApiMcpMetrics). No bundled dashboard charts it, a recorded decision in MeterDashboardCoverageEnrolmentTests; ApiMcpMetricsDocCoverageTests in test/lattice.api.mcp holds this row in place. A tool call rejected as the caller's mistake is answered with an MCP error result instead of being thrown, and the MCP host logs it at Debug without a stack, because the ModelContextProtocol SDK logs every thrown tool exception at Error with its stack; this counter is where that rate stays visible (issue #3761). The Debug line is only the host's own record: the repository-context tools also log every call that reaches one of them and fails at Warning with its exception, a classified client error included, so a caller mistake on a repocontext_* tool still leaves a Warning line with a stack beside it. An undeclared argument, a refused region and a refusal by the registered ILatticeApiMcpAuthorizer are turned away before the tool runs, so they leave no such line. Server faults, faults a tool does not classify as a client error, authorization denials and region refusals are still thrown and still log at Error. Not primed: an absent series means no client error of that tool and reason has happened in this process.

Instrument Type Tags Dashboard Panel(s)
orleans.lattice.api.mcp.tool.client_errors counter ({error}) tool; reason = invalid_argument, unknown_argument, rejected_content, not_found; tenant = _platform_ (none) not charted