Table of Contents

Orleans.Lattice.Api.Telemetry

A backend-neutral telemetry facade for a Lattice cluster. It answers a small, curated set of named queries over a Prometheus-compatible metrics backend, scopes every answer to the caller's tenant on the server, and refuses anything outside the curated set.

It exists because a client head - the Explorer among them - cannot be trusted to enforce tenant scoping itself - a head that derived its own tenant filter would be asking the client to police its own access. So the facade derives the scope, the head renders whatever the server pinned, and the backend is never reachable directly.

What it is not

It is not a query proxy. A caller names a query id from a catalogue; it never supplies PromQL. There is no wire field through which query text could arrive, so the discovery surface and the execution surface cannot disagree about what is runnable.

Registration

using System;
using Microsoft.Extensions.DependencyInjection;
using Orleans.Lattice.Api.Telemetry;

var services = new ServiceCollection();

services.Configure<LatticeTelemetryOptions>(options =>
{
    options.BackendAddress = new Uri("https://metrics.internal:9090");
    options.AuthMode = LatticeTelemetryBackendAuthMode.Bearer;
    options.Credential = new LatticeTelemetryBackendCredential { BearerToken = "backend-token" };

    // Fail closed: serve only the metrics named here.
    options.MetricAccess = LatticeTelemetryMetricAccessMode.DenyAllExceptAllowed;
    options.AllowedMetrics.Add("orleans_lattice_shard_*");

    // Refuse a window the backend would answer at ruinous cost.
    options.MaxRange = TimeSpan.FromHours(24);
    options.MaxStep = TimeSpan.FromMinutes(5);
});

services.AddLatticeTelemetryApi();

AddLatticeTelemetryApi() is idempotent and wires the backend client itself, so a transport binding layered on top neither repeats nor reconfigures it.

Options

Property Type Default Meaning
BackendAddress Uri? null The Prometheus-compatible endpoint. Must be absolute. Unset means no backend is configured, and every query reports as unoffered.
AuthMode LatticeTelemetryBackendAuthMode None None, Bearer, Basic, MutualTls, or DynamicBearer (a token resolved per request through ITelemetryBackendTokenProvider).
Credential LatticeTelemetryBackendCredential? null The static credential for Bearer (BearerToken), Basic (BasicUsername / BasicPassword), or MutualTls (ClientCertificate). Required for those three modes; not consulted under None or DynamicBearer.
RequestTimeout TimeSpan 30 seconds Per-request timeout against the backend. Must be strictly positive and no longer than int.MaxValue milliseconds (about 24.8 days), the longest finite timeout HttpClient accepts.
MaxRange TimeSpan 24 hours The widest window a range query may evaluate.
MaxStep TimeSpan 1 hour The coarsest step a range query may request.
MetricAccess LatticeTelemetryMetricAccessMode ReadAll ReadAll, or DenyAllExceptAllowed to serve only AllowedMetrics.
AllowedMetrics IList<string> empty The allow-list consulted under DenyAllExceptAllowed. Each entry is an exact metric name or a * wildcard pattern (for example orleans_lattice_wal_*). Matching is whole-name and case-sensitive: a wildcard entry is anchored at both ends, its * never matches a newline (so a trailing newline cannot ride past the check), and it is matched without backtracking, in time linear in the name's length. Ignored under ReadAll.

The proxy stamps the configured backend credential on every backend request and never forwards the caller's Lattice credential to it: the caller-side grant and the backend-side credential are two independent halves of the trust boundary.

AddLatticeTelemetryApi() registers no options validation - the host owns binding and validating the options. Register LatticeTelemetryOptionsValidator as an IValidateOptions<LatticeTelemetryOptions> to enforce the rules above (a supplied, absolute backend address, defined AuthMode and MetricAccess values, the credential member each static auth mode needs, a strictly positive timeout no longer than int.MaxValue milliseconds, strictly positive guardrails, and a non-empty allow-list with no blank entry under DenyAllExceptAllowed) when the options are first resolved. Under the validator an unset BackendAddress fails that resolution rather than degrading to the empty catalogue.

The allow-list is enforced on extracted names, not on the raw string

Under DenyAllExceptAllowed, every metric name a query will actually evaluate is extracted from its PromQL by PromQlMetricExtractor and checked against AllowedMetrics (each entry an exact name or a * wildcard pattern such as orleans_lattice_wal_*).

The extractor is deliberately conservative rather than a full PromQL parser: it recognises an identifier as a metric name only where one may legally appear - not when followed by (, not inside a string or a numeric/duration literal, not inside the parenthesised label list of a grouping modifier (by, without, on, ignoring, group_left, group_right), and not inside a {...} label matcher unless it is the reserved __name__ label. A keyword is skipped only where Prometheus reads it as one: Prometheus also accepts the aggregation operators, and / or / unless, by, without, offset, start, and end as a bare metric name wherever an operand is expected, so up or min evaluates the metric min, and the extractor reports it as a referenced name. Erring towards extracting more, rather than fewer, names is what keeps it fail-closed: a name it cannot resolve is refused, not admitted.

These rules are load-bearing, because the extractor and the backend must agree about what will be evaluated:

  • a # comment is discarded as whitespace exactly as Prometheus's own lexer discards it, before any string or brace state is entered. An earlier version had no rule for #, so a quote opened inside a comment was scanned as a string opener and swallowed the rest of the query - hiding a metric name from the allow-list that the backend then evaluated anyway;
  • a grouping modifier's label list is skipped with the same comment and string awareness, so an unmatched ( inside a comment (sum by (job # () or inside a quoted label (sum by ("(")) cannot stretch the skip over the rest of the query and hide the aggregated metric selector from the allow-list;
  • an exact __name__="up" matcher contributes its value as a referenced name, while a regex (__name__=~) or negative (__name__!=, __name__!~) matcher cannot be reduced to a fixed set, so it sets PromQlMetricReferences.HasUnresolvableNameMatcher and the gate fails closed. That shuts the bypass where a caller named a denied series only through __name__;
  • a top-level {...} label selector - terminated or not - that is neither anchored to a metric name nor pinned by an exact __name__ matcher, such as the right-hand side of up or {job="api"}, selects series across every metric name, so it sets PromQlMetricReferences.HasUnconstrainedSelector and the gate fails closed even when the expression also names an admitted metric. A query from which no metric name can be extracted at all is refused too.

Facade surface

ILatticeTelemetry (implemented by the public LatticeTelemetry, which AddLatticeTelemetryApi() registers) has two methods:

Method Signature
GetCatalogAsync Task<TelemetryQueryCatalog> GetCatalogAsync(CancellationToken cancellationToken = default)
QueryAsync Task<TelemetryQueryResponse> QueryAsync(TelemetryQueryRequest request, CancellationToken cancellationToken = default)

The backend proxy beneath the facade

AddLatticeTelemetryApi() builds on the public AddLatticeTelemetryBackend(), which a binding that needs the backend without the curated facade - the MCP telemetry tool group - calls directly. It is idempotent and registers:

  • TelemetryMetricAccessPolicy - the MetricAccess / AllowedMetrics posture, compiled once from the bound options (IsReadAll, and IsAdmitted(metric) for one exposition name).
  • IPrometheusQueryClient - the read-only backend client: instant query, range query, metric-name listing, and metric metadata. The default PrometheusQueryClient runs over an HttpClient bound to BackendAddress and RequestTimeout and stamps the configured backend credential; it is registered only when the host has not registered its own IPrometheusQueryClient first.

Public helpers let such a binding enforce the same rules the facade applies: TelemetryAccessAuthorizer.AuthorizeClusterTelemetryAsync runs the cluster-wide Telemetry capability check the facade opens every query with, throwing LatticeAuthorizationDeniedException when it is not granted; TelemetryQueryAuthorizer.TryAuthorizeQuery(policy, query, out denialMessage) gates a PromQL expression against the allow-list with the extractor rules above (admitting without scanning under ReadAll); and TelemetryRangeGuardrails.TryValidateRange(options, start, end, step, out violationMessage) applies the deployment-wide MaxRange / MaxStep guardrails. AddLatticeTelemetryBackend() does not register the capability check, so a binding that evaluates caller-supplied PromQL registers TelemetryAccessAuthorizer itself and runs it before any backend call, as the MCP tool group does. The allow-list only narrows what an authorized caller may read and is no substitute for the capability: its ReadAll default admits every metric.

The curated catalogue

LatticeTelemetryQueries.Definitions is the complete built-in catalogue: fifteen server-authored entries at catalogue revision LatticeTelemetryQueries.Version (1), in ascending query-id order. Each entry's TelemetryQueryDescriptor names the OpenTelemetry instruments it reads (for example orleans.lattice.shard.reads); its PromQL template reads the Prometheus exposition names the backend holds when the host publishes through the OpenTelemetry Prometheus exporter - orleans_lattice_*, with the _total suffix on counters and the unit word on histogram buckets (_milliseconds_bucket).

Query id Kind Unit Prometheus series read
tenant.quota.byte_utilization Instant 1 orleans_lattice_tenancy_usage_bytes, orleans_lattice_tenancy_quota_bytes
tenant.usage.bytes Instant By orleans_lattice_tenancy_usage_bytes
tree.admission.utilization Instant 1 orleans_lattice_admission_utilization
tree.atomic_write.outcome_rate Range {saga}/s orleans_lattice_atomic_write_completed_total
tree.cache.hit_ratio Range 1 orleans_lattice_cache_hits_total, orleans_lattice_cache_misses_total
tree.read.operation_rate Range {op}/s orleans_lattice_shard_reads_total
tree.scan.latency_p95 Range ms orleans_lattice_leaf_scan_duration_milliseconds_bucket
tree.storage.bytes Instant By orleans_lattice_storage_total_bytes
tree.storage.bytes_trend Range By orleans_lattice_storage_total_bytes
tree.tombstones.created_rate Range {tombstone}/s orleans_lattice_leaf_tombstones_created_total
tree.tombstones.reaped_rate Range {tombstone}/s orleans_lattice_leaf_tombstones_reaped_total
tree.wal.saturation_state Instant 1 orleans_lattice_wal_saturation_state
tree.write.latency_p95 Range ms orleans_lattice_leaf_write_duration_milliseconds_bucket
tree.write.operation_rate Range {op}/s orleans_lattice_shard_writes_total
tree.write.record_rate Range {record}/s orleans_lattice_shard_records_written_total

The two tenant.* entries read the tenancy add-on's orleans.lattice.tenancy meter; on a cluster without it they evaluate cleanly and return no series. Every Range entry accepts a time range, a step, and a tree filter; tree.storage.bytes, tree.admission.utilization, and tree.wal.saturation_state accept a tree filter; the tenant.* entries take no parameters. The tree filter matches the metrics' tree dimension. For every series these entries read that is the logical tree id, which stays the same across a resize, a shadow-cutover restore or a schema remediation (see The tree dimension across aliasing), so a query filtered on a tree keeps returning that tree's series after its data moves to a new physical copy. That includes the storage-usage and admission gauges behind tree.storage.bytes, tree.storage.bytes_trend and tree.admission.utilization: the background poller and the cluster storage roll-up still walk the physical copy's registered id, but the copy's aggregator publishes under the logical id too, so an unfiltered answer never lists the copy as a further tree. A tree's WAL-only and deep storage-usage aggregators are placed independently, so more than one silo can export the same tree's storage series; tree.storage.bytes and tree.storage.bytes_trend therefore take max by (tree) across silos rather than summing, which would double-count that tree. The filter is matched verbatim: the facade does not compose an unqualified name into the caller's tenant namespace, so on a tenancy cluster a tenant's tree is filtered by its full t/{tenant}/{name} id, the value its tree label carries. Each entry also declares TelemetryQueryBounds: a requested step is clamped into the entry's step budget, but a window outside the entry's bounds - or outside the deployment-wide MaxRange / MaxStep guardrails - is refused with TelemetryQueryBoundsException rather than silently narrowed.

Under DenyAllExceptAllowed, each entry's footprint - the exposition names its template reads - is checked against AllowedMetrics once, when the catalogue is built. An entry whose series are not all admitted is left out of the catalogue and is unreachable by id, indistinguishable from an unknown id, so allow-list the exposition names above (for example orleans_lattice_shard_*), not the dotted instrument names. AddLatticeTelemetryApi() registers the built-in LatticeTelemetryQueryCatalog with TryAdd, so a host that curates its own queries registers a catalogue built from its own TelemetryQueryDefinition set first.

GetCatalogAsync degrades; it does not fail

Discovery never surfaces a backend fault. An unconfigured backend, and a caller entitled to no query, both receive TelemetryQueryCatalog.Empty rather than an exception, so a client renders no panels instead of erroring - and the two cases stay indistinguishable, so a refusal leaks nothing about the deployment.

This is load-bearing for callers, and changing it would break them silently. A client may therefore treat a transport-level Unavailable from GetCatalog as the surface is unreachable, because a mere metrics-store outage cannot produce one. The Explorer relies on exactly that: its Telemetry area hides itself whenever the catalogue read fails - a refused caller, a cluster that does not serve telemetry, or one it cannot reach - while a metrics-backend outage reaches it only through a query, as a retryable error on the chart that asked.

If this method were ever changed so that a backend fault could escape it, the Explorer would begin hiding its Telemetry area during ordinary metrics outages. No test in the client would catch it, because the client's tests exercise its own classification rather than this contract. Treat the degradation as part of the published behaviour of ILatticeTelemetry, not as an implementation detail of the current backend.

Tenant scope is derived, never accepted

TelemetryTenantScopeResolver decides the effective scope from the caller's own identity. A request may state a preference; the resolver may refuse it. The response's Scope (a TelemetryTenantScope) reports what actually happened:

Field Meaning
RequestedVisibility What the caller asked for.
EffectiveVisibility What the server granted.
TenantId The tenant the answer is pinned to.
WasDowngraded The request was narrowed. A UI must say so rather than silently showing less.
IsCrossTenant The answer spans tenants, which only a platform operator can obtain.

A caller-supplied tenant id is never trusted. A non-operator asking for a different tenant is pinned to its own, and the answer is marked downgraded.

Platform-operator validation asks the registered access gate for Admin on the reserved authorization policy tree. Without the authorization add-on, the core no-op gate that AddLattice registers allows that for every caller, so a widening request is honoured; a host with no access gate registered at all never validates one, so the request degrades.

Failure surface

Three exceptions, all declared in Orleans.Lattice.Api.Abstractions so every transport binding can name them without referencing this package:

Exception Means
TelemetryQueryNotFoundException The query id is unknown or not offered by this deployment. The two are deliberately indistinguishable, so a caller learns nothing about the deployment from a refusal.
TelemetryQueryBoundsException A well-formed request whose window the entry's declared bounds or the deployment-wide MaxRange / MaxStep guardrails refuse - descending, too long, starting too far back, yielding too many points, or with a step above MaxStep (on an entry that declares no step ceiling, a step too large for the window arithmetic saturates and is refused by the MaxStep guardrail rather than overflowing). Its Violation carries the typed TelemetryBoundsViolation reason.
TelemetryBackendException The backend was unreachable, timed out, or answered unusably. Not the caller's fault.

QueryAsync can also refuse the caller. Before anything else it throws LatticeAuthorizationDeniedException when a real access gate is registered and the caller lacks the cluster-wide Telemetry capability, checked by the public TelemetryAccessAuthorizer (discovery instead degrades to the empty catalogue); once the query id and the entry's bounds pass, it throws LatticeTenantAccessDeniedException when the caller cannot be attributed to any tenant, and ArgumentException when the tree filter of an entry that accepts one contains a control character. The two Lattice refusals (LatticeAuthorizationDeniedException and LatticeTenantAccessDeniedException) are core Orleans.Lattice types, so a binding can name them too.

A binding must not forward TelemetryBackendException.Message to a remote caller. It embeds the underlying transport fault, which routinely carries the backend host and port. Log it, and answer with a fixed reason naming only what the caller already supplied.

See also