Table of Contents

Metrics scraping

Part of Container quickstart.

GET /metrics serves a Prometheus text exposition (text/plain; version=0.0.4) on the same listener as MCP and the health probes, so a scraper needs no second port and no sidecar. Like the probes it is unauthenticated and always on: the listener is expected to sit on a private network, exactly as the sample compose file wires it.

The endpoint exposes every instrument published on a meter whose name starts with one of three prefixes, matched case-insensitively: orleans.lattice, which covers the core meter and every per-package meter, Orleans.Lattice.Api.Mcp.RepoContext included; Microsoft.Orleans, the Orleans runtime's own meters; and System.Runtime, the base class library's runtime meter, which carries garbage-collection, working-set, allocation, JIT, thread-pool and exception series. Instruments are selected by meter name, never by meter instance, so an instrument is exposed regardless of which type created it, and no package reference is needed for any of the three. This host's own instruments - the lattice_repocontext_* series on this page - are published on a single meter, orleans.lattice.repocontext.host, which the orleans.lattice prefix already covers: subscribe to it with AddMeter("orleans.lattice.repocontext.host") to export them elsewhere, or filter a scrape on the lattice_repocontext_ name prefix, because series are named from the instrument rather than the meter. Unlike the library's instruments they carry no tenant tag, because each describes the host process rather than any tenant's traffic. The four lattice_metrics_* self-report series below are on no meter at all - the endpoint renders them itself - so they exist only on this scrape.

The last two were added by issue #2543, and they are why a container-level memory question is now answerable from inside. The listener previously matched orleans.lattice alone, so every measurement the .NET and Orleans runtimes published was enumerated and then discarded at the subscription predicate. The effect was not a degraded runtime view but the complete absence of one: no heap size, no working set, no per-generation collection counts, no allocation rate, and nothing whatever from Orleans. The families now on the endpoint are named dotnet_* - dotnet_gc_last_collection_heap_size, dotnet_process_memory_working_set, and dotnet_gc_collections_total tagged by generation, among others - and microsoft_orleans_*.

Which prefixes a build subscribes to is itself a series, because that question could otherwise be answered only by reading the image's source:

Gauge Meaning
lattice_metrics_subscribed_meters Distinct meters that have published at least one instrument under each subscribed prefix, labelled prefix.

Read it three ways, and the third is the one it exists for. Absent means this build does not subscribe to that prefix at all, so an older image or a revert. Zero means the build subscribes and no meter has published under it, which is the honest reading on a host with no Orleans silo in it. A positive count says how many meters matched. Every prefix is minted at zero when the collector is constructed, before the listener starts, so the zero is always present and can never be read as the absence. Without that mint the two render identically, and an empty runtime section on a dashboard would be unattributable between "nothing published" and "this deployment does not collect it".

Three properties are worth knowing when reading a scrape:

  • An instrument that has never recorded a measurement still announces itself with # HELP and # TYPE lines and no samples, so "the instrument is absent" and "the instrument has not fired yet" are distinguishable from the payload alone.

  • A Histogram<T> renders as a Prometheus summary carrying _sum and _count, and no _bucket series. The listener reports raw measurements and does not surface bucket boundaries - not even the explicit boundaries orleans.lattice.wal.replay.permit_queue_wait has declared as InstrumentAdvice since issue #3044, which only an exporter that reads advice (such as the OpenTelemetry Prometheus exporter) can honour - so emitting a histogram family would mean inventing buckets and reporting invented quantiles as measurements.

    This has a consequence worth stating plainly, because it fabricates a plausible number rather than an obvious gap. A PromQL histogram_quantile over a _bucket series returns nothing here, and the common dashboard idiom of appending or vector(0) then substitutes a literal zero. The shipped OrleansLatticeCommitPath dashboard does exactly that for orleans_lattice_leaf_deactivation_checkpoint_delta, so scraped from this endpoint its p95 panel reads a flat zero - which is indistinguishable from the sustained-zero cold-arm fault shape that same dashboard tells you to look for. Read _sum and _count from this endpoint and treat any quantile panel as unavailable, not as measured. A pipeline that needs true quantiles needs a real histogram exporter, not this endpoint.

  • The endpoint self-reports its own limits, so a truncated scrape says so rather than reading as a quiet zero:

    Instrument Meaning
    lattice_metrics_series Live series currently held by the collector.
    lattice_metrics_dropped_measurements_total Measurements dropped since start because a ceiling was reached.
    lattice_metrics_dropped_measurements_by_family_total The same drops attributed to the metric family that caused them, labelled family, and to the ceiling that refused them, labelled ceiling (family or global).

    The counters say that a ceiling was reached, but not when. That instant is what decides which historical absences can be read: before it, an absent series is a real absence, and after it, the series may simply have been refused. So the collector also logs the transition itself (issue #2519). The first time each ceiling refuses a series, it writes exactly one Warning with event id 1 / MetricsCeilingReached under the Orleans.Lattice.Api.Mcp.RepoContext.Host.RepoContextMetricsCollector category. The record carries these fields:

    • Ceiling: family or global.
    • Limit: the ceiling's value.
    • SaturatedAtUtc: the instant of the first refusal.
    • Family: the refused family.
    • FamilySeries and TotalSeries: the series counts at that moment.

    The per-family ceiling announces once per family, and the global backstop announces once per process. A crossing that happens during startup, before logging is wired up, is written as soon as the logger is attached, still stamped with its original instant. To find the boundary, search the log for MetricsCeilingReached: absences in that family (or in every family, for global) are unambiguous only before its SaturatedAtUtc.