Table of Contents

Anti-entropy digest probe

Cross-cluster replication in Orleans.Lattice.Replication is eventually consistent: every mutation rides the per-tree WAL to each peer, the receiver applies it HLC-monotonically, and concurrent edits converge through the per-tree LatticeMergeMode. In the steady state every cluster eventually holds the same data for a given shard. Silent divergence - two clusters that have applied different effective state for the same shard and stay that way - should never happen, but a transport bug, a partial GC, or an operator mistake can produce it. The digest probe is the detection half of the anti-entropy pipeline: a low-frequency, read-only background pass that compares the local content digest of each shard the tree's live shard map routes to against every peer's digest and surfaces a metric when they disagree.

The probe detects divergence; it does not repair it. Localisation and repair are layered on top by later anti-entropy stages. The probe never mutates data and never advances any replication cursor.

What it compares

Every shard maintains a LeafProjectionDigest - a hash that folds the shard's content, entry count, and highest checkpoint offset, carried alongside that entry count, that checkpoint offset, and a contribution-function Version - read through the core library's ILattice.GetLeafProjectionDigestAsync(shardIndex). Each pass resolves the tree's routing afresh and walks the physical shard indices its shard map routes to, so a shard an adaptive split or a grow added above the pinned shard count is probed, and an index a shrink retired is not. A pass that cannot resolve the routing is not counted: it logs a warning and is retried on the next phase tick. The probe asks each peer for the same shard's digest over a dedicated read-only RPC and classifies the pair:

Outcome Meaning Mismatch counted?
Match Versions agree and the hashes are byte-identical - the shard's content, entry count, and highest checkpoint offset agree on both clusters. No
Mismatch Versions agree but the hashes differ - the shard's content has diverged, or the two clusters' checkpoint offsets differ. The hash folds each cluster's own WAL replay position, so a mismatch is a conservative trigger rather than proof of divergence. Yes
VersionSkew The digests carry different contribution-function Version values, so the hashes are not comparable (e.g. a rolling upgrade in flight). No
RemoteUnavailable The peer could not produce a digest (projection-digest maintenance disabled or latched off remotely). No

Only Mismatch can signal divergence, so only Mismatch increments the dedicated mismatch counter. Every comparison - including the three non-mismatch outcomes - increments the per-comparison counter tagged with its outcome, so a dashboard can distinguish a hash mismatch from a peer that simply has digesting turned off. A probe RPC that faults (for example a peer the transport cannot reach) produces no comparison and no counter increment: it is logged and retried on the next cadence.

The classification itself is exposed as the pure, stateless DigestProbeComparer:

var local = new LeafProjectionDigest
{
    Hash = new byte[] { 1, 2, 3 },
    EntryCount = 3,
    CheckpointOffset = 1,
    Version = LeafProjectionDigest.CurrentVersion,
};

var response = new DigestProbeResponse
{
    DigestAvailable = true,
    Digest = new LeafProjectionDigest
    {
        Hash = new byte[] { 9, 9, 9 },
        EntryCount = 3,
        CheckpointOffset = 1,
        Version = LeafProjectionDigest.CurrentVersion,
    },
};

DigestProbeOutcome outcome = DigestProbeComparer.Compare(local, response);
// Versions agree but the hashes differ -> Mismatch.
System.Diagnostics.Debug.Assert(outcome == DigestProbeOutcome.Mismatch);

Enabling it

The probe ships dark: DigestProbeEnabled defaults to false, so an un-opted host sees no new RPC traffic, no scheduler, and no behaviour change. Enable it per tree and tune the cadence:

siloBuilder.AddLatticeReplication(o =>
{
    o.ClusterId = "cluster-a";
    o.ReplicatedTrees = new Dictionary<string, LatticeMergeMode>
    {
        ["orders"] = LatticeMergeMode.LwwRegister,
    };

    // Anti-entropy digest probe (off by default).
    o.DigestProbeEnabled = true;
    o.DigestProbeInterval = TimeSpan.FromMinutes(5);
    o.DigestProbeJitter = 0.2;
});
Option Default Notes
DigestProbeEnabled false Per-tree master switch. When false the scheduler returns early without advancing its cadence.
DigestProbeInterval 5 minutes Base cadence between probe passes. Validated > TimeSpan.Zero.
DigestProbeJitter 0.2 Multiplicative spread applied to the interval so a fleet of silos does not probe in lockstep. Validated in [0, 1] (rejects NaN).

The cadence is deliberately low: the probe is a slow background safety net, not a hot-path check. A five-minute interval with 20% jitter spreads probe passes across a window and keeps the added RPC and digest-read cost negligible against live replication traffic.

Interaction with projection-digest maintenance

The probe respects the core library's MaintainProjectionDigest opt-out. A tree with MaintainProjectionDigest = false (including the system-tree default) has no digest to read, so the scheduler skips it - but still advances its cadence so the skip is cheap and quiet. If the local digest read throws InvalidOperationException (digest maintenance is disabled or latched off for the tree), the probe stops probing for the activation lifetime rather than retrying on every pass; any other local read fault skips only that shard for the current pass. Remotely, a peer with digesting disabled returns a response whose DigestAvailable is false, which the comparer classifies as RemoteUnavailable rather than a mismatch.

Observability

Counters on the orleans.lattice.replication meter chart cross-cluster divergence:

Metric Tags Emitted
orleans.lattice.replication.digest_probe.compared tree, shard, peer, outcome, tenant Once per shard/peer comparison, every pass.
orleans.lattice.replication.digest_probe.mismatch tree, shard, peer, tenant Only when the outcome is Mismatch.

A non-zero, sustained mismatch rate for a (tree, shard, peer) triple is the signal to localise that shard and, if you have opted in, repair it. Because the digest also folds each cluster's own checkpoint offset, a mismatch is a conservative trigger rather than proof of divergence; see Cross-cluster comparison basis. A burst of outcome=version_skew during a rolling upgrade is expected and self-clears once both sides run the same contribution-function version. A steady outcome=remote_unavailable means the peer has digesting turned off for that tree, or that no real probe transport is registered (the default no-op transport answers every probe as unavailable).

The metric name constants are exposed for dashboards that build queries from the public surface: LatticeReplicationMetrics.DigestProbeComparedName and LatticeReplicationMetrics.DigestProbeMismatchName.

Transport seam

The probe RPC travels over the existing replication push transport but is exposed through its own pluggable seam, IReplicationDigestProbeTransport, so it can be substituted independently of the live-push IReplicationTransport. The default DI registration is a no-op that lets the detection pipeline be wired up in isolation; the gRPC binding replaces it with a real implementation that invokes the probe over the same per-peer channel cache the push transport uses. A host can register its own transport before AddLatticeReplication:

public sealed class MyProbeTransport : IReplicationDigestProbeTransport
{
    public Task<DigestProbeResponse> ProbeDigestAsync(
        string targetClusterId,
        DigestProbeRequest request,
        CancellationToken cancellationToken)
    {
        // Forward the read-only probe to the peer over your own channel.
        return Task.FromResult(new DigestProbeResponse { DigestAvailable = false });
    }
}

Both the request (DigestProbeRequest) and the response (DigestProbeResponse) are immutable, Orleans-serializable value types, so they are safe to ship across the grain-proxy and cross-cluster boundaries unchanged.