Table of Contents

Tag indexes

Part of Lattice Public API Reference.

A tag index associates string tags with the keys of a tree and lets you query keys back by tag. It is built entirely on the public ILattice surface: tag-membership rows live in a sibling ordinary Lattice tree resolved as tag-{indexName}, each keyed tag \0 treeId \0 key with a flag value. No standalone grain is introduced.

Open an index through the injected ILatticeTagIndexFactory (registered by AddLattice). The factory sources the membership convergence mode and the local replica id from the host's replication configuration, so the same code runs single-cluster (last-writer-wins) and active-active (flag-CRDT). The subject tree supplied to Create stays the receiver - it supplies the tree segment of every membership row.

using Microsoft.Extensions.DependencyInjection;

// Inject ILatticeTagIndexFactory wherever you need an index; here it is resolved
// from the client's service provider for illustration.
var tagIndexFactory = client.ServiceProvider.GetRequiredService<ILatticeTagIndexFactory>();
var tagIndex = tagIndexFactory.Create(tree, "by-color");

// Per-key tag CRUD. SetAsync replaces (read-modify-write diff); AddAsync /
// RemoveAsync are additive / subtractive.
await tagIndex.Key("item:1").AddAsync(["red", "round"], cancellationToken);
_ = await tagIndex.Key("item:1").GetAsync(cancellationToken);

// Intersection (every tag) and de-duplicated union (any tag). Each query is a
// lazy IAsyncEnumerable<string> with CountAsync.
await foreach (var key in tagIndex.WithAllTags("red", "round").WithCancellation(cancellationToken))
{
    _ = key;
}
_ = await tagIndex.WithAnyTags("red", "blue").CountAsync(cancellationToken);

// Enumerate the distinct tags that currently have at least one member key.
await foreach (var tag in tagIndex.TagsAsync(cancellationToken))
{
    _ = tag;
}

// Write a value and its tags together. Eventual by default (two independent
// durable writes); .Atomic() lowers to the cross-tree atomic-write saga so the
// value and its tag rows become visible together.
await tagIndex.SetValueWithTags("item:2", "payload"u8.ToArray(), "blue")
    .Atomic()
    .CommitAsync(cancellationToken);

// On-demand reconcile removes membership rows whose key no longer exists in the
// primary tree, over an optional key range. Idempotent.
TagReconcileReport report = await tagIndex.ReconcileAsync(cancellationToken: cancellationToken);
_ = report.OrphanRowsRemoved;

// The multi-tree view spans every covered tree, yielding TaggedKey, and
// supports InTree(treeId) narrowing.
var multi = tagIndexFactory.CreateMultiTree("by-color");
await foreach (var hit in multi.WithAnyTags("red").WithCancellation(cancellationToken))
{
    _ = (hit.TreeId, hit.Key);
}

Surface

Type Role
ILatticeTagIndexFactory The injected entry point for opening a tag index (Create(tree, indexName) / CreateMultiTree(indexName, allowedTrees?)). Pre-wires the index to the host's replication configuration, so the same call runs single-cluster (last-writer-wins) and active-active (flag-CRDT). Registered as a singleton by AddLattice.
ILatticeTagIndex Single-tree surface: IndexName, TreeId, Key, WithAllTags, WithAnyTags, SetValueWithTags, TagsAsync, ReconcileAsync, MultiTree.
ILatticeKeyTags Per-key tag surface: GetAsync, SetAsync (replace), AddAsync, RemoveAsync.
ILatticeTagQuery Lazy IAsyncEnumerable<string> of matching keys with CountAsync.
ILatticeValueTagWrite Staged value+tags write: Atomic() / Eventual() / CommitAsync().
ILatticeMultiTreeTagIndex Multi-tree view: IndexName, Tree(treeId), WithAllTags, WithAnyTags, TagsAsync, CoveredTreesAsync, ReconcileAsync.
ILatticeMultiTreeTagQuery Lazy IAsyncEnumerable<TaggedKey> with InTree(treeId) narrowing and CountAsync.
TaggedKey (TreeId, Key) pair yielded by multi-tree queries.
TagReconcileReport Counts from a reconcile pass: TreesCovered, KeysScanned, MembershipRowsScanned, OrphanRowsRemoved; Empty and Combine(other) aggregate passes.
TagConsistency Eventual (default) or Atomic durability coupling for SetValueWithTags.
ILatticeReplicationContext Injectable replication-configuration seam (root Orleans.Lattice namespace). Exposes IsReplicationEnabled, the LocalReplicaId, and ResolveMergeMode(treeId). The ILatticeTagIndexFactory captures it to select flag-CRDT membership; the core default reports replication disabled, and AddLatticeReplication swaps in a configured implementation sourced from LatticeReplicationOptions.

Notes

  • Producer-side acceptance. In the default open mode a subject tree must already be registered - which happens on its first use, reads included - before its keys can be tagged; supplying a closed allowedTrees allowlist to ILatticeTagIndexFactory.CreateMultiTree restricts membership writes to the listed trees. A single-tree index only ever writes to its bound subject tree, so it has no allowlist. Acceptance is validated on the write path only - never as an apply-time gate.
  • Additive SetValueWithTags. It associates the supplied tags with the key; it does not remove previously-associated tags. Use Key(key).SetAsync(tags) for replace semantics.
  • Active-active replication. By default membership rows are last-writer-wins - correct and lossless for single-writer-per-key, add-mostly indexes, but a concurrent add/remove of the same row across clusters resolves by clock and can drop the add. For convergent membership under concurrent writes from multiple clusters, the same tagIndexFactory.Create(tree, indexName) call authors flag-CRDT membership: the factory captures the ILatticeReplicationContext seam, which is the single source of truth for both the membership mode (resolved per index tree from LatticeReplicationOptions.ReplicatedTrees) and the local replica id, so the caller never threads a membershipMode / replicaId pair that could drift from server config. Declaring the index tree (tag-{indexName}) under OrFlag (enable-wins, the recommended default) or RwFlag (remove-wins, for revocation / blocklist facets where a removal must win the tie) selects the mode; the core default seam reports replication disabled and falls back to the LWW path. A flag mode authors every membership, key-major mirror, and covered-marker row as a typed flag-CRDT delta (an enable on add, a disable on remove) so the shipped WAL record carries a real OrFlagDelta / RwFlagDelta, and every read decodes flag state so a disabled / tombstoned row reads as absent. The index tree (tag-{indexName}) must be declared with the matching LatticeMergeMode in LatticeReplicationOptions.ReplicatedTrees; the background reconciliation coordinator auto-detects the same mode server-side from the replication-configuration seam, so its orphan cleanup also authors flag disables. Under a flag mode, SetValueWithTags(...).Atomic() honours the all-or-nothing coupling between the value and its tag rows: the cross-tree saga stages each membership row with a freshly minted flag-enable delta (and stores the merged flag state as the row value), so the value and its membership rows become visible together and every other cluster converges each row by replaying the author's enable dot (enable-wins under OrFlag, remove-wins under RwFlag). The value write on the subject tree stays a plain last-writer-wins set. Flag membership reads scan and decode row values (the LWW path scans keys only), the documented cost of convergent membership.
  • Reserved characters. Neither a tag nor a tree id may contain the NUL (\0) separator. The covered-tree set surfaced by the multi-tree view is an over-approximating set of idempotent marker rows on the index tree, self-healing from a full scan when absent.
  • Query cost. Both directions are bounded prefix scans: tag-to-keys (WithAllTags / WithAnyTags) over the tag-major rows, and key-to-tags (Key(key).GetAsync() / Key(key).SetAsync(...)) over a key-major mirror row maintained alongside each membership. The mirror roughly doubles membership write and storage cost (the usual both-directions index trade-off). TagsAsync() and ReconcileAsync() still scan the whole index tree, since enumerating every distinct tag and visiting every row are inherently full passes - reserve them for occasional maintenance.

Background reconciliation

ReconcileAsync is also driven automatically in the background, so an index stays clean without an operator scheduling a pass. The first time a tree is covered by an index, a per-index coordinator (keyed by {indexName}, built on the shared crash-/restart-safe coordinator machinery) registers a recurring schedule reminder and, on each firing, runs a digest-gated sweep.

Orphan membership rows arise only from key deletions in a covered tree, and every such deletion folds a tombstone into that tree's LeafProjectionDigest. The sweep exploits this: it folds each covered tree's per-shard leaf-projection digests into a fingerprint and compares it to the baseline captured at the last successful reconcile. Trees whose fingerprint is unchanged are skipped with digest reads only - no scans, no writes - so a clean index incurs only digest-probe cost. Only trees whose digest diverged (or whose digest is unavailable) are deep-scanned and repaired through the same ReconcileAsync path, after which their baseline advances.

Gating is tree-granular rather than sub-shard: a divergent tree is reconciled in full. The Merkle-walk machinery that would narrow a dirty tree to a key sub-range lives in the replication package and is not referenced by the core library, so it is intentionally not used here.

Tune reconciliation per index with LatticeTagIndexReconciliationOptions, resolved via IOptionsMonitor<LatticeTagIndexReconciliationOptions>.Get(indexName) (the DefaultInterval, MinimumInterval, and DefaultChunkSize constants hold the defaults and the interval floor below):

Option Default Meaning
Enabled true Whether the background coordinator runs. Setting it false unregisters any existing schedule.
Interval 1h Cadence between sweeps. Floored at the 1-minute Orleans reminder minimum.
ChunkSize 16 Covered trees probed per phase tick, bounding per-activation work.
ProbeOnly false When true, the sweep detects and reports dirty trees but never repairs them or advances the baseline. 
siloBuilder.ConfigureLatticeTagIndexReconciliation("by-color", o =>
{
    o.Interval = TimeSpan.FromHours(6);
    o.ProbeOnly = false;
});

The digest probe reads each covered tree's LeafProjectionDigest, which requires LatticeOptions.MaintainProjectionDigest (enabled by default) on the covered trees.