Table of Contents

Chaos Tests

Orleans.Lattice ships a suite of integration tests that bombard a running cluster (single-cluster and multi-site) with concurrent reads, writes, scans, atomic-write sagas, topology mutations, and inter-site network partitions, then assert that the system's public correctness guarantees hold. They act as the end-to-end contract for the properties described in Consistency and Replication - specifically that the public ILattice API is strongly consistent across arbitrary concurrent shard splits, online resizes, and online reshards; that point-mutation public-API calls (DeleteRangeAsync, SetIfVersionAsync (the public compare-and-swap entry point), ScanKeysAsync / ScanEntriesAsync cancellation) hold their stated invariants under concurrent contention; that SetManyAtomicAsync remains atomically visible (zero-or-all keys per poll) on the authoring site and on every receiver site; that SetManyAtomicAsync keeps a multi-tree saga's keys all-or-nothing across every participating tree under concurrent shard splits; and that the per-merge-mode CRDT dispatch paths converge across partitioned sites. The single-cluster suite also exercises the recovery protocols (resumable splits, two-phase root promotion, shadow-write atomicity, shadow-forwarding, and registry version stamping) under random storage-write faults. The replication suite extends those guarantees to the production shipper, WAL trim, per-peer liveness, tombstone-reap filtering, and the gRPC transport; the Azure Table WAL suite pins append-batch atomicity and offset monotonicity against the local Azurite emulator. The lattice.schema companion suite additionally pins that SetManyAtomicAsync stays all-or-nothing while a tree's schema enforcement policy or target schema version is changed concurrently, and that every stored value stays self-describing and decodable across a version advance and an eager background migration.

Every fixture is tagged [Category("Chaos")], so any test filter carrying TestCategory!=Chaos - the Tier 2 filter of the tiered test workflow among them - skips them. Most are also marked [NonParallelizable] so each has its cluster to itself; AdmissionControlChaosTests, BoundedCacheEvictionChaosTests, ClusterSplitConcurrencyChaosTests, ShardConsolidationChaosTests, LatticePredicatePushdownChaosTests and the replication suite's AntiEntropyRemediationGuardChaosTests are not.

Core chaos suite (test/lattice/BPlusTree/, plus one fixture in test/lattice/Predicates/)

Test class File Purpose
Happy-path chaos ChaosIntegrationTests.cs Strong invariants during heavy concurrent load with manually-triggered splits.
Chaos + storage faults ChaosWithFaultsIntegrationTests.cs Parametrized theory that injects random storage faults; asserts eventual convergence after the fault window closes.
Chaos + online resize ChaosResizeIntegrationTests.cs Full-workload chaos while ResizeAsync changes fan-out in the background under SnapshotMode.Online. Exercises the TreeResizeGrain phase machine (Snapshot → Swap → Reject → Cleanup), shadow-forwarding on every source shard, and the alias swap.
Chaos + online reshard ChaosReshardIntegrationTests.cs Full-workload chaos while ReshardAsync grows the physical shard count (4 to 8) in the background. Exercises the reshard coordinator's migration loop and its dispatch-budget clamping (MaxConcurrentMigrations), and ShardMap convergence when reshard-dispatched splits race with workload writes. The same fixture also shrinks a tree (4 to 2) under the same workload, exercising fold planning, every fold's drain, freeze, swap and finalise racing live writes, and the release of each retired donor's storage while scans and counts may still hold a pre-fold shard map.
Atomic-write reader isolation AtomicVisibilityChaosTests.cs Strict reader isolation: a continuous reader concurrent with SetManyAtomicAsync always observes either the full pre-saga snapshot, the full post-saga snapshot, or all keys hidden - never a partial view. Quiescent topology (no concurrent split/resize/reshard).
Atomic-write reader isolation across shard split ShardSplitTopologyTests.cs Same zero-or-all visibility invariant as AtomicVisibilityChaosTests, but the topology mutator drives a manual SplitAsync on shard 0 concurrently with a chain of SetManyAtomicAsync sagas. Exercises shadow-forward of saga prepares onto the destination shard and the saga terminal-broadcast retry onto the new owner via StaleShardRoutingException.
Atomic-write reader isolation across online resize ResizeTopologyTests.cs Same zero-or-all visibility invariant, but the topology mutator runs an online ResizeAsync (MaxLeafKeys / MaxInternalChildren to 8) concurrently with the saga chain. Exercises shadow-forwarding from the source physical tree to the destination, the alias swap, and the saga terminal-broadcast retry onto the new owner via StaleTreeRoutingException.
Atomic-write reader isolation across online reshard ReshardTopologyTests.cs Same zero-or-all visibility invariant, but the topology mutator runs a 4-shard → 8-shard ReshardAsync concurrently with the saga chain. Exercises the retroactive prepared-mutation sweep at BeginShadowWrite, the registry's TxDecisionRetention tombstone window, and the saga terminal-fan-out shadow-forward fallback that mirrors TxCommit / TxAbort marks onto the destination shard via the post-Complete MovedAwaySlots lookup.
Digest determinism under load ChaosDigestIntegrationTests.cs ILattice.GetLeafProjectionDigestAsync is byte-stable across repeated calls in a write-quiescent window after concurrent writer / scanner load, and per-shard EntryCount sums equal CountAsync.
Range delete under load ChaosRangeDeleteIntegrationTests.cs A worker repeatedly issues DeleteRangeAsync over the middle band of a 600-key universe (rd-000200 to rd-000400) while point writers, a refill writer re-inserting delete-band keys, scanners and a split coordinator run. Post-window, every key in the two protected bands is present and envelope-valid - the delete never strays outside its range - and scanners never observe a malformed value. The live count inside the delete band is deliberately not pinned, because deletes and refills race.
Compare-and-swap under contention CompareAndSwapChaosTests.cs Four writers increment an eight-key counter universe through a read-then-SetIfVersionAsync (CAS) loop while a split coordinator churns shards. A lost CAS returns false and the caller re-reads with GetWithVersionAsync and retries; post-window, every stored counter equals the number of successful CAS calls made against it, so no update was lost, and no caller saw an exception outside the documented transient class.
Scan cancellation under load ScanCancellationChaosTests.cs Scanner workers repeatedly open ScanKeysAsync / ScanEntriesAsync, cancel 5-24 ms in, and re-open while writers churn the universe. Cancellation must surface as OperationCanceledException (an enumeration abort is tolerated), a partial scan must never yield an unknown key or a malformed value, and a fresh full scan after the window must return exactly the pinned universe - the check that no cancelled enumerator left state behind.
Multi-silo restart under load MultiSiloRestartChaosTests.cs Two-silo TestCluster, sustained write/read load on an ILattice tree, secondary silo restarted every ~2.5 s via TestCluster.RestartSiloAsync. Post-window invariants: pinned CountAsync and an envelope-valid value on every key, read back with a bounded retry (up to 60 s) that absorbs only silo-reactivation faults; during the window every exception is tolerated and counted, and only a read that returns a malformed value fails the run. Uses ProcessScopeMemoryGrainStorage (a static-dictionary-backed IGrainStorage shared across every silo in the test process) so secondary-silo restart does not wipe the shard-root / registry topology - per-silo Orleans memory storage would otherwise let a re-placed shard-root activation read empty state and overwrite the live topology with a fresh leaf root (the underlying split-brain that previously surfaced as InvalidCastException). It also registers one process-shared InMemoryWalStorageProvider on every silo through AddWalStorage, because a reactivated leaf rebuilds its entries from the write-ahead log and the default per-silo provider would vanish with the restarted silo, silently emptying every WAL partition it hosted.
Cross-tree atomic write under shard churn ChaosCrossTreeAtomicWriteIntegrationTests.cs A commit worker drives one all-or-nothing SetManyAtomicAsync saga per generation into the same logical slot of three distinct trees, while per-tree split coordinators churn shards and reader workers probe every settled committed generation. Asserts cross-tree all-or-nothing: a settled committed key is present in all three trees or none, and every committed generation is durably present in all trees post-window, even as splits move keys between shards mid-saga on every tree.
Admission cap under cross-shard pressure AdmissionControlChaosTests.cs Concurrent cross-shard writers drive one tree with an enforcing LatticeOptions.MaxLiveKeys cap and a second tree with only an advisory LatticeOptions.AdmissionAdvisoryLiveKeys ceiling. The enforcing cap must reject some writes with LatticeQuotaExceededException and keep rejecting once its aggregate settles above the cap - overshoot past the configured value is allowed - while the advisory tree never rejects.
Bounded read-through cache eviction BoundedCacheEvictionChaosTests.cs Pins LatticeOptions.MaxCacheValueBytes so small that almost every cached payload is evicted down to its metadata, so nearly every read takes the eviction path back to the primary leaf. Under concurrent overwrite churn, eviction must never turn a live key into a false miss or surface a stale or cross-key payload.
Guarded atomic write under split churn ChaosPredicateAtomicSetManyIntegrationTests.cs Guarded SetManyAtomicAsync<T> batches (Score >= Guard, evaluated server-side against each key's pre-saga value) under split churn and concurrent point writes: a batch over an always-matching band always commits and stamps every key, and a batch holding a permanently failing key always returns PreconditionFailed and writes nothing.
Conditional batch write under split churn ChaosPredicateConditionalSetManyIntegrationTests.cs The guarded SetManyAsync<T> overload stamps a marker onto a band of keys while splits move them and point writers rewrite them: a key whose current value fails the guard is never written, a matching key is, and only submitted keys are ever considered. After the window it also audits the quiesced tree for leaves that no descent from their shard root reaches, and reads every key back through the routing path, so a split that lost its separator fails the run.
Predicate-filtered cursors under split churn ChaosPredicateCursorIntegrationTests.cs Predicate-filtered key, entry and snapshot-entry cursors paged across a mid-paging split: pages stay in ascending key order, every surfaced item satisfies the predicate, and a stable band returns exactly its matching keys.
Conditional range-delete cursor under split churn ChaosPredicateDeleteRangeCursorIntegrationTests.cs A resumable conditional range-delete cursor stepped to completion in bounded pages under split churn and conflicting writes tombstones every in-range key that matches the predicate and nothing else.
Predicate GetManyAsync under split churn ChaosPredicateGetManyIntegrationTests.cs The predicate GetManyAsync<T> overload, evaluated server-side on the owning leaf: every returned value satisfies the predicate, and a stable band returns exactly its matching keys however splits move them.
Conditional range delete under split churn ChaosPredicateRangeDeleteIntegrationTests.cs The conditional DeleteRangeAsync<T> overload tombstones exactly the in-range keys whose value matches the predicate, and never touches a key outside the range, while splits and point writers race it.
Predicate scans under split churn ChaosPredicateScanIntegrationTests.cs The predicate ScanEntriesAsync<T> / ScanKeysAsync<T> / ScanValuesAsync<T> overloads across a mid-scan split: output stays in ascending key order, every surfaced value satisfies the predicate, and a stable band is returned in full.
Cluster-wide split admission after a crash ClusterSplitConcurrencyChaosTests.cs Models a silo crashing mid-split by reporting split footprints that saturate the LatticeOptions.MaxClusterConcurrentAutoSplits ceiling and are then never refreshed; once their time-to-live lapses a fresh grant must succeed, so a crash cannot wedge splitting cluster-wide.
Retry policy masks storage faults RetryPolicyChaosTests.cs Parametrized theory (5%, 15% and 30% fault probability) that arms one-shot storage write faults and requires every caller-side SetAsync to succeed through BoundedExponentialRetryPolicy under an ambient LatticeIdempotencyContext; a companion test arms a fault on every 15 ms tick while incrementing a PnCounter and requires the counter to equal the number of increments that succeeded, so a retry never double-counts.
Shard consolidation under churn ShardConsolidationChaosTests.cs Online shard-consolidation folds run while a split driver shatters the tree, writers ingest, and shard roots are force-deactivated: no key ever becomes unreachable, no acknowledged write is lost, and no virtual slot is left unrouted.
Predicate translator and evaluator agree Predicates/LatticePredicatePushdownChaosTests.cs Eight workers evaluate a storm of structurally random predicates against a shared pool of 256 encoded documents, both through the server-side predicate evaluator and as the compiled lambda; any disagreement fails the run.

Cross-cluster, gRPC, and Azure Table WAL suites

The replication packages and the Azure Table WAL provider ship their own chaos suites - cross-cluster convergence and atomic visibility (test/lattice.replication/Chaos/), the gRPC transport suite (test/lattice.replication.grpc/Chaos/), and the Azure Table WAL suite (test/lattice.storage.azuretable/Chaos/). They drive the real replication pipeline using in-process test clusters, a simulated delivery pump, an in-memory test server, and the Azurite emulator, and are documented in the replication chaos tests.

Schema enforcement and versioning atomicity suite (test/lattice.schema/Chaos/)

The lattice.schema companion package ships its own chaos fixtures that prove the core atomic-write guarantee is preserved while the schema control plane mutates underneath it - a policy is set or cleared, or a target schema version is advanced and eagerly migrated - concurrently with a chain of atomic sagas. They run on a single-silo TestCluster with core lattice, schema enforcement, and schema versioning all registered (SchemaAtomicChaosClusterFixture), and are tagged [Category("Chaos")] [NonParallelizable] like most other suites.

Test class File Purpose
Atomic write under policy churn AtomicWriteUnderPolicyChurnChaosTests.cs A committer drives cross-tree SetManyAtomicAsync sagas (alternating a compliant and a non-compliant leg) while a churner flips each participating tree's enforcement policy between "require JSON" and "no policy". Asserts every saga is decided as a unit against the policy current at admission - it either commits every leg or rejects the whole batch and mutates no tree - and a concurrent reader never observes a torn (cross-generation) snapshot.
Atomic write under version advance AtomicWriteUnderVersionAdvanceChaosTests.cs A committer drives single-tree SetManyAtomicAsync batches into one versioned tree while a churner advances the target schema version v1 -> v2 -> v3 concurrently, then a quiesced eager background migration re-stamps the tree. Asserts every batch lands all-or-nothing, that every read is always envelope-stripped and upcast to the current target, and that the eager migration preserves the last committed snapshot.

The workload

The four full-workload single-cluster fixtures (Tests 1-4 below) run a parallel workload over a fixed key universe. The resize and reshard fixtures pre-register their tree at 4 shards with MaxLeafKeys = 4; the happy-path and faults fixtures seed a fresh tree id without pre-registering it, so their trees take the library defaults (64 shards, 128 keys per leaf, 128 children per internal node) and their topology churn comes from the split driver. Writers only rewrite existing keys with monotonically-increasing values of the form v-{keyIndex}-{writerId}-{seq}. Any value matching that envelope proves the byte array is internally consistent.

The atomic-visibility fixtures (Tests 5-6, 8), the per-mode convergence fixtures (Test 9), the multi-site smoke (Test 10), the range-delete / CAS / scan-cancel public-API fixtures, the production-shipper fixtures (WAL trim, liveness + inbound stats, compaction + shipping), and the downstream-package fixtures (gRPC transport, Azure Table WAL) do not follow this exact shape - each defines its own universe and worker mix appropriate to the invariant it targets. See the per-test sections, the suite tables at the top of this document, or the replication chaos tests for the replication and storage fixtures.

Fixture and parameter differences:

Test Fixture MaxLeafKeys MaxInternalChildren Key prefix Universe
Happy-path FourShardClusterFixture (tree not pre-registered: 64 shards) 128 (library default) 128 (library default) chaos-{i:D5} 500
Chaos + faults MultiShardFaultInjectionClusterFixture (tree not pre-registered: 64 shards) 128 (library default) 128 (library default) fchaos-{i:D5} 200
Chaos + resize FourShardClusterFixture (4 shards) 4 -> 16 mid-run 128 (library default) -> 16 mid-run resize-chaos-{i:D5} 200
Chaos + reshard FourShardClusterFixture (4 shards -> 8, or -> 2 in the shrink run) 4 128 (library default) reshard-chaos-{i:D5} 200 
flowchart LR
    Seed[Seed universe<br/>N keys] --> Chaos

    subgraph Chaos[Chaos window]
      direction TB
      PW[Point writers] --> Tree
      BW[Bulk writers] --> Tree
      AW[Atomic writers] --> Tree
      PR[Point readers] --> Tree
      BR[Bulk readers] --> Tree
      SC[Scanners] --> Tree
      CT[Counters] --> Tree
      TM[Topology mutator<br/>split / resize / reshard<br/>± fault injector] --> Tree
      Tree[(ILattice)]
    end

    Chaos --> Assert[Assert invariants]

Worker categories (the exact mix and counts vary per test; each fixture declares them as constants at the top of its source):

  • Point writers - SetAsync on random universe keys.
  • Bulk writers - SetManyAsync with batches of 8 random keys (happy-path / faults only).
  • Atomic writers - SetManyAtomicAsync with batches of 2 random keys (happy-path / faults only).
  • Point readers - GetAsync; validates envelope if a value is returned.
  • Bulk readers - GetManyAsync for 16 random keys (happy-path / faults only).
  • Scanners - happy-path: rotating ScanKeysAsync, ScanEntriesAsync, reverse scan and range scan, each of which must yield exactly the universe (or the range) in order, with no duplicates and no unknown keys; resize and reshard: full-tree ScanKeysAsync with no duplicates and no unknown keys; faults: ScanEntriesAsync, checking order, unknown keys and envelopes.
  • Counters - CountAsync must always equal the pinned universe size (the faults fixture lets the count drift during its fault window).
  • Topology mutator - test-specific:
    • happy-path: every ~200 ms drives a manual split, and then a split pass, on a randomly chosen non-empty shard.
    • faults: the same split driver at a ~500 ms cadence, plus a fault injector that arms random WriteStateAsync faults.
    • resize: initiates ResizeAsync once at the window start and pumps the coordinator to completion.
    • reshard: initiates ReshardAsync(8) once at the window start and pumps the reshard coordinator and the per-shard split passes to completion; the shrink run initiates ReshardAsync(2) instead and pumps the coordinator and every fold it starts.

Test 1 - Happy-path chaos (ChaosIntegrationTests)

This test establishes that ILattice's consistency guarantees hold during the chaos window, not just after it closes. Every operation observes a fully consistent view of the tree.

What it proves

Invariant Mechanism under test
CountAsync returns the exact universe size, always Per-slot count routing against the authoritative ShardMap, each shard counted in work-bounded batches, plus version stability check
ScanKeysAsync / ScanEntriesAsync yield exactly the universe, no duplicates, no unknowns, in strict sorted order In-line reconciliation-cursor injection into the k-way merge + HashSet dedup
ScanKeysAsync(null, null, reverse: true) yields the full universe in reverse Reverse-scan path also reconciles
ScanKeysAsync(start, end) yields exactly the in-range slice Range pruning is slot-aware
GetAsync / GetManyAsync never return a corrupt value Writes are atomic per-shard; CRDT LWW resolves concurrent rewrites
No public-API call throws an unhandled exception Stale routing retries and enumeration aborts are transparent
Splits during a scan never cause data loss, duplication, or out-of-order output MovedAwaySlots + version stamping + in-line reconciliation

Tolerated transients

The test counts these as transient rather than as failures:

  • EnumerationAbortedException - a stream cursor grain deactivated mid-iteration. The caller re-issues the scan.
  • The stale shard-routing signal - a routing activation used a cached shard map after a concurrent split committed its swap. The routing tier invalidates its map and retries within a bounded wall-clock budget, so the signal reaches the caller only once that budget is spent.
  • An InvalidOperationException reporting that an atomic write failed and was rolled back (a saga aborted by a transient routing fault mid-split), or that a count, scan or GetManyAsync exhausted LatticeOptions.MaxScanRetries while the topology or the saga rate kept changing.
  • TimeoutException - an Orleans call timeout under saturated load.

Any other exception, or any observed envelope/duplicate/missing-key violation, fails the test.

Pass criteria

After the chaos window closes:

  • CountAsync matches the pinned universe size exactly.
  • ScanKeysAsync yields exactly the pinned universe (no gaps, no extras).
  • Every worker category performed at least one operation (proves the workload ran under real concurrency, not a degenerate single-thread schedule); splits and atomic writes count attempts.
  • Once the atomic writer's committed and transiently failed sagas total at least three, at least half of them committed.
  • Zero envelope violations were observed during the window.

Test 2 - Chaos + storage faults theory (ChaosWithFaultsIntegrationTests)

This parametrized theory layers random storage faults on top of the same workload. Unlike the happy-path test, per-operation invariants are weakened during the fault window - arbitrary exceptions are tolerated because a failed WriteStateAsync legitimately cascades into split aborts, stale routing, and count drift. Instead, the test asserts eventual convergence: once faults stop and the cluster quiesces, the tree must recover to the exact same pinned universe with every value still matching its envelope.

faultProbability is the probability, per 20 ms tick, that the fault injector arms a fresh one-shot WriteStateAsync fault on a randomly chosen target grain (the initial leaf and the shard-root grain of shards 0-3, a subset of the tree's 64 shards). Orleans' FaultInjectionGrainStorage consumes each armed fault on the next write for that grain, so the injector re-arms continuously to approximate a steady-state failure rate.

Note: Orleans' one-shot fault API caps concurrent armed faults at ≈ |targets|. Higher faultProbability primarily drives faster re-arm latency rather than a linear increase in fault count. The gradient is still meaningful for exercising recovery paths under progressively heavier disruption.

Test phases

sequenceDiagram
    participant Test
    participant Tree as ILattice (64 shards)
    participant Injector
    participant Workers
    Test->>Tree: Seed universe (faults off)
    Test->>Injector: Start at p=faultProbability
    Test->>Workers: Start 13 role workers + split coordinator
    loop Chaos window (8 s)
        Injector-->>Tree: AddFaultOnWrite(random target)
        Workers-->>Tree: mixed reads/writes/scans/splits
        Note over Workers: exceptions tolerated<br/>envelope-check values if observed
    end
    Test->>Injector: Stop (cts fires)
    Test->>Tree: DrainAndHealAsync (up to 15 s)
    Note over Tree: retry writes over universe<br/>until 3 consecutive clean passes
    Test->>Tree: Assert strong invariants

Tolerated during faults

Every exception type is tolerated and counted (tolerated-write-errors, tolerated-read-errors, tolerated-scan-errors, etc.). A single storage fault cascades into many observable shapes:

  • Direct InvalidOperationException from the faulted write.
  • OrleansException wrappers when a faulted grain deactivates.
  • EnumerationAbortedException if a stream cursor was on the deactivated grain.
  • StaleShardRoutingException after a shard map swap when the split coordinator crashed and resumed mid-phase.
  • ArgumentException from the injector itself when a target already has an armed fault pending (skipped).

Envelope violations (a value that doesn't start with v-{index}-) are not tolerated - CRDT LWW is supposed to preserve atomicity of the value payload even when the wrapping write fails.

Healing phase (DrainAndHealAsync)

After the fault injector stops, lingering armed faults remain on whichever targets weren't hit during the chaos window. The test drains them by replaying writes over the entire universe until 3 consecutive passes complete exception-free, bounded by a 15 s timeout. This loop:

  • Consumes any remaining one-shot faults (each fires once on its next write, clearing itself).
  • Gives resumable splits and pending root promotions time to reach their RunSplitPassAsync keepalive tick and replay.
  • Rewrites every universe key with a fresh value on each pass (v-{i}-heal-{pass}), so after three clean passes every key holds a value a fault-free pass wrote, and that value matches its envelope.

Pass criteria (post-quiescence)

After healing:

  • CountAsync == UniverseSize exactly.
  • ScanKeysAsync yields exactly the pinned universe.
  • ScanEntriesAsync yields exactly the pinned universe with every value matching its envelope.
  • Every universe key appears in the post-heal ScanKeysAsync.
  • Zero envelope violations were observed during the whole run.
  • Point writes, point reads, scans, counts and splits each completed at least once and at least one atomic write was attempted; the injector armed at least one fault (for p > 0); in the no-fault baseline at least 70% of atomic writes committed (with fewer than ten attempts: at least one committed and at most one failed).

Test 3 - Chaos + online resize (ChaosResizeIntegrationTests)

This test targets the online resize path. A full concurrent workload runs against a seeded tree while ResizeAsync changes the B+ fan-out to MaxLeafKeys = 16 / MaxInternalChildren = 16 under SnapshotMode.Online. The resize starts inside the chaos window and its phases - snapshot drain, alias swap, per-shard reject phase, cleanup - are pumped there under live traffic; any phase still running when the window closes is finished in a post-window drain of up to 30 s before the invariants are checked.

Recovery surfaces exercised

  • TreeResizeGrain phase machine (Snapshot → Swap → Reject → Cleanup) under sustained traffic.
  • Shadow-forwarding on every source shard - the workload's live point writes during the drain must be mirrored to the destination with their original HLCs (the forward carries last-writer-wins writes only; typed CRDT deltas and bulk appends are not mirrored).
  • Alias swap - mid-flight GetAsync / SetAsync on a stateless-worker LatticeGrain activation holding a stale alias must transparently re-resolve and retry.
  • Strongly-consistent CountAsync / ScanKeysAsync during the online snapshot drain and Rejecting phase.

Tolerated transients

The same set as the happy-path test, plus StaleTreeRoutingException raised during the alias swap window.

Pass criteria

After the chaos window closes:

  • CountAsync matches the pinned universe size exactly.
  • ScanKeysAsync yields exactly the pinned universe.
  • The resize coordinator reports itself idle, so the resize ran to completion.
  • Every worker category performed at least one operation; the resize was driven to completion.
  • Zero envelope violations observed during the window.

Test 4 - Chaos + online reshard (ChaosReshardIntegrationTests)

This test targets the online reshard path - growing the physical shard count from 4 to 8 while the tree continues to serve traffic. The reshard is kicked off synchronously before the chaos timer starts (so cold-activation cost doesn't burn the window on slow Release CI runners); the in-window driver only pumps the migration loop to completion. A second test in the fixture runs the same workload while ReshardAsync(2) shrinks a fresh 4-shard tree to 2 shards, its driver pumping the coordinator and every fold it starts. Both runs finish any residual work in a post-window drain of up to 30 s.

Recovery surfaces exercised

  • TreeReshardGrain migration loop under sustained traffic - eligibility filtering, dispatch-budget clamping (MaxConcurrentMigrations), re-evaluation across ticks.
  • ShardMap convergence when reshard-dispatched splits race with workload writes (shadow-write, drain, swap, reject, permanent MovedAwaySlots).
  • No invariant drift across the full reshard window.
  • Shrink run: fold planning and concurrency under sustained traffic, every fold's drain, freeze, swap and finalise racing live writes, and the release of each retired donor's storage while scans and counts may still hold a pre-fold shard map.

Neither run exercises the autonomic hot-shard monitor's rule that suppresses its passes while a reshard is in flight: the fixture keeps the default HotShardSampleInterval and AutoSplitMinTreeAge, so every monitor pass returns inside the minimum-tree-age grace period, before it reaches that rule, until well after a reshard normally completes, and no assertion reads the monitor.

Tolerated transients

EnumerationAbortedException, the stale shard-routing signal, TimeoutException, and the same saga-rollback and MaxScanRetries-exhaustion InvalidOperationException messages as the happy-path test.

Pass criteria

After the chaos window closes:

  • CountAsync matches the pinned universe size exactly.
  • ScanKeysAsync yields exactly the pinned universe.
  • The reshard coordinator reports itself idle, so the reshard ran to completion.
  • The post-reshard ShardMap has at least ReshardTarget distinct physical shards.
  • Every worker category performed at least one operation.
  • Zero envelope violations observed during the window.

The shrink run checks the same count, scan, idle-coordinator, worker and envelope criteria, and additionally that the final ShardMap holds exactly two distinct physical shards and that each of the two shards that left the map reports itself retired and holds no leaves.

Test 5 - Atomic-write reader isolation (AtomicVisibilityChaosTests)

This test asserts the universal reader-isolation invariant for SetManyAtomicAsync: every poll of a continuous reader concurrent with an in-flight saga must observe either the full pre-saga snapshot, the full post-saga snapshot, or all keys hidden - never a partial view. The invariant holds per poll, with no bounded-window caveat, across 50 sequential saga rounds at a 10 ms reader cadence.

What it proves

Invariant Mechanism under test
Continuous reader observes zero-or-all keys at every poll The leaf's prepared-write commit path stages every saga write in a per-transaction pending bucket, and every key a read reaches resolves against one transaction-registry decision, the tree-wide visibility flip
Saga drives 16 keys spanning multiple leaves through the full prepare → terminal pipeline AtomicWriteGrain per-shard terminal broadcast, idempotent under concurrent retry
Final post-round value is preserved across 50 iterations LWW resolution under saga commit ordering

Workload

  • Seed phase - SetAsync for each of 16 keys (atomic-00 … atomic-15) at round 0, followed by a single SetManyAtomicAsync at round 0 to land all keys through the saga path before reader rounds begin.
  • Saga rounds - 50 sequential rounds. Each round starts a continuous reader task that polls all 16 keys via GetManyAsync every 10 ms, and concurrently issues SetManyAtomicAsync with the post-round value envelope.
  • Reader classification - every poll is bucketed: fullPre (every key at the previous round's value), fullPost (every key at the new round's value), fullHidden (every key missing during the prepare → terminal window), or split (any mixed observation, which fails the test).

Pass criteria

  • Zero split-view failures across all 50 rounds.
  • totalPolls > 0 and fullPostPolls > 0 (proves the reader and saga ran under real concurrency).
  • Final GetManyAsync of all 16 keys yields the round-50 envelope on every key.

Tolerated transients

The reader's GetManyAsync may observe OperationCanceledException at the round boundary; saga writes are not expected to surface any transient - the saga's own retry-on-stale-routing logic absorbs split / resize / reshard activity at the API layer.

Companion observability

See Metrics.

Test 6

Three sibling fixtures extend Test 5's reader-isolation invariant across each of the three online topology mutations (shard split, online resize, online reshard). Every fixture seeds the same 16-key universe, then drives 15 sequential SetManyAtomicAsync rounds while the topology mutator runs in parallel; a continuous reader polls all 16 keys every 10 ms and every poll must observe either the full pre-round value, the full post-round value, or all 16 keys hidden - never a partial subset.

What they prove

Invariant Mechanism under test
Saga prepares survive a mid-flight shard split Source shard's shadow-forward pipeline mirrors prepared entries to the destination during the split's drain phase; saga terminal-broadcast retries onto the new owner via StaleShardRoutingException
Saga prepares survive a mid-flight online resize Source physical tree shadow-forwards every last-writer-wins write (including saga prepares; typed CRDT deltas and bulk appends are not mirrored) to the destination physical tree during the snapshot drain; the alias swap is observed via the stale tree-routing signal and the saga terminal-broadcast retries onto the new owner
Saga prepares survive a mid-flight online reshard TreeReshardGrain migration loop dispatches per-shard splits; the retroactive prepared-mutation sweep at BeginShadowWrite and the terminal-fan-out shadow-forward fallback together mirror prepares and the saga's TxCommit / TxAbort marks onto the destination shard via the post-Complete MovedAwaySlots lookup; the registry's TxDecisionRetention tombstone window absorbs duplicate terminals

Workload (per fixture)

  • Seed phase - SetAsync for each of 16 keys at round 0, followed by a single SetManyAtomicAsync at round 0 so the universe is pinned through the saga path before the topology mutator starts.
  • Topology kick-off - exactly once before the saga loop: SplitAsync(shard 0) for ShardSplitTopologyTests, ResizeAsync(MaxLeafKeys=8, MaxInternalChildren=8) for ResizeTopologyTests, or ReshardAsync(8) for ReshardTopologyTests. A background driver pumps the coordinator's RunSplitPassAsync / RunResizePassAsync / RunReshardPassAsync to completion while the saga loop runs.
  • Saga loop - 15 sequential rounds. Each round starts a continuous reader task that polls all 16 keys via GetManyAsync every 10 ms and concurrently issues SetManyAtomicAsync with the post-round value envelope.
  • Reader classification - identical to Test 5: fullPre, fullPost, fullHidden, or split (mixed observation, fails the test).
  • Drain phase - after the saga loop, the test pumps the coordinator to idle and asserts the final post-round value is present on every key.

Pass criteria (per fixture)

  • Zero split-view failures across all 15 rounds.
  • totalPolls > 0 and fullPostPolls > 0.
  • Final GetManyAsync of all 16 keys yields the round-15 envelope on every key.
  • The split, resize or reshard coordinator reports itself idle before the test exits.

Tolerated transients

  • The stale shard-routing and stale tree-routing signals on the reader's GetManyAsync - every fixture's reader loop catches both and retries.
  • OperationCanceledException at the round boundary when the reader's CTS fires.

Companion observability

Same orleans.lattice.atomic_write.* histograms as Test 5, plus the topology-mutator-side counters that fire when the split / resize / reshard actually mutates the tree (orleans.lattice.leaf.splits, orleans.lattice.shard.splits_committed, and the per-split orleans.lattice.split.retroactive_forward.duration / .entries pair on shadow-forward). See Metrics.

Test 7 - Digest determinism under load (ChaosDigestIntegrationTests)

This test exercises ILattice.GetLeafProjectionDigestAsync under sustained concurrent load and asserts two determinism invariants that gate the digest's value as a cross-silo divergence detector: byte-identical repeated calls in a write-quiescent window, and per-shard EntryCount sums equal CountAsync.

What it proves

Invariant Mechanism under test
Digest hash is byte-stable across repeated calls when no writes occur in between The digest is an order-independent XOR fold of per-entry XxHash128 contributions, maintained at each mutation and aggregated up the internal nodes, so once the last coalesced publish has landed, repeated reads in a write-quiescent window hash the same persisted aggregate
Sum of per-shard EntryCount equals CountAsync Shard-level digest counts are accountable against the tree's own population view
Digest computation is safe to call concurrently with foreground writer / scanner traffic No exception is observed on any worker - digest rendering does not block or interfere with the read / write path

Workload

  • Seed phase - 200 keys (chaos-digest-{i:D5}) preloaded with SetAsync.
  • Chaos window (~8 s) - 4 writer tasks rewriting random keys, 2 scanner tasks calling KeysAsync, 1 digest poller calling GetLeafProjectionDigestAsync for every shard in a tight loop.
  • Quiesce phase - after the chaos window closes, the digest is sampled twice in succession with no intervening writes.

Pass criteria

  • No exception observed on any worker during the chaos window (EnumerationAbortedException is tolerated on the scanner; everything else is fatal).
  • For every shard, secondPass[s].Hash equals firstPass[s].Hash and EntryCount is equal.
  • Sum of firstPass[s].EntryCount equals tree.CountAsync().
  • tree.CountAsync() equals 200 (writers only update existing keys; no inserts or deletes).

Tolerated transients

  • EnumerationAbortedException from the scanner's KeysAsync enumeration when a stream cursor grain deactivates mid-iteration.

Test 8 - Cross-cluster atomic-visibility chaos (CrossClusterAtomicVisibilityChaosTests)

This test asserts the cross-cluster receiver-side reader-isolation invariant for SetManyAtomicAsync: a saga authored on one site and shipped via the WAL replication transport to two peer sites must be observed all-or-nothing on every receiver, even when the inter-site delivery topology is partitioned and healed mid-workload. It is the cross-cluster sibling of Test 5 and exercises the same reader-isolation mechanism - prepared writes staged in per-transaction pending buckets and made visible by one transaction-registry decision - through the receiver-side prepared/terminal apply seam.

What it proves

Invariant Mechanism under test
Every saga's keys land all-or-nothing on every receiver site Receiver-side prepared/terminal apply seam staging prepared writes in the leaf's per-tx pending bucket, then flipping them on terminal arrival
Source HLC rides through the wire verbatim LatticeHlcOverrideContext wrapping the apply call so the receiver does not stamp a fresh local HLC
Repeated terminal delivery is idempotent The per-tree transaction registry classifies an arriving terminal against the decision already recorded and returns without a write when the outcome matches, so a redelivery inside the retention window changes nothing; the leaf additionally absorbs a second delivery within one activation from its recently-terminal set
Mid-workload partition does not produce partial-saga visibility on any site Prepares queued behind the partition and the matching terminal both ship after heal; the receiver-side staging buffer holds prepared entries off the visible projection until the terminal arrives
Producer-side per-key WAL filter does not strand terminals ReplicationShipperGrain.ShouldShip bypasses KeyFilter / KeyPrefixes for TxCommit / TxAbort records

Workload

  • Topology - three independent TestCluster instances (site-0, site-1, site-2) wired through MultiSiteClusterFixture, each with its own MemoryGrainStorage-backed Lattice and a per-site ReplicationApplier driven by the chaos delivery pump.
  • Author phase - every site concurrently runs 6 local SetManyAtomicAsync sagas (4 keys per saga, deterministic per-saga key prefix), so each saga emits 4 prepared Set records + 1 TxCommit per touched shard onto its source site's WAL.
  • Partition cycle - site-0's loop isolates site-2 after the first third of its workload and heals it after the second third. The ChaosDeliveryPump continues polling but does not advance the cursor on the partitioned edges, so the prepares and terminals authored during the outage queue at the source and ship en bloc after heal.
  • Drain phase - after every author task completes, pump.HealAllAndDrainAsync heals every edge and waits for every per-edge cursor to catch up to its sender's WAL tail, with a 60 s timeout.

Pass criteria

  • On every receiver site, for every authored saga: the count of visible keys is either 0 (saga not yet shipped or aborted) or KeysPerSaga (saga fully visible). Any partial-visibility count is a saga-atomicity violation and fails the test.
  • On every receiver site, every saga's keys are present after the drain - every authored saga is a local commit, so universal visibility is the strong post-drain assertion.
  • pump.PumpErrors is empty - a transient grain failure during the run surfaces here without aborting the loop, but the convergence assertion remains the source of truth.

Tolerated transients

The chaos pump's per-edge loop catches and queues transient grain exceptions onto PumpErrors; only sustained faults that prevent convergence within the drain timeout fail the test. The universe is small (3 sites x 6 sagas x 4 keys = 72 keys) so authoring completes in seconds and the drain typically settles in under a second after heal.

Companion observability

Saga writes emit orleans.lattice.atomic_write.duration / orleans.lattice.atomic_write.batch_size on the authoring site (see Metrics). On the receiver side every inbound entry's apply attempt records orleans.lattice.replication.apply.duration, tagged with the tree, the source peer's cluster id (peer), the apply outcome and the owning tenant.

Test 9

Four of the replication suite's per-merge-mode convergence fixtures - last-writer-wins, OR-Set, PN-Counter and MV-Register; the suite covers every other LatticeMergeMode value too, catalogued in the replication chaos tests - prove that the producer-side change-feed → shipper → receiver-side applier pipeline converges every site to the same final state under concurrent multi-site writes and mid-workload partitions. Every fixture wires three sites through MultiSiteClusterFixture / ChaosDeliveryPump, declares the test tree under the relevant merge mode, lets every site author a disjoint workload while one site is isolated and re-healed mid-flight, drains the pump, then asserts the mode-specific convergence invariant pointwise across sites.

What they prove

Fixture Mode Convergence invariant Mechanism under test
LwwRegisterConvergenceChaosTests LwwRegister Every site reads the same VersionedValue after drain - the lexicographic (HLC, originClusterId) winner across all authored writes The receiver applies each shipped write through its replication-apply seam with the source HLC, so LWW resolution picks the same winner on every site; per-edge change-feed cursors do not skip entries across partition heal
OrSetConvergenceChaosTests OrSet Every site's OrSet(key).GetAsync() yields exactly the union of authored adds (test 1), or the union of authored adds minus the union of authored removes (test 2) The receiver folds each shipped typed delta into its local state under LatticeOriginContext (a bootstrap entry that carries full state instead of a delta is merged and written back by compare-and-swap); OR-Set's commutative-monoid merge absorbs out-of-order receive
PnCounterConvergenceChaosTests PnCounter Every site's PnCounter(key).ValueAsync() returns the same algebraic sum of authored deltas The same receiver-side typed-delta fold as OR-Set; per-replica P/N maps merge by component-wise max
MvRegisterConvergenceChaosTests MvRegister Every site's MvRegister<T>(key).ValuesAsync() yields exactly the dot-frontier expected from the authored history: concurrent writes survive as a multi-value set, and any write whose dot is causally dominated by a later writer's observed context is superseded on every replica The same receiver-side typed-delta fold under LatticeOriginContext; dot-context merge drops dominated entries and pointwise-maxes the per-replica context maps

Workload (per fixture)

  • Topology - 3 TestCluster instances wired through MultiSiteClusterFixture declared with the fixture's merge mode; ChaosDeliveryPump drives every inter-site edge.
  • Author phase - every site authors a disjoint family of writes against a single key (k):
    • LWW: 40 sequential SetAsync calls per site.
    • OR-Set test 1: 25 sequential AddAsync calls per site.
    • OR-Set test 2: 15 adds + 2 observed-removes per site.
    • PN-Counter: 30 increments + 10 decrements per site.
    • MV-Register test 1: every site is isolated before any write, each writes one value concurrently, and all three values survive the heal as a multi-value set.
    • MV-Register test 2: two-phase scenario - site 0 issues two sequential SetAsync calls and drains so every peer observes its dot context; then sites 1 and 2 write concurrently behind a partition that isolates site 2 from site 1, producing two surviving concurrent dots that both dominate the site-0 entry.
  • Partition cycle - one site's loop isolates a target site for part of its workload, and both the window and the (driver, target) pair vary per fixture: site 2 isolates site 1 over the middle half of the LWW writes; site 0 isolates site 2 from half-way to its last add in OR-Set test 1, and over the middle third in test 2; site 1 isolates site 0 over the middle third for PN-Counter; in MV-Register every site is isolated for test 1's concurrent writes, and site 2 for test 2's concurrent phase.
  • Drain phase - after every author task completes, pump.HealAllAndDrainAsync(30 s) heals every edge and waits for every per-edge cursor to catch up.

Pass criteria

  • LWW: pointwise equality of (Value, Version) across all 3 sites after drain.
  • OR-Set: pointwise set-equivalence of Elements() across all 3 sites against the closed-form expected union.
  • PN-Counter: pointwise equality of ValueAsync() across all 3 sites against SiteCount * (IncrementsPerSite - DecrementsPerSite).
  • MV-Register: pointwise set-equivalence of ValuesAsync() across all 3 sites against the closed-form expected frontier (surviving concurrent dots only).

Tolerated transients

  • PN-Counter and MV-Register wrap IncrementAsync / DecrementAsync and SetAsync in a call-site retry (8 attempts with linear backoff, then one final unretried call) that catches an InvalidOperationException carrying CAS budget exhausted. The wrapper is defensive only: both accessors now apply one producer-side delta per call - a single read, then one ApplyCrdtDeltaAsync, with no compare-and-swap loop - so neither raises that exception, and a foreign-origin merge that lands mid-call cannot abort the local write.
  • OR-Set / LWW: no per-call retry needed - the pipeline is non-blocking from the producer side; transient pump errors queue onto pump.PumpErrors and surface in the post-drain assertion if convergence stalls.

Companion observability

Producer-side: the orleans.lattice.replication.ship.* family (see Metrics) - the ship.duration, ship.effective_batch_size and ship.ack_latency histograms plus its payload, manifest and dictionary counters. Receiver-side: orleans.lattice.replication.apply.duration, tagged with the tree, the source peer's cluster id (peer), the apply outcome and the owning tenant - it carries no merge-mode tag, so each fixture's streams are told apart by its tree - and a 3-site run records two peer values on each receiver site (2 inbound edges each), six in all.

Test 10 - Multi-site fixture smoke (MultiSiteClusterFixtureSmokeTests)

These are not chaos tests in the bombardment sense - they are deterministic single-write smoke tests that pin the simpler invariants the cross-cluster convergence suite relies on. They ship under [Category("Chaos")] so they run in the chaos tier alongside the suites they diagnose; a failure here short-circuits failure analysis on the larger convergence / atomic-visibility fixtures by isolating which seam broke.

What they prove

Test Invariant
Site_change_feed_yields_locally_authored_lww_entry A locally-authored SetAsync lands on the per-site producer-side WAL, the local IChangeFeed yields the captured WalRecord, and the record carries the site's OriginClusterId and declared LatticeMergeMode
Delivery_pump_ships_lww_entry_from_site_0_to_site_1 A locally-authored SetAsync on site 0 propagates end-to-end through the chaos delivery pump to site 1, where a subsequent GetAsync returns the authored value

Tolerated transients

None - these tests run on a quiescent 2-site fixture with no partition / concurrency. Any exception is fatal.

Test 11 - Cross-tree atomic write under shard churn (ChaosCrossTreeAtomicWriteIntegrationTests)

This test asserts the cross-tree all-or-nothing visibility invariant for SetManyAtomicAsync: a key committed by a cross-tree saga is present in every participating tree or in none of them - a reader must never observe a partial cross-tree commit (some trees flipped, others not), even while shard splits move keys between shards mid-saga on every tree.

A single commit worker writes a fresh per-generation key into the same logical slot of three distinct trees as one all-or-nothing cross-tree saga; three reader workers continuously probe committed generations across all three trees; and one split coordinator per tree churns shards continuously throughout the ~15 s window.

What it proves

Invariant Mechanism under test
A settled committed generation key is present in all three trees Coordinator's single global decision flips visibility uniformly across every participating tree's ITxRegistryGrain via delegation
A settled committed key never vanishes under concurrent split churn Per-tree finalize promotes prepared entries into the main store before settlement; splits are read-transparent for promoted committed keys
Every committed generation is durably present in all trees post-window Cross-tree all-or-nothing completeness across the whole run

Workload

  • Seed phase - 200 SetAsync keys per tree across three trees (xct-{run}-0..2), each on a four-shard fixture (FourShardClusterFixture) so splits are enabled.
  • Commit worker - one cross-tree saga per generation at a 50 ms cadence: a LatticeTreeBatch per tree writing x-{gen} into all three trees under operation id xctop-{run}-{gen}. Asserts each saga returns Committed.
  • Reader workers - three readers repeatedly probe the six most recently settled generations (ending at the commit frontier minus a SettleMargin of 5) in all three trees; a settled generation that reads absent in any tree is a failure.
  • Split coordinators - one per tree at a 250 ms cadence, picking a random non-empty shard and driving SplitAsync + RunSplitPassAsync.

Pass criteria

  • Zero invariant violations (no settled committed key absent from any tree).
  • Post-window: every committed generation 0..lastCommitted present in all three trees.
  • The commit worker committed at least one saga, readers performed at least one probe, and split coordinators attempted at least one split (proves real concurrency).

Why the invariant is checked only on settled generations

The cross-tree feature guarantees all-or-nothing commit, not a simultaneous cross-tree read snapshot. Each per-tree GetAsync is an independent point-in-time read, so the global decision flip landing between a reader's three samples yields a benign transient skew that is not a partial-commit violation. The reader therefore checks monotonicity only on generations the commit frontier has advanced a safety margin past, by which point every participant's finalize has promoted the entry into its main store. Post-window completeness then closes the gap for the in-flight tail.

Tolerated transients

The stale shard-routing signal, EnumerationAbortedException, cursor snapshot expiry or pin exhaustion, TimeoutException, and the saga's own rollback / topology-churn / fan-out-saturation / "fewer than 2 virtual slots" InvalidOperationException messages are bucketed as transient - the commit worker moves on to its next generation, and the readers and split coordinators keep looping; any other exception fails the test.

Test 12 - Cross-cluster cross-tree atomic visibility (CrossClusterCrossTreeAtomicVisibilityChaosTests)

This test (in the replication suite) asserts the receiver-side cross-tree all-or-nothing invariant: when a cross-tree SetManyAtomicAsync batch spanning two replicated trees ships to a remote cluster, a reader on the receiver must never observe one tree committed while a sibling tree is still pre-saga. Because each tree's terminals replicate on their own WAL feed, the receiver routinely applies one tree's terminal before the other's; the receiver coordinator grain (ILatticeCrossTreeReceiverGrain) must hold every participating tree invisible until all of the batch's replicated terminals have arrived, then flip them together.

Workload

  • Two sites, each authoring SetManyAtomicAsync batches spanning two replicated trees (chaos-xt-a, chaos-xt-b) under deterministic per-batch key namespaces and a stable operationId.
  • Two independent ChaosDeliveryPump instances - one per tree - ship the two trees' change feeds with independent partition cycles, so the receiver routinely sees one tree's terminal before the other's.
  • A mid-workload partition isolates each tree's edges at staggered points and heals them later, forcing terminal skew across the two trees.

Pass criteria

  • On every receiver site, for every authored batch, tree A's keys and tree B's keys have identical presence - both fully visible or both fully absent. Tree A visible while tree B is absent (or vice versa) is a cross-tree atomicity violation.
  • No single-tree partial view (every batch's per-tree key set is wholly present or wholly absent).
  • Post-drain: every batch's keys are present on both trees on every site.

Test 13 - Atomic write under enforcement-policy churn (AtomicWriteUnderPolicyChurnChaosTests)

This test (in the lattice.schema suite) asserts that cross-tree SetManyAtomicAsync stays all-or-nothing while a tree's schema enforcement policy is changed live. Schema enforcement validates the whole batch once, up front at the coordinator, against whatever policy is current at admission - so concurrent policy churn may change which outcome a given saga gets, but must never split a batch.

A committer drives one cross-tree saga per generation across two trees - one leg always compliant, the other alternating compliant (odd generations) and deliberately non-compliant (even generations) - while a churner flips each tree's policy between "require JSON" and "no policy" at a 5 ms cadence, and a reader polls both trees on every generation.

What it proves

Invariant Mechanism under test
A committed saga installs every leg on every tree Admission-time whole-batch validation decides the saga atomically before any prepare
A rejected saga mutates no tree A single admission failure rejects the whole batch; no leg is applied
A reader never sees a torn per-tree snapshot Atomic-write reader isolation holds under concurrent policy mutation

Workload

  • Committer - one cross-tree SetManyAtomicAsync saga per generation (60 generations) over treeA (always compliant) and treeB (compliant on odd, non-compliant on even generations), each value embedding its generation.
  • Policy churner - flips each tree's policy between a LatticeSchemaRule.Json() policy and no policy at a 5 ms cadence.
  • Reader - a concurrent per-generation read of both trees, asserting each tree's key set is at one coherent generation (never a torn cross-generation mix).

Pass criteria

  • Zero invariant violations: every committed generation is present on both trees on every key; every rejected generation advanced neither tree.
  • At least one generation committed and at least one was rejected by the churned policy (proves the race actually exercised both outcomes).

Tolerated transients

The committer treats a LatticeSchemaViolationException (or a non-committed outcome) as the reject branch; any other exception fails the test.

Test 14 - Atomic write under schema-version advance and eager migration (AtomicWriteUnderVersionAdvanceChaosTests)

This test (in the lattice.schema suite) asserts that single-tree SetManyAtomicAsync stays all-or-nothing and every stored value stays self-describing and decodable while a versioned tree's target schema version is advanced concurrently, and that the one-call eager background migration preserves that atomic snapshot when it re-stamps the tree. It runs in two phases matching the two distinct version-change operations and their concurrency contracts.

A target-version advance is a config-only change (it moves no data), so it is safe to run concurrently with live writes: new writes stamp at the new target and existing values are upcast on read. The eager data migration is a shadow-build with alias cutover whose v1 contract requires the tree be write-quiescent, so it is validated in a quiesced window - not hammered under live writes.

What it proves

Invariant Mechanism under test
Every atomic batch lands as a unit under a concurrent version advance The write interceptor stamps each delta at admission; the advance never tears the saga
A read never returns raw enveloped bytes and never throws for a reachable version The read decoder strips the envelope and upcasts to the current target through the registered hop chain
Eager migration preserves the last committed snapshot The quiesced shadow-build re-stamps each value from its own version to the target and cuts the alias over atomically

Workload

  • Committer - single-tree SetManyAtomicAsync batches (60 generations) of 12 keys into one versioned tree, each value embedding its generation.
  • Version churner - advances the target schema version v1 -> v2 -> v3 (once each) concurrently with the write loop.
  • Reader - a concurrent per-generation read that fails on any raw enveloped byte leak or torn batch.
  • Quiesced migration - after the write loop drains, one MigrateToTargetVersionAsync re-stamps every value (written across v1..v3) to the current target, followed by an idempotent second call.

Pass criteria

  • Zero invariant violations across the concurrent phase (all-or-nothing, decodable, no raw-envelope leak).
  • The target advanced exactly twice (v1->v2->v3).
  • The eager migration succeeded, preserved the last committed generation on every key, and the idempotent re-migration also succeeded.

Tolerated transients

None beyond the churner's own cancellation; any unexpected exception during the advance, the write loop, or the migration fails the test.

Observed recovery surfaces

Between them, the chaos tests exercise every recovery path documented in shard-splitting.md, online-reshard.md, tree-sizing.md, tombstone-compaction.md, wal.md, wal-storage-providers.md, state-primitives.md, ../lattice.replication/README.md, ../lattice.replication/transport.md, ../lattice.replication.grpc/README.md, and the architecture notes:

The table below covers the four full-workload topology-mutation chaos fixtures (Tests 1-4). The atomic-write reader-isolation (Test 5), atomic-visibility-across-topology siblings (Test 6), digest-determinism (Test 7), cross-cluster atomic-visibility (Test 8), per-mode convergence chaos (Test 9), multi-site smoke (Test 10), and the per-test invariant fixtures (range delete, CAS, scan cancel, multi-silo restart, WAL trim, liveness + inbound stats, OR-Map convergence, compaction + shipping, gRPC transport, Azure Table WAL) target orthogonal invariants and are documented in their own grids below.

Surface Happy path Faults Resize Reshard
Concurrent reads/writes during split shadow phase ✅ ✅ - ✅
ScanKeysAsync / ScanEntriesAsync in-line reconciliation ✅ ✅ ✅ ✅
CountAsync per-slot routing + version stability + bounded retry ✅ ✅ ✅ ✅
StaleShardRoutingException transparent retry ✅ ✅ - ✅
StaleTreeRoutingException transparent retry across alias swap - - ✅ -
Permanent MovedAwaySlots rejection after split completion ✅ ✅ - ✅
Resumable SplitInProgress intent replay across crashes - ✅ - -
Two-phase root promotion (PendingPromotion) replay - ✅ - -
Shadow MergeManyAsync atomicity under failed source write - ✅ - -
Registry ShardMap.Version stamping under retry - ✅ - ✅
Idempotent drain chunks - ✅ ✅ ✅
TreeResizeGrain phase machine under live traffic - - ✅ -
Per-source-shard shadow-forwarding under live traffic - - ✅ -
TreeReshardGrain migration loop + dispatch-budget clamping - - - ✅

Saga-atomicity surfaces (Tests 5, 6, 8)

Surface Atomic vis. quiescent Split + saga Resize + saga Reshard + saga Cross-cluster
Continuous reader observes zero-or-all keys per poll ✅ ✅ ✅ ✅ ✅
AtomicWriteGrain per-shard terminal broadcast idempotent under retry ✅ ✅ ✅ ✅ ✅
Prepared-write commit path holds keys hidden until terminal arrives ✅ ✅ ✅ ✅ ✅
Saga prepares shadow-forwarded onto destination shard mid-split - ✅ - ✅ -
Saga prepares shadow-forwarded onto destination physical tree mid-resize - - ✅ - -
Retroactive prepared-mutation sweep at BeginShadowWrite - - - ✅ -
Saga terminal-fan-out shadow-forward fallback via post-Complete MovedAwaySlots - - - ✅ -
Registry TxDecisionRetention tombstone absorbs duplicate terminals - - - ✅ ✅
Receiver-side prepared/terminal apply seam holds prepares off projection - - - - ✅
LatticeHlcOverrideContext preserves source HLC on receiver - - - - ✅
ShouldShip bypass for TxCommit / TxAbort records - - - - ✅

Per-mode convergence surfaces (Test 9)

Surface LWW OR-Set PN-Counter MV-Register
Producer-side change feed yields locally-authored mutations with origin id ✅ ✅ ✅ ✅
Shipper drives delivery to every peer site under partition-cycled topology ✅ ✅ ✅ ✅
Receiver-side mode-specific dispatch (source-HLC LWW apply / typed-delta fold) ✅ ✅ ✅ ✅
LatticeOriginContext flagging foreign-origin writes during apply ✅ ✅ ✅ ✅
Commutative-monoid CRDT merge absorbs out-of-order receive - ✅ ✅ ✅
Dot-context supersession of causally-dominated entries on merge - - - ✅

Per-test invariant surfaces (range delete, CAS, scan cancel, restart, replication, storage)

The chaos tests below target surfaces that the four full-workload topology grids above do not cover: per-call public-API invariants (range delete, CAS, scan cancellation), Orleans membership churn (multi-silo restart), the production replication pipeline (WAL trim, liveness + inbound stats, OR-Map convergence, compaction + shipping), and the two downstream-package suites (gRPC transport, Azure Table WAL). The range-delete, CAS, scan-cancel and multi-silo-restart columns map to rows of the core suite table at the top of this document; the WAL-trim, liveness, OR-Map, compaction, gRPC and Azure Table WAL columns map to fixtures catalogued in the replication chaos tests.

Surface Range delete CAS Scan cancel Multi-silo restart WAL trim Liveness + inbound OR-Map Compaction + shipping gRPC transport Azure Table WAL
DeleteRangeAsync range exclusivity under concurrent writers ✅ - - - - - - - - -
DeleteRangeAsync cross-shard fan-out + tombstone scope ✅ - - - - - - - - -
SetIfVersionAsync (CAS) linearisable winner under contention - ✅ - - - - - - - -
SetIfVersionAsync (CAS) lost update returns false, and the re-read-and-retry loop loses no increment - ✅ - - - - - - - -
ScanKeysAsync / ScanEntriesAsync cooperative cancellation surfaces OperationCanceledException - - ✅ - - - - - - -
Cancelled scan leaks no grain-side enumerator state - - ✅ - - - - - - -
TestCluster.RestartSiloAsync mid-workload preserves universe (count, envelope) - - - ✅ - - - - - -
Process-shared IGrainStorage isolates membership churn from storage disappearance - - - ✅ - - - - - -
Producer-side WAL trim cannot prune un-acked entries - - - - ✅ - - - - -
Real AddLatticeReplication + loopback transport under sustained writes - - - - ✅ ✅ - ✅ - -
Outbound liveness probe: peer.last_contact_seconds climbs while a peer edge is isolated and falls below half a probe interval within ten probe intervals of the heal - - - - - ✅ - - - -
Budget-driven receiver-side apply faults drain in full, the receiver still converges, and the inbound peer-stats row records contact with no outbound-only backlog or in-flight counts - - - - - ✅ - - - -
OR-Map convergence under concurrent multi-site mutation + partition - - - - - - ✅ - - -
Per-tree typed-CRDT shape resolution end-to-end on producer + receiver dispatch - - - - - - ✅ - - -
ReplicationShipperGrain.ShouldShip keeps MutationKind.Tombstone envelopes off the wire - - - - - - - ✅ - -
ITombstoneCompactionGrain.RunCompactionPassAsync mid-shipment preserves receiver convergence - - - - - - - ✅ - -
gRPC sender retry loop under 15% and 30% per-call channel faults delivers every attempted entry - - - - - - - - ✅ -
Re-deliveries the channel faults induce are absorbed as no-ops by the receiver's exact-identity dedupe cache or the idempotent leaf apply - - - - - - - - ✅ -
Azurite-backed WAL keeps every shard's offsets dense, gap-free and duplicate-free under sustained concurrent appends across shards - - - - - - - - - ✅

Legend: ✅ = covered by a live test.

Runtime characteristics

Every chaos suite shares the same shape: a short chaos window (a few seconds of sustained workload with topology churn and / or fault injection), a bounded heal / drain phase, and a post-quiescence invariant assertion. Per-suite cost scales with the workload described in each suite's "What it proves" / "Purpose" column rather than being fixed, so this section stays accurate as suites are added without a per-suite runtime table to maintain.

  • Single-cluster suites (test/lattice/BPlusTree/) run key universes of 8 to 600 keys (16 keys for the saga-atomicity suites) across up to ~16 parallel workers - writers, scanners, and a topology or saga driver - with a chaos window of roughly 6-20 s (up to 90 s for the shard-consolidation suite) and a heal / drain budget up to ~60 s for the saga-atomicity suites. Most trees start at 4 shards (64 where a fixture does not pre-register its tree; the multi-silo restart and bounded-cache fixtures pin 2 and 1) and grow under split or reshard; the shrinking reshard run folds its tree from 4 shards to 2.
  • Cross-cluster suites (test/lattice.replication/Chaos/) run two or three in-process clusters over a fault-injectable inter-site delivery layer, with a single-key or single-tree universe (the atomic-visibility suite uses ~72 keys), a chaos window of a few seconds with a site or edge partitioned mid-workload, and a drain budget up to ~60 s. See the replication chaos tests for the per-suite catalog.

The cross-cluster, gRPC, and Azure Table suites drive the real replication pipeline (and the real gRPC transport service and Azure Table provider where applicable) rather than test doubles of that logic, but they run in-process: in-process test clusters with a simulated delivery pump, an in-memory test server, and the Azurite emulator stand in for networked silos and a real cloud storage account.

Derived-state recovery across an identity swap

Derived state that tails a source tree's write-ahead log - a folded / materialised view, and a tag index maintained in a sibling index tree - must recover when the source's PHYSICAL identity is repointed under its logical registry alias (a restore-style cutover, possibly repeated). Two chaos suites, each hosted in the sibling package whose fixture already stands up the machinery, cover this and are tracked by issue #1167:

Suite Where What it proves
Materialised-view recovery across an identity swap test/lattice.replication/Chaos/DerivedStateRecoveryAcrossIdentitySwapChaosTests.cs A folded view tails a logical source tree whose physical identity is repointed under its registry alias repeatedly, under a sustained mutation workload that accretes backlog between drains. The maintainer rebinds to the source's current physical id event-driven when the tree registry pushes an alias change (with a coarse backstop re-resolve as a safety net), and on a change rebuilds against the new physical source and rebinds its tail. After the workload quiesces the view reflects exactly the final identity's contents: keys dropped by a cutover are retracted (a WAL tail alone can never retract a key the restored source never had), changed values win, and a large abandoned backlog never survives. Complements the deterministic single-swap regression that landed with the maintainer heal.
Tag-index reconcile under repeated restore test/lattice.backup/Chaos/BackupRestoreReconcileChaosTests.cs A tag index over a live subject tree is driven through a real shadow-cutover restore under a large tagged working set with a concurrent reader hammering the tag query. The restore fires a prompt reconcile that drops every membership row absent from the restored point-in-time; the reader never observes an out-of-universe key or a torn membership. A second case injects one aborted catalogue enumeration into the restore's tag-index discovery and requires the restore to recover that scan rather than swallow the abort and leave every orphan row in place (issue #3233). Repeated restores to successively earlier point-in-times narrow membership monotonically to each restore's subject with no row stranded from a superseded restore.

Routing and cross-tree recovery across a shadow-cutover restore

A shadow-cutover restore swaps a logical tree's alias to a freshly loaded shadow tree while retaining the previous physical tree so the restore can be reverted. Two consumers must recover across that swap: stateless-worker routing activations that cache the logical-to-physical alias, and an in-flight cross-tree atomic-write saga that drives a participant through that alias. Two chaos suites, hosted in the backup package whose fixture stands up both the restore machinery and the core lattice, cover this and are tracked by issue #1167:

Suite Where What it proves
Routing self-heal across a shadow-cutover test/lattice.backup/Chaos/ShadowCutoverRoutingSelfHealChaosTests.cs Many stateless-worker routing activations are warmed against a tree's pre-cutover identity under sustained concurrent reads, then the tree is cut over to a restored shadow while the readers keep hammering. The retained tree refuses logical-alias traffic with the internal staleness signal, which the routing tier catches and re-resolves onto the shadow. No reader ever surfaces that signal or observes an out-of-universe value, and after the cutover every activation converges onto the restored snapshot with no key left resolving to pre-cutover content. The repeated-cutover case restores to successively earlier point-in-times and proves the heal re-arms for each.
Cross-tree atomic write across a participant cutover test/lattice.backup/Chaos/CrossTreeAtomicWriteAcrossCutoverChaosTests.cs A sustained stream of cross-tree atomic writes spans two trees while one participant is repeatedly cut over to a restored shadow underneath the saga. The saga's deadline-bounded retry loops must absorb the staleness signal the retired physical tree raises, re-resolve onto the shadow, and reach a clean terminal decision - no call leaks the signal to the caller. After the cutover storm a fresh all-or-nothing batch still commits and is atomically visible on both trees, proving the cut-over participant self-healed and is not wedged. Complements the mocked prepare-side and terminal-broadcast stale-routing retry regressions by exercising the whole coordinator over a real restore.

See also

  • Consistency - the per-operation guarantees these tests verify against the public API under topology mutation.
  • Replication - the producer-side change-feed → shipper → receiver-side applier pipeline that the cross-cluster suite exercises.
  • Adaptive Shard Splitting - the split protocol exercised by the happy-path, faults, reshard, and split + saga tests.
  • Online Reshard - the reshard coordinator and its interaction with autonomic splits.
  • Tree Sizing - the online resize path exercised by ChaosResizeIntegrationTests and ResizeTopologyTests.
  • Architecture - grain layers, root promotion, bounded retry, and the invariants the chaos tests verify.
  • State Primitives - HLC and LWW, which guarantee the value envelope holds even under concurrent rewrites.