Table of Contents

Chaos tests

The Azure Table WAL package has a focused chaos suite that exercises the real AzureTableWalStorageProvider against an Azurite-backed Azure Table endpoint. It complements the core chaos tests and the replication chaos tests by proving the durable WAL backend preserves its storage invariants under concurrent append pressure.

Every suite here is tagged [Category("Chaos")]; the Azure-backed suite is also tagged [Category("AzureStorageEmulator")].

dotnet test --filter "TestCategory!=Chaos"

Azure Table WAL suite (test/lattice.storage.azuretable/Chaos/)

AzureTableWalChaosTests drives concurrent append load across multiple shards against a real Azurite endpoint. The suite creates an isolated table per run, appends fixed-size batches from one writer per shard, reads each shard back after the workload, and verifies the WAL invariants that recovery and materialization rely on.

Suite What it proves
Sustained concurrent appends across shards Parallel shard writers preserve dense per-shard offset namespaces, no duplicate offsets, monotone read order, and the expected highest offset after every append batch has completed.

Runtime characteristics

Property Azure Table WAL suite
Backend Real Azurite emulator via AzureTableWalStorageProvider
Shards 6
Workload 10 batches per shard, 4 entries per batch
Compression Off (Compression = LatticeCompression.None), so payloads are stored verbatim and the default Zstandard path is not exercised
Total entries 240
Writers One writer per shard, all shards active concurrently
Validation Full readback, duplicate detection, gap detection, monotone offset assertion, highest-offset assertion
Visibility barrier Drains outstanding commit completions after the write window and before readback, so the invariants are asserted against a quiesced shard rather than against a batch still in flight
Skip behaviour Calls Assert.Inconclusive when the default Azurite development endpoint is not reachable

The workload runs on the shipping commit-pipeline defaults, under which an append can return before its own commit completion lands. The suite therefore drains the outstanding completions through the provider's flush barrier before reading back, so it keeps covering the pipelined path without asserting read-after-write that the default does not promise. See Architecture.

The suite intentionally uses the public provider surface rather than a fake backend. It does not require the full replication pipeline; the goal is to isolate the Azure Table WAL storage contract: append-batch atomicity, dense offset preservation, ordered readback, and correct tail reporting.

Running locally

Start Azurite on the default development endpoint, then run the chaos category for the Azure Table test project:

dotnet test test\lattice.storage.azuretable\Orleans.Lattice.Storage.AzureTable.Tests.csproj --filter "TestCategory=Chaos"

If Azurite is not reachable, the suite reports inconclusive instead of failing. This keeps normal developer machines from failing only because the emulator is absent - but it is also a false-green trap: NUnit counts an inconclusive result as neither passed, nor failed, nor skipped, so the console banner still reads Passed! with Skipped: 0 and only the Total / Passed counts drop. Confirm the suite actually executed from those counts rather than from the banner (see Starting Azurite). In CI every test leg runs an Azurite service container, and a package shard that executes zero tests fails the build rather than being reported as passed.

See also

  • Architecture - storage layout, append pipeline, and recovery behaviour under test.
  • Configuration - options that affect retry, pipelining, completion timeout, saturation handling, and compression.
  • Core WAL Storage Providers - public provider seam and durable backend catalogue.
  • WAL tuning - provider pressure, batching, and saturation envelope.
  • Replication chaos tests - cross-cluster and transport chaos suites that can use this provider as the durable WAL backend.