Storage pressure
This page documents Orleans.Lattice.Scaling 9.9.0, in the documentation for Orleans.Lattice 9.9.0 (release line 9.9), built 2026-10-04. It is also published as markdown, with every table and list, at storage-pressure.md, and llms.txt lists every page.The storage axis reports whether any write-ahead-log storage account is hot and,
when one is, recommends spreading its partitions across more accounts. It is
signal-only: acting on a recommendation means calling the ILatticeAdmin
WAL-move surface yourself. Nothing on this axis ever changes the compute
ScaleValue.
What it measures
StoragePressure is a cluster-aggregate snapshot with:
OverThreshold-truewhen budgeted retained WAL bytes have crossed the configured threshold (the advisory fraction of the summed per-treeWalMaxRetainedBytesceilings). Each tree contributes its own resolved ceiling to that sum rather than one silo-wide value, and the comparison is made once over the summed bytes and ceilings, not tree by tree; a tree with no ceiling contributes neither bytes nor budget to the comparison.WalRetainedBytes- total retained WAL bytes across every catalogue key.Accounts- aWalAccountPressureperIWalStorageProviderCatalogkey that backs a WAL partition (nevernull; empty when nothing is tracked).Recommendation- an optionalWalRebalanceRecommendation, ornullwhen no rebalance is warranted orStorageRecommendationsEnabledisfalse.
Throughput-bound versus capacity-bound
Each account is classified by WalPressureClassification, because the two kinds
of pressure have different remedies:
| Classification | What it means | The remedy |
|---|---|---|
None |
Healthy: neither backend-saturated nor over the retained-bytes threshold. | Nothing to do. |
ThroughputBound |
A single hot account has topped out its backend write rate (its per-tree WalSaturationState is Throttled or Saturated, in practice around 22-24 thousand entries per second for one storage account) continuously for AccountSaturationWindow. |
Spread the account's hot partitions across more accounts (a WAL move). Adding retention headroom does not help. |
CapacityBound |
Retained WAL bytes have grown past RetainedBytesAdvisoryRatio of the account's budget - the sum of the effective per-tree WalMaxRetainedBytes ceilings of the trees holding partitions there. |
Reclaim retained bytes or provision more retention. Spreading throughput does not help. |
An account that is both saturated for the window and over its retained-bytes
threshold is classified ThroughputBound; its OverThreshold flag still reads
true.
WalAccountPressure.OverThreshold is the capacity-bound trigger specifically -
true when that account's budgeted retained bytes crossed the advisory fraction
of its budget. The budget is the sum of the effective per-tree ceilings
attributed to that account, so each tree's own ceiling counts rather than one
silo-wide number: trees on different accounts cross at different points, while
trees sharing an account are weighed together against their summed budget.
The rebalance recommendation
When a hot account is found and recommendations are enabled, the collector emits
one WalRebalanceRecommendation naming:
TreeandPartition- the WAL partition to relocate.CurrentProviderKey- the catalogue key that backs it today.TargetProviderKey- a registered key with spare headroom to accept it, whenHasHeadroomistrue.HasHeadroom-falsewhen every registered key is already hot; thenTargetProviderKeyis empty and the remedy is to provision or register another account before any move can help.Rationale- a human-readable explanation of why the move is recommended.Classification- whether the current account is throughput-bound or capacity-bound, so you know whether a move (spread throughput) or added retention will actually help.
Acting on a recommendation
The collector never moves anything. To act on a recommendation, drive the
ILatticeAdmin move workflow:
using System.Threading;
using Orleans.Lattice;
using Orleans.Lattice.Scaling;
async Task RebalanceAsync(
ILatticeScalingSignal signal,
ILatticeAdmin admin,
CancellationToken cancellationToken)
{
ScalingSignal current = await signal.GetScalingSignalAsync(cancellationToken);
if (current.Storage.Recommendation is not { HasHeadroom: true } recommendation)
{
return; // nothing actionable, or every account is already hot
}
// 1. Plan the move to the recommended target account.
_ = await admin.PlanWalMoveAsync(
recommendation.Tree,
recommendation.Partition,
recommendation.TargetProviderKey,
cancellationToken);
// 2. Execute it.
_ = await admin.ExecuteWalMoveAsync(
recommendation.Tree,
recommendation.Partition,
recommendation.TargetProviderKey,
options: null,
cancellationToken);
// 3. Later, reclaim the vacated source once the move has settled.
_ = await admin.ReclaimMovedWalSourceAsync(
recommendation.Tree,
recommendation.Partition,
recommendation.CurrentProviderKey,
cancellationToken);
}
When HasHeadroom is false, register another WAL storage account under a new
key with AddLatticeWalStorageProvider on every silo first - the
IWalStorageProviderCatalog has no runtime registration method and lists the
keys registered at startup, and every silo must register the same set - then
re-read the signal: the next recommendation will name the new account as the
target.
Turning it off
Set StorageRecommendationsEnabled = false to keep the OverThreshold flag and
the per-account breakdown while suppressing the Recommendation (it stays
null). This is useful when an external system owns WAL placement and you only
want the storage axis for observability.