Orleans.Lattice.Backup

Causally-consistent backup and restore for Orleans.Lattice.

What is it?

Orleans.Lattice.Backup captures a named, timestamped, point-in-time backup of a selected scope of a lattice tree and restores it later without losing a single bit of causal metadata. It builds on the core snapshot, write-ahead-log, and merge machinery and adds:

Full capture - a scoped snapshot exported through the core zero-observable-writes cursor, with a self-describing manifest that records the consistency cut, the shard topology, a per-key shape and merge-mode map, per-origin provenance high-water marks, and an optional compression-dictionary reference.
Incremental capture - a forward write-ahead-log differential layered on a base backup, resuming from the base backup's per-partition WAL offsets, and falling back to a full capture when the resume point has been trimmed off the WAL, a range delete surfaces in the delta window, or the base chain was captured on a different cluster.
Restore - a mode-faithful replay that reinstalls every entry's hybrid-logical-clock, version vector, origin cluster id, expiry, and tombstone flag exactly as captured, either in place (a bulk-load into a never-registered tree restored from a single whole-tree full backup with no narrower sub-scope, otherwise a last-writer-wins merge into whatever the target holds) or via an atomic shadow-cutover to a fresh tree. Every artifact is validated against its content digest before anything is applied, and a restore is idempotent under retry.
Scheduling and retention - opt-in recurring full and incremental schedules per scope, and a chain-aware retention policy that never prunes the base chain of a retained increment.
A pluggable sink - the storage surface a backup is written to and read from, defaulting to an in-cluster dogfooded tree, with a durable Azure Blob Storage sink shipped as a sibling package.
A cross-tree causal fence - an opt-in fence for a multi-tree backup set that waits for in-flight cross-tree atomic writes to drain before the members are captured, so a cross-tree atomic write is never torn across the set boundary. Each member is still captured by its own snapshot, one tree after another, so the fence is a quiescence window rather than a single point-in-time cut across the trees.

The package registers the storage and engine surface; the Orleans.Lattice.Api.Backup facade and its gRPC binding add the remotely-drivable control plane.

Core properties

Causally faithful. A restore preserves the captured history verbatim: entries replay through the HLC-preserving last-writer-wins merge and bulk-load seams, so a restored tree converges identically to the source.
Point-in-time isolation. A capture rides the core snapshot cursor for a stable read, inherits the core snapshot shedding and replay-budget behaviour, and fails fast when the in-scope size would exceed the replay budget.
Fail-closed authorization. Every capture and restore authorizes its scope before touching data, through the same access gate the data path uses, against a dedicated Backup (capture) or Restore (author / bulk-load) capability; the list, describe, and delete operations of the control facade authorize each manifest's scope the same way. The engine's own catalog-store and sink seams run under system origin and perform no authorization check. A restore governs both the trees it names: Restore over the tree written into, and - only when it retargets the backup onto a different tree - Backup over the tree the manifest was captured from, so a cross-tree restore can never materialize a tree the caller holds nothing on.
Idempotent by construction. A backup's id is the SHA-256 content address of its captured payload (an incremental's id also folds in its base backup id), so a retried capture that produces identical bytes re-registers the same backup - keeping its original capture time - rather than creating a duplicate; registering the same manifest twice, or re-running the same restore, converges without duplication. Artifact ids are not content-addressed: each capture streams its payload to a fresh per-capture artifact id, which the re-registered manifest then references.
Opt-in and hidden. Registering the package installs the storage surface but schedules no capture or retention until an operator opts in; the periodic backup-health monitor is on by default, but stays inert unless a durable sink is registered. The catalog and store live in reserved sys-backup-* trees that inherit the core sys- catalog-hiding filter, so the backup surface is the sole enumeration point for backups.

Features

Feature	Surface	Summary
Full capture	`ILatticeBackupCaptureService.CaptureAsync`	Scoped point-in-time snapshot registered as a `BackupManifest`.
Backup set	`ILatticeBackupCaptureService.CaptureSetAsync`	One full backup per scope under a single set manifest, optionally cross-tree consistent.
Tracked operations	`ILatticeBackupOperations.StartBackupAsync` and related facade verbs	Accept-then-poll backup and restore with durable status, progress units, cancellation, and result maps.
Incremental capture	`ILatticeBackupIncrementalCaptureService.CaptureIncrementalAsync`	Forward-WAL differential layered on a base backup, with full-capture fallback.
Restore	`ILatticeBackupRestoreService.RestoreAsync`	Mode-faithful, validated, idempotent replay of a manifest chain.
Revert	`ILatticeBackupRestoreService.RevertRestoreAsync`	Undoes a shadow-cutover restore by swapping the registry alias back.
Backup-set restore	`ILatticeBackupRestoreService.RestoreSetAsync`	Restores every member tree of a multi-tree backup set as one unit via shadow-cutover, as a single all-or-nothing coordinated saga when any member is replicated.
Trigger / schedule / prune	`ILatticeBackupScheduler`	On-demand triggers, recurring schedules, and chain-aware retention per scope.
Catalog	`ILatticeBackupCatalogStore`	Durable, introspectable index of manifests keyed by backup id.
Catalog rebuild / scrub	`ILatticeBackupCatalogRebuildService.RebuildFromSinkAsync` / `ILatticeBackupCatalogScrubService.ScrubAsync` (on the control facade: `ILatticeBackupControl.RebuildCatalogFromSinkAsync` / `ScrubCatalogAgainstSinkAsync`)	Re-derive the catalog from the sink, or reconcile and prune rows whose sink payload is gone.
Cold restore	`ILatticeBackupColdRestoreService.ColdRestoreAsync` (on the control facade: `ILatticeBackupOperations.StartColdRestoreAsync`)	Restore into a fresh cluster from the sink alone, with no surviving catalog.
Health monitoring	`ILatticeBackupHealthService` / `ILatticeBackupControl` health ops	Periodic presence + content-hash verification of each backup's durable sink payload, gated on a durable sink.
Sink	`ILatticeBackupSink`	Pluggable streamed-artifact + manifest storage.
Reserved-namespace guard	`LatticeBackupReservedTrees`	Lets an application validate its own tree ids against the reserved `sys-backup-*` namespace.
Replication and tenancy seams	`IRestoreSagaDispatcher`, `IReplicatedTreeMembership`, `IBackupSinkSharingProbe`, `ILatticeBackupTenantScope`	Inert by default; the replication package supplies the coordinated-restore dispatch, the replicated-tree membership, and the cross-cluster sink-sharing probe, and the tenancy add-on confines capture and restore to the active tenant's namespace and quota.
Observability	`BackupMetrics` / `LatticeBackupMetrics`	A dedicated `orleans.lattice.backup` meter for space, throughput, failures, and inventory.

Quick Start

using Orleans.Lattice;
using Orleans.Lattice.Backup;

siloBuilder
    .AddLattice((silo, storageName) =>
    {
        // Configure the storage provider named by storageName.
    })
    .AddLatticeBackup(options =>
    {
        // Durable per-key history on the catalog tree is on by default;
        // widen the cross-tree-set fence drain budget if needed.
        options.CrossTreeFenceDrainTimeout = TimeSpan.FromSeconds(45);
    });

Capture and restore through the registered services:

using Microsoft.Extensions.DependencyInjection;
using Orleans.Lattice.Backup;

IServiceProvider serviceProvider = null!;
var captureService = serviceProvider.GetRequiredService<ILatticeBackupCaptureService>();
var restoreService = serviceProvider.GetRequiredService<ILatticeBackupRestoreService>();

// Capture a full backup of a whole tree.
var capture = await captureService.CaptureAsync(
    new LatticeBackupCaptureRequest("nightly", BackupScopeSelector.WholeTree("orders")),
    cancellationToken);

// Restore it later into a fresh tree via an atomic shadow-cutover.
await restoreService.RestoreAsync(
    new LatticeRestoreRequest(
        capture.BackupId,
        targetTreeId: "orders",
        mode: LatticeRestoreMode.ShadowCutover),
    cancellationToken);

Enable a recurring schedule and retention for a scope (opt-in; everything is disabled by default):

using Orleans.Lattice.Backup;

siloBuilder.ConfigureLatticeBackupSchedule("orders-scope-key", options =>
{
    options.FullBackupScheduleEnabled = true;
    options.FullBackupInterval = TimeSpan.FromHours(6);
    options.IncrementalBackupScheduleEnabled = true;
    options.RetentionEnabled = true;
    options.RetentionKeepLast = 30;
});

The scope key passed to ConfigureLatticeBackupSchedule is the value returned by BackupScopeKey.For(scope). Configuring the schedule alone registers no reminder: call ILatticeBackupScheduler.EnsureScheduleAsync(scope) to register (or update) the scope's schedule reminders from these options.

Migration note

The backup API facade now exposes accept-then-poll backup and restore operations with progress. Prefer ILatticeBackupOperations.StartBackupAsync, StartIncrementalBackupAsync, StartBackupSetAsync, StartRestoreAsync, StartColdRestoreAsync, StartBackupHealthCheckAsync, StartCatalogRebuildAsync, and StartCatalogScrubAsync for operator surfaces. The older blocking facade verbs are deprecated and will be removed in the next major version; see Backup operations. The engine service interfaces in this package (ILatticeBackupCaptureService, ILatticeBackupRestoreService, ILatticeBackupColdRestoreService, and related seams) are not deprecated.

Reference

API reference - every public type and member, by name, with signatures.
Configuration - every public options property, its type, and its default.
Architecture - the capture, incremental, restore, scheduling, and sink pipelines and the core seams they attach to.
Disaster recovery - the sink-is-truth model, catalog rebuild and scrub, cold restore into a fresh cluster, and periodic health monitoring.
Observability - the orleans.lattice.backup meter and its instruments.

Table of Contents