Orleans.Lattice.Api.Backup.Grpc architecture

This page describes the code-first gRPC binding and its two-layer, fail-closed authorization model. The gRPC service, method definitions, marshallers, interceptor, and the default header credential bridge / options auth-scheme source are internal and are described here by behaviour; the public client, options, and seams are named.

Code-first binding

The binding defines its RPCs in C# rather than a .proto. A method-definition singleton builds one gRPC Method per remote operation from the Orleans serializers resolved out of DI, under the service name orleans.lattice.api.backup. The server-side service and the public LatticeBackupApiGrpcClient share those definitions, so the wire contract is identical on both ends by construction and there is no generated stub to keep in sync. Some facade and engine operations remain in-process-only and have no RPC, including inventory, catalog rebuild / scrub, cold restore, and backup-set restore.

Every wire message is marshalled with the Orleans binary serializer. Most RPCs use the package's own [GenerateSerializer] request / response records, which wrap facade DTOs where needed; three reuse facade types directly - ListBackups takes a BackupCatalogRequest and returns a BackupCatalogPage, StreamBackups streams BackupManifest messages, and ProbeCapabilities returns a BackupScopeCapabilities. That is why the client's Create factory takes an IServiceProvider with AddSerializer() registered: the per-message marshallers are built from those serializers, so a client and server that share the Orleans serialization configuration cannot disagree on the wire format.

The operations map to two gRPC shapes: unary for capture, backup-set capture, list, describe, delete, restore, revert, auth-scheme discovery, scheduling, scope status, capability, and health operations; server-streaming for StreamBackups (whole-catalog drain) and ExportArtifact (chunk-wise artifact export). The two streaming RPCs are what let a large catalog or artifact move with bounded memory end to end - the facade streams, the service forwards each item as it arrives, and the client re-exposes it as an IAsyncEnumerable<T>.

Two-layer authorization

Every protected call passes through two independent, fail-closed gates. The unauthenticated GetAuthScheme discovery RPC is exempt from the transport gate so a client can learn how to sign in before it holds a credential.

1. Transport meta-authorizer

An authorization interceptor runs first, before the facade is touched for protected calls. It decodes the inbound call into a LatticeBackupApiAuthorizationContext - the LatticeBackupApiOperation, an optional TargetId (the backup id for a call keyed by one, the scope tree id for a single-scope capture, schedule, or scope-status call, and null for the whole-catalog, backup-set, capability-probe, and availability calls - see the API reference), and the underlying ServerCallContext for header and peer inspection - and asks the registered ILatticeBackupApiAuthorizer whether the call may run at all. It defaults to DenyAllBackupApiAuthorizer, so every protected call is rejected with PermissionDenied until the host registers a permissive authorizer (or the opt-in AllowAllBackupApiAuthorizer) or sets RequireAuthorization to false. The interceptor is registered globally but scopes its enforcement to the backup control-API service by service-name prefix, so other gRPC services on the same host are unaffected.

An operation the interceptor does not recognise is presented to the authorizer as Unknown rather than being waved through, so a deny-by-default policy refuses a future or unmapped RPC instead of having it masquerade as a benign catalog read. The ProbeCapabilities RPC currently has no entry in the operation map, so it too reaches the authorizer as Unknown.

2. Facade scope authorization

Once past the transport gate, the service stamps the caller identity onto the ambient Lattice credential context - via the ILatticeBackupApiCredentialBridge, whose built-in default reads the configured bearer-style header - and invokes the control facade. The facade then authorizes the operation's scope against the backup access gate, exactly as an in-process facade caller would. When auth-backed backup control is active, an anonymous caller (no resolvable credential) is denied here even when the transport gate allowed the call.

The two gates are complementary, not redundant: the transport gate is a coarse edge control keyed by headers, operation, and target, while the facade gate is the engine's own fine-grained, per-scope, fail-closed authorization. A deployment can run a permissive transport gate behind a trusted boundary and still get full per-scope enforcement from the facade, or tighten both.

Credential bridging

The credential bridge is the identity seam. The default implementation reads a single configurable header (CredentialHeaderName, default authorization), strips a case-insensitive scheme prefix (CredentialScheme, default Bearer), and lifts the remaining token onto the ambient LatticeCredential for the registered credential authenticator to resolve into a subject. A host with a bespoke identity source - a client TLS certificate, a signed edge header, a pre-resolved principal - registers its own bridge before the binding's registration runs, and the built-in default steps aside. Returning null leaves the caller anonymous; when auth-backed backup control is active an anonymous caller is denied, so a missing or malformed credential header can never drive a destructive operation.

When the authorization add-on is not registered, the default bridge still reads and bridges the header, but the core no-op access gate ignores the credential, so the backup control API behaves exactly as it does without a credential layer.

Auth-scheme discovery

GetAuthScheme is deliberately unauthenticated so a client can discover how to sign in before it holds a credential. The response is built by the ILatticeBackupApiAuthSchemeSource; the default implementation projects the host-configured AdvertisedAuthSchemes (empty by default, so a client falls back to manual or Basic selection). Because the advertisement is served without a credential, an implementation must return only public configuration - scheme ids and public parameters like an OIDC authority or client id - and never a secret or user-specific data.

Status mapping

A denial from either gate reaches the client as a PermissionDenied RpcException, so a caller handles authorization failure uniformly regardless of which layer refused the call.

The binding also translates the facade's other failure shapes into stable gRPC status codes, so a client can branch on the code rather than parse a message:

Facade outcome	gRPC `StatusCode`	Notes
Authorization denied (transport gate or facade scope check), a fail-closed tenant resolution (`LatticeTenantAccessDeniedException`), or the tenant-isolation boundary refusing a tree outside the caller's namespace (`LatticeBackupTenantIsolationException`)	`PermissionDenied`	The detail is safe to surface; never names a secret.
Unknown backup / missing id (`KeyNotFoundException`)	`NotFound`
Restore pre-apply validation failure (`LatticeRestoreValidationException`)	`FailedPrecondition`	A missing manifest or artifact, a digest mismatch, an out-of-scope request, or a coordinated (replicated-tree) restore that was refused before it started - infeasible on the coordinator cluster, or a peer unreachable - or that aborted because a peer could not prepare. The detail names the backups, trees, artifacts, or peer clusters involved - an infeasible-capacity refusal names the refusal, not the target's stored size or shard count - so an operator UI can render it directly.
Invalid argument (`ArgumentException`)	`InvalidArgument`
A fault whose exception chain holds a `TimeoutException` or the reminder service's still-initializing signal - typically a transient reminder-registry or storage failure that outlived the scheduler's bounded retry, though any timed-out call on a unary RPC maps here	`Unavailable`	Retryable. The message carries a correlation id tied to the logged server exception, never the exception's own detail.
Request cancelled	`Cancelled`	On the two server-streaming RPCs a cancelled call simply ends the stream instead.
Any other fault	`Internal`	The detail is deliberately opaque (a correlation id at most); the real exception is logged server-side, not returned.

That last row also catches two refusals of a shadow-cutover restore or a revert that this binding does not map specifically: the InvalidOperationException raised while the target tree is deleted, a delete of it is pending, or another alias change holds its alias reservation, and the LatticeTreeOwnershipDeniedException raised when the registered tree ownership guard refuses the alias swap. Both reach the client as Internal with an opaque message, and their reason is only in the server log.

The FailedPrecondition shape is the one an operator most often needs to act on: a coordinated (replicated-tree) restore fails this way when the backup store is not actually shared across every cluster, because a peer that never captured the backup cannot resolve it. The detail is preserved end to end so a UI can turn it into a clear, fixable message rather than an opaque error.

Table of Contents