Table of Contents

Index source strategies

Part of Container quickstart.

Where a repository's content comes from is a per-repository choice between two strategies.

The mounted workspace is the default and is what every section above describes: a client registers a path under LATTICE_WORKSPACE_ROOT and the background reconcile walks that tree. The git source is opt-in and hub-only: the host is told a remote url and a ref, fetches it into a staging work tree, and indexes the commit that ref resolved to. The two are mutually exclusive per repository - a git-sourced repository is refused by repocontext_add_repo with a clear error, so a mount can never silently shadow the configured remote.

Mounted workspace (default) Git source (opt-in)
Where the truth lives Outside the host: whoever mounts the volume decides what is indexed, and two hosts can mount divergent content. In the host's own configuration - a remote url plus a ref - so the declared truth is verifiable and identical everywhere it is deployed.
What a generation is anchored to Nothing. "Which revision am I serving?" has no answer. The resolved commit SHA, reported by repocontext_list_repos as indexedCommit.
How the change set is computed A directory walk with modification-time pruning plus a periodic full sweep. A diff of the new commit's tree against the stored per-file digests. No walk.
How a delete is detected Inferred from absence on disk, so an unmounted or half-synced volume looks like a mass deletion. Read exactly from the commit's change set.
What it needs A read-only bind mount. Reach to a git remote, plus credentials unless the remote is anonymous.
What a pass costs A stat of every file in every directory the prune cache cannot skip, on every reconcile. No network, and no second copy of the tree. A shallow fetch and a SHA comparison. A refresh that finds the ref unmoved does no walk, no read, and no write at all - but the staging work tree means the repository is on disk twice.
How fresh it is Whatever is on the volume right now, uncommitted work included, within the reconcile bound. The tracked ref as last fetched. Work that is uncommitted, or committed but not pushed to that remote, does not exist to it.
What it serves Any content: a local dev loop, non-git trees, air-gapped hosts, and work in progress. Any reachable git remote at a committed ref - a hosted forge, or a bare repository on local disk.
Cluster role Any. Hub only; on a spoke the strategy is inert, as the whole index pass is.

Neither strategy changes what the retrieval tools see. A git-sourced repository is recalled, scanned, searched, and bundled exactly like a mounted one; only how its records get there differs.

Choosing a strategy

Pick by which of two properties matters more for that repository.

  • Mount the workspace when freshness is the point. A dev loop in which an agent must see the file you just saved - before it is committed, let alone pushed - only works on a mount. That is the common case for a single-node, local-first deployment, and it is why the mount is the default.
  • Source from git when a verifiable revision is the point. A shared or multi-replica host gains three things a mount cannot give it: every replica can name the commit it is serving, deletes are read from the commit rather than inferred from absence on disk, and the declared truth lives in the host's own configuration rather than in whoever mounted the volume.

The two cost profiles differ, but cost is rarely the deciding factor and should not be read as the headline. A git source does replace a per-reconcile directory walk with a fetch and a SHA comparison, so a repository that is idle most of the time settles into a cheaper steady state: an unchanged ref costs one shallow fetch and nothing else. It is not free, though - it needs reach to the remote on every refresh, and the staging work tree means the repository occupies disk twice. Treat the reduced walk as a secondary benefit of choosing a git source for the reasons above, never as a reason to give up a dev loop that has to see uncommitted work.

The choice is per repository, so nothing forces one strategy for the whole host: a host can mount the tree it is actively editing and source a stable dependency from its remote.

Configuring a git source

The feature is inert until LATTICE_REPOCONTEXT_GIT_REPOS names at least one repository. Listing a repository there is the whole opt-in: it registers the git strategy, refuses the mount path for that repository, and starts the refresh loop.

Variable Default Purpose
LATTICE_REPOCONTEXT_GIT_REPOS (unset) Semicolon- or comma-separated repository ids to source from git. Absent or blank leaves every repository on the mounted-workspace default and the whole subsystem inert.
LATTICE_REPOCONTEXT_GIT_STAGING_ROOT a lattice-repocontext-git directory under the system temp path The directory staging work trees are created under. Point it at a writable volume with room for a shallow checkout of every configured repository.

Every remaining setting is per repository. The repository id is folded to an upper-case identifier - non-alphanumeric characters become _ - so a repository named my-repo reads LATTICE_REPOCONTEXT_GIT_MY_REPO_URL:

Variable (suffix) Default Purpose
_URL (unset) The remote url to fetch from. A repository declared without one never indexes: it fails closed rather than falling back to a mount.
_REF refs/heads/main The ref to track. A bare main or v1.2.0 is qualified to a branch ref; pass refs/tags/v1.2.0 to track a tag.
_DEPTH 1 Shallow-fetch depth, clamped to 0-100000. 0 means a full-history fetch.
_REFRESH_SECONDS 300 How often the refresh loop re-fetches the ref, clamped to 30-86400. Each pass is spaced by this plus up to LATTICE_RECONCILE_JITTER_SECONDS of random jitter, and starts on the first self-index tick after that.
_FETCH_TIMEOUT_SECONDS 300 How long a single fetch may run before it is abandoned, clamped to 10-3600. The last-good index keeps serving across an abandoned fetch.
_AUTH token The credential mode: token (read a per-repository token) or anonymous (an explicit opt-in for a public or local remote). Anonymous is never a fallback.
_TOKEN (unset) The read-only token or password for token mode. Required in that mode; without it the repository does not index.
_USERNAME x-access-token The username paired with the token. The default suits a GitHub App installation token or a fine-grained PAT.
_INCLUDE (unset) Semicolon- or comma-separated include globs; when set, only matching files are indexed.
_EXCLUDE (unset) Semicolon- or comma-separated exclude globs; a match drops a file even when it also matched an include.
_EXCLUDE_BINARY true Whether files that look binary are dropped. Set false to ingest blobs too.

A minimal opt-in for a repository id of my-repo:

LATTICE_REPOCONTEXT_GIT_REPOS=my-repo
LATTICE_REPOCONTEXT_GIT_MY_REPO_URL=https://github.com/acme/my-repo.git
LATTICE_REPOCONTEXT_GIT_MY_REPO_REF=refs/heads/main
LATTICE_REPOCONTEXT_GIT_MY_REPO_TOKEN=<read-only token>

A git source does not require a hosted forge. Any url git can fetch from works, including a bare repository on a local volume, and anonymous is the explicit opt-in for a remote that needs no credential. That keeps the commit-anchored generation and the exact delete detection on a host with no outbound network at all:

LATTICE_REPOCONTEXT_GIT_REPOS=my-repo
LATTICE_REPOCONTEXT_GIT_MY_REPO_URL=/srv/git/my-repo.git
LATTICE_REPOCONTEXT_GIT_MY_REPO_REF=refs/heads/main
LATTICE_REPOCONTEXT_GIT_MY_REPO_AUTH=anonymous

The path is resolved inside the container, so mount the bare repository in as you would any other volume, and give the staging root somewhere writable to check out into. The trade is unchanged by the remote being local: the index still tracks a committed ref, so work that is uncommitted - or committed but not yet pushed to that remote - stays invisible until it lands there. A repository you are actively editing belongs on a mount.

What a refresh does

Shortly after startup the host arms every configured repository's self-index grain, retrying with backoff until the cluster is accepting calls, and the grain then drives the loop from its own tick, which its keep-alive reminder keeps running: a pass starts once _REFRESH_SECONDS, plus up to LATTICE_RECONCILE_JITTER_SECONDS of random jitter, has passed since the previous one. Each pass:

  1. Fetches the configured ref into the repository's staging work tree. The index is never read from a tree mid-fetch, and because the self-index grain is a singleton, a fetch already in flight is never stacked on top of.
  2. Resolves the ref to a commit. If it equals the SHA the last completed generation was stamped with, the pass is a no-op - no diff, no embedding, no write.
  3. Otherwise diffs the new commit against the stored per-file digests and applies exactly that add / modify / delete set. Deletes come from the commit, not from absence on disk.
  4. Stamps the repository record with the resolved commit SHA. repocontext_list_repos reports it as indexedCommit, and in a hub-and-spoke topology it replicates to spokes with the rest of the index, so every replica can state the revision it is serving.

A fetch that fails, times out, or authenticates badly leaves the previous generation in place and serving; nothing is pruned on the way in. The pass is safe to repeat, so a late or duplicated trigger costs at most one no-op fetch.

Security posture

The git source is the only part of the host that makes an outbound, credentialed call, so it is deliberately narrow:

  • Fail closed. A repository configured for token auth with no token resolves no credential and does not index. It never degrades to an anonymous fetch, and never falls back to a mounted walk. Anonymous access must be asked for by name.
  • Per-repository isolation. Credentials are resolved per repository id; there is deliberately no ambient, un-suffixed token variable that several repositories could share, so one repository's credential cannot fetch another's remote.
  • Never logged. Tokens are redacted from every log line and from every error message, including the userinfo component of a remote url, so a failed fetch cannot leak a secret into a diagnostic.
  • Read-only. The staging work tree is a fetch-and-checkout cache. Nothing is ever pushed, and the staging root is the only path the git source writes to; it does not confine the rest of the host, which still writes its data root and, when configured, the memory archive.
  • Hub only. On a spoke, the whole index pass is inert, so a spoke performs no fetch and needs no credential.

The credential lookup sits behind a small provider seam. The shipped provider reads the per-repository environment variables above; a host that would rather mint short-lived GitHub App installation tokens can replace it without touching the fetch, diff, or indexing paths.