Backups

What to back up

The data directory (/var/lib/brain/data, the brain-data volume). It contains, per shard:

• arena.bin — the vector arena.
• wal/ — the write-ahead log segments.
• metadata.redb — memories, entities, statements, relations, predicates, idempotency, audit.
• Index state — the HNSW and tantivy artifacts (also reconstructable from the durable records above).

If you can restore this directory, you can restore Brain. The brain-models volume is only a model cache — it re-downloads on first run if lost, so it doesn't need backing up.

What NOT to back up

• The models volume — a cache; it repopulates.
• Container images — pull from the registry.
• Config / secrets — keep in your secrets manager (see Security).

Snapshots

Brain has a first-class snapshot mechanism — a point-in-time, self-consistent copy of a shard. It's driven through the admin surface:

• ADMIN_SNAPSHOT_CREATE — take a snapshot.
• ADMIN_SNAPSHOT_LIST — list snapshots and their metadata.
• ADMIN_SNAPSHOT_DELETE — remove one.

A shard can also snapshot itself automatically on a cadence — workers.snapshot_interval_sec (default 6 hours).

How a snapshot is taken

1. Drain writes briefly (a fraction of a second).
2. Force a checkpoint so the arena and metadata store are current with the WAL.
3. Reflink-copy arena.bin and metadata.redb, plus the WAL segments since the checkpoint. On a copy-on-write filesystem this is a FICLONE — near-instant and near-zero extra space until blocks diverge.
4. Resume writes — the live files continue; the snapshot is independent.
5. Verify + record — checksum the snapshot files and write a manifest with the snapshot's LSN.

The drain is dominated by the checkpoint; on NVMe that's typically tens of milliseconds.

Filesystem support

Reflink-based snapshots need a copy-on-write filesystem:

For fast, space-efficient snapshots put the data directory on btrfs or reflink-enabled xfs. On ext4 the snapshot still works, just as a full copy.

Restore

A snapshot directory is self-contained (arena.bin, metadata.redb, wal/, and a manifest). To restore:

1. Stop Brain (or at least the affected shard).
2. Move aside the existing arena.bin, metadata.redb, and wal/ in the data directory.
3. Copy the snapshot files into the data directory (reflink or full copy).
4. Restart Brain.
5. Brain runs recovery on restart, replaying the WAL records in the snapshot to bring state up to the snapshot's LSN.

Restoring the same shard onto another node works the same way — the manifest carries the shard UUID, which must match the destination shard.

Point-in-time recovery

A snapshot restores you to its LSN. To recover past the snapshot to a later moment, apply WAL records captured after it — recovery replays them on top of the restored state. This requires those later WAL segments to be available, either retained in the shard (bounded by shard.wal_retention_segments) or archived externally. If you need tight RPO, ship WAL segments off-box continuously and size retention so nothing you'd need is discarded before it's archived.

Volume-level backups

If you don't want to drive snapshots yourself, a filesystem or volume snapshot of the data directory works too — but take it against a quiesced shard (or a Brain snapshot) so the arena, WAL, and redb are mutually consistent. A raw copy of a live, mid-write data directory can capture a torn state; recovery relies on the WAL to reconcile, so include the entire wal/ directory in any copy. Prefer ADMIN_SNAPSHOT_CREATE, which guarantees the consistent point for you.

Restore drill

Practice restore before you need it.

1. Provision an empty data directory on a spare host or container.
2. Copy in the most recent snapshot directory.
3. Start brain-server pointed at it and watch recovery complete in the logs.
4. Verify liveness and that expected data is present:

curl http://localhost:9091/healthz

If recovery reports a CRC or checksum mismatch, the copy is incomplete — Brain fails stop rather than serving corrupt data, so a clean boot is your signal the backup is good.

Operational hygiene

• Push snapshots off-box to object storage; a snapshot on the same volume doesn't survive a volume loss.
• Encrypt backups at rest.
• Watch snapshot freshness — alert when the newest successful snapshot is older than your RPO.
• Keep the restore runbook off-Brain so it's reachable during an incident.