Write pipeline | Brain Docs

encode is Brain's one write verb. It does not run an N-stage LLM pipeline before returning — it acks as soon as the write is durable, then derives everything else in the background. For the request/response shapes see API: memories; for the SDK method see SDK: encode.

Two phases split by the WAL-fsync ack barrier. The client waits only for validate → embed → reserve → persist; edges, the typed graph, the text index, and HyPE run after the response returns.

The synchronous fast path

Four phases run before the response is sent. The handler blocks on them; nothing downstream can make the client wait.

Phase	What it does
`validate`	Input validation + router policy (kind, salience); idempotency peek by `RequestId`.
`embed`	BGE-small → the 384-dim vector — the memory's semantic key.
`reserve`	Allocates the arena slot and mints the version-stamped `MemoryId`.
`persist`	Writes the vector to the arena, fsyncs the WAL (the ack barrier), commits the redb metadata row, and inserts the memory's HNSW point.

The WAL fsync in persist is the acknowledgement barrier: encode never returns success until that record is durable (core invariant — WAL before ack). The response carries the MemoryId, was_deduplicated, the server-stamped salience, auto_edges_added, persisted_at, and the embedding model fingerprint. Typical latency is p50 ~5–10 ms, p99 ~25 ms — dominated by the embedder, dropping to ~2–3 ms on an embedding-cache hit.

No LLM on the ack path

There is no extraction, no dedupe LLM judge, and no reference-resolution call between the request and the ack. The only model on the fast path is the local BGE-small embedder. Everything that needs the LLM runs after the response returns.

Content deduplication

Text encode deduplicates by content as an always-on DB policy, not a client flag. Before reserving a slot, Brain consults a per-(shard, agent, context) fingerprint index keyed on BLAKE3(text). On a hit it returns the existing MemoryId — no new slot, no WAL record, no HNSW node — and sets was_deduplicated = true. Re-saving byte-identical text under the same identity and context yields one memory, never a pile of identical rows.

This is distinct from RequestId idempotency, which dedupes exact replays by request identity. Fingerprint dedup dedupes by content identity. Genuine near-duplicates (paraphrases, the same fact re-stated) are not collapsed here — they're reconciled later by the consolidation worker. Only the direct-vector op keeps an explicit opt-in deduplicate flag, because a caller supplying raw vectors owns that decision.

The asynchronous derivation

After the ack, per-shard workers derive everything else. Each stage is observable live over SUBSCRIBE, keyed by the write's LSN. Three of them are first-class stages that emit completion events:

Stage	What it does	Output
`auto_edge`	HNSW k-NN of the new vector → `SimilarTo` edges.	edges
`temporal_edge`	Session adjacency → `FollowedBy` edges.	edges
`extractor`	The three-tier pipeline (pattern → classifier → LLM): entities, statements, relations → the typed graph, plus statement-text indexing.	the knowledge graph

Two more derivations run asynchronously but aren't first-class stages: the memory_text tantivy upsert (so the lexical retriever can find the memory), and write-time HyPE — hypothetical-question generation whose embeddings widen the recall surface. HyPE and the LLM extractor tier are why a valid LLM provider key is a hard boot requirement.

Observing a write

Because derivation is asynchronous, Brain gives you three ways to watch it — the write itself is unchanged whether or not anyone is looking.

wait: "ack" (the default) — return the instant the WAL is durable. The response carries the LSN and the set of pending stages; the graph fills in behind it.
wait: "derived" — block until the async stages settle (bounded by the shard's drain window), then return a full EncodeTrace: the per-stage timeline and the concrete artifact each stage produced (the embedding vector, the stored record, analyzed text-index terms, HyPE questions, and the entities / statements / relations graph). A stalled worker can't hang the call — stragglers are recorded as Timeout and still land in MEMORY_INSPECT.
SUBSCRIBE — stream StageCompleted events for the LSN as they happen, without blocking the encode.

The same per-stage artifact bundle is persisted per memory in redb, so any memory — not just a fresh write — can be inspected later with MEMORY_INSPECT. Sync fields (vector, record, keyword terms) are present the instant the write acks; the graph and HyPE fields fill in as the workers settle.

wait for writes, trace for reads

A write's completion timing and its observability payload are one decision — the only reason to wait for derivation is to observe it — so writes carry a single wait enum. A read is fully synchronous with nothing to wait for, so recall / plan / reason carry a pure trace: bool toggle instead. No op carries both.

Durability and crash safety

If Brain crashes between the WAL fsync and the network ack, the memory is already durable — on recovery the WAL is replayed and the memory becomes fully visible. The client may see a network error; retrying with the same RequestId returns the cached response. The caller never sees a half-written memory: encode is a single atomic commit, and partial state is never visible to other clients.

Latency budget

Rough p50 numbers, single-shard:

Phase	p50
validate	< 1 ms
embed	2–8 ms (BGE-small, local; cache hit ~0)
reserve	< 1 ms
persist (incl. WAL fsync)	2–15 ms
synchronous total (the ack)	~5–25 ms
async derivation (post-ack)	100 ms – seconds (LLM-bound), off the response path

Batched encodes push throughput to ~10K/sec/shard; per-encode latency rises slightly due to batching delay.

The ack path is bounded and LLM-free by design. If you need the typed graph before you proceed, use wait: "derived" — but understand you are then paying for the LLM round trip on that call. For steady-state throughput, default to wait: "ack" and consume the graph on the next read.