Observability

Observability in Brain is built from the crates it already runs on — tracing + tracing-subscriber for structured logs, and opentelemetry for distributed spans. There is no separate metrics database and no Postgres observability schema; each shard emits its own signals and the audit of record is the WAL.

Four surfaces

Tracing and OpenTelemetry

Every operation is instrumented with tracing spans and exported through OpenTelemetry. The span tree is server-rooted: a request enters at the connection layer (Tokio) and its parent span is carried across the boundary into the shard's Glommio executor when the op is dispatched, so a single trace spans both runtimes rather than breaking at the hand-off. Public operations, error paths, and background workers all open spans at the right layer, so a trace shows the operation, its phases, and their timings end to end.

Logging is a single knob — one log-level setting in TOML, no scattered per-module switches (part of the config consolidation: all tuning in structured TOML, one env override form).

Watching a write derive — SUBSCRIBE

Because the typed graph is derived after the write acks (Write pipeline), Brain lets you watch it happen. SUBSCRIBE streams StageCompleted events for a write's LSN as the async stages settle — auto_edge, temporal_edge, and extractor each announce completion. This is the live view: no polling, no blocking the encode. It's how a client learns "the graph for the memory I just wrote is now ready" without holding the write open.

Inspecting a write — EncodeTrace and MEMORY_INSPECT

Two views expose the concrete artifacts a write produced, not just its timing:

• EncodeTrace — returned when you call encode with wait: "derived". It's a one-shot view of a fresh write: the per-stage timeline plus each stage's output (the embedding vector, the stored record, the analyzed text-index terms, the HyPE questions, and the entities / statements / relations graph). A stage that doesn't finish inside the shard's drain window is recorded as Timeout — it still lands durably.
• MEMORY_INSPECT — the same artifact bundle, persisted per memory in redb and readable for any memory at any later time. Sync fields (vector, record, keyword terms) are present the instant the write acks; the graph and HyPE fields fill in as the workers settle, so inspecting a just-written memory shows exactly how far its derivation has progressed. Scope is enforced by the memory's owner: a memory_id owned by another tenant reads as "not found," indistinguishable from a missing one.

Together they answer "how was this one memory built" — the observability analogue of provenance (see Supersession and provenance).

The WAL as audit trail

Every write — encode, forget, link, schema changes — is recorded in the WAL before it's acknowledged (the core WAL-before-ack invariant). The WAL is therefore a complete, ordered, replayable audit trail: on recovery it reconstructs shard state, and for compliance it answers "show me every forget in the last 30 days." A hard forget zeroes the memory's bytes but the forget event itself remains in the log, so you can prove a deletion happened even though its content is gone.

Introspecting a shard — GET_CAPABILITIES

After the WELCOME handshake, a client calls GET_CAPABILITIES to learn which capabilities are live on the connected shard — most usefully whether the cross-encoder reranker is loaded (the one deploy-time C1 toggle). This lets a client know before it reads whether it's getting reranked ordering or RRF-only ordering, without guessing from the results.