Scaling

The scaling model

Brain is thread-per-core. The process splits into shards; one shard pins one core and runs its own executor (Glommio, io_uring). Each shard owns its own arena, WAL, redb metadata, HNSW indexes, and tantivy indexes. There is no shared lock across shards, and no external store that all shards contend on.

The consequence: adding capacity means adding shards (vertical, more cores on one box) or adding machines (horizontal, more boxes). There is no database tier to size separately, no connection pool to tune, no shared_buffers to grow.

flowchart TD
  A[Throughput growing] --> B{Cores free on the box?}
  B -->|Yes| C[Raise shard_count]
  B -->|No| D{Working set fits in RAM?}
  D -->|Yes| E[Add a machine, distribute shards]
  D -->|No| F[Add RAM or a bigger box: HNSW + models are RAM-resident]

Vertical: more shards per box

Grow [storage] shard_count (or BRAIN__STORAGE__SHARD_COUNT) up to the core count. Each shard is an independent write path — more shards means more concurrent writes and reads, because the single-writer-per-shard discipline no longer serializes across the whole process.

Guidelines:

• One shard pins one core. Don't set shard_count above the available cores; you'll just oversubscribe.
• Each shard has its own arena. Per-shard arena capacity (shard.arena_capacity_bytes, default 1 GiB ≈ 660K vectors) multiplies by shard count — budget RAM and disk accordingly.
• HNSW is RAM-resident. All three indexes per shard live in memory. The working set is (roughly) vectors + graph + models. Size RAM so the whole set stays resident; unlike a paged database there's no "spill to disk and get slower" mode you want to rely on.

Horizontal: more machines

Run more brain-server processes on more machines and place shards across them. Because a shard is self-contained (its data directory holds everything), a shard can be moved or provisioned onto another node from a snapshot. Route clients to the machine holding the relevant shard at your ingress layer.

There is no cross-machine database to keep consistent — each shard is its own durable unit. Coordination is about placement and routing, not about replicating a shared store.

What to scale first

1. Raise shard_count until you've used the box's cores. This is the cheapest lever.
2. Add RAM if the working set (vectors + graph + bundled models) is approaching physical memory. Models alone are several GiB; account for them before data.
3. Add machines once a single box is saturated on cores or RAM.

The single most common mistake carried over from database-backed systems is looking for a "DB tuning" step. There isn't one — the storage engine is the process.

The LLM provider is the shared dependency

The one thing every shard reaches out to is the LLM provider API (write-time HyPE + extraction). That's an external service call, and its throughput and rate limits are the shared ceiling that sharding does not relieve. If write throughput plateaus while cores sit idle, the LLM provider — not Brain — is likely the limit. Options:

• Tune the extractor worker's LLM budget per cycle (workers.extractor.llm_budget_per_cycle_micro_usd) so cost/throughput matches your provider tier.
• Raise your provider's rate limits, or use a higher tier / larger quota.

Rerank cost

The read-path reranker ([rerank] enabled) loads a cross-encoder into RAM per shard and adds compute per read. If enabled, factor its memory into per-shard RAM. Disabling it returns RRF-only ordering and costs nothing — it's the one capability safe to turn off (see Configuration).