Idempotency | Brain Docs

Prerequisites

A Brain client.
Familiarity with Adding memories.
An at-least-once source (Kafka, SQS, webhook delivery) that may redeliver.

What you'll build

A queue worker that ingests messages exactly-once into Brain by reusing one request id per logical message across retries, and understands when a repeat produces was_deduplicated: true versus a cached response.

Two mechanisms, two jobs

They are easy to confuse but do different things:

Request-id idempotency protects against retries (the same call sent twice). Content dedup protects against re-ingestion (the same text arriving through different calls). A retry may hit either: the cached response if the id repeats, or was_deduplicated if the id is fresh but the text already exists.

Step-by-step

The wire builders mint a fresh requestId on build(). To make a retry idempotent, build the request once and resend the same object — the id is stable, so the server dedupes the retry to one logical write.

import { EncodeBuilder } from "@brain-db/sdk";

// Build once — the requestId is minted here. Resend the SAME object on a
// retry and the server dedupes it to one logical write.
const req = new EncodeBuilder(msg.text).context(msg.topicId).build();

let res;
for (let attempt = 0; attempt < 5; attempt++) {
  try { res = await client.encode(req); break; }
  catch (e) { if (attempt === 4) throw e; }
}

from brain_db_sdk import EncodeBuilder, with_retry

# Build once — the request_id is minted here and reused on every attempt.
req = EncodeBuilder(msg.text).context(msg.topic_id).build()
res = with_retry(lambda: client.encode(req))

use brain_db_sdk::EncodeBuilder;

// Build once — the request_id is minted here. Resend the SAME object on a
// retry and the server dedupes it to one logical write.
let req = EncodeBuilder::new(&msg.text).context(msg.topic_id).build();

let mut res = None;
for attempt in 0..5 {
    match client.encode(&req).await {
        Ok(r) => { res = Some(r); break; }
        Err(e) => if attempt == 4 { return Err(e.into()); },
    }
}

Do not rebuild the request inside the retry loop — a new build() mints a new id and defeats the mechanism. Same id + same params → the cached response; same id + different params → a Conflict.

The HTTP encode uses a stable server-side request id and is retried automatically on 503 and transport failures per the client's retry policy — so a lost response on the wire is safe. Other HTTP verbs run exactly once. See Retries and cancellation.

const brain = new BrainHttpClient({
  apiKey: process.env.BRAIN_API_KEY!,
  retry: { maxAttempts: 5 },
});
const res = await brain.encode({ text: msg.text, context: msg.topicId });

from brain_db_sdk.http.retry import HttpRetryPolicy

brain = BrainHttpClient(
    os.environ["BRAIN_API_KEY"], retry=HttpRetryPolicy(max_attempts=5),
)
res = brain.encode(text=msg.text, context=msg.topic_id)

use std::time::Duration;
use brain_db_sdk::{BrainHttpClient, HttpRetryPolicy};
use brain_db_sdk::http::EncodeInput;

let brain = BrainHttpClient::new("https://api.arc-labs.ai", &std::env::var("BRAIN_API_KEY")?)
    .with_retry_policy(HttpRetryPolicy::new(5, Duration::from_millis(100), Duration::from_secs(2)));

let res = brain
    .encode(&EncodeInput {
        text: msg.text.clone(),
        context: Some(msg.topic_id),
        ..Default::default()
    })
    .await?;

When the same text is submitted again (a replayed message, a re-summarized document), content dedup returns the existing memory instead of creating a duplicate. The fingerprint is BLAKE3(text) scoped to (agent, context).

const a = await brain.encode({ text: "User upgraded to Pro." });
// … message redelivered …
const b = await brain.encode({ text: "User upgraded to Pro." });

console.log(a.memory_id === b.memory_id); // true
console.log(b.was_deduplicated);          // true — no new row was created

Sometimes identical text is a genuinely distinct observation (the same status re-stated at a different time). Opt out of content dedup with allowDuplicates on the wire client, ideally paired with a distinct occurredAt.

await client.encode(
  new EncodeBuilder("Deploy succeeded.")
    .occurredAt(BigInt(Date.now()) * 1_000_000n)
    .allowDuplicates()
    .build(),
);

End-to-end: at-least-once into exactly-once

Derive a stable context (or request id) from the upstream identifier so retries collapse to one write, and let content dedup catch any that slip through with a fresh id.

async function ingest(msg: KafkaMessage) {
  // One requestId per message; a retry resends this same object and dedupes
  // server-side. Any that slip through with a fresh id hit content dedup.
  const req = new EncodeBuilder(msg.value.toString())
    .context(BigInt(msg.partition))
    .build();
  await client.encode(req);
}

def ingest(msg):
    req = EncodeBuilder(msg.value().decode()).context(msg.partition()).build()
    # One request_id per message; retries reuse it and dedupe server-side.
    with_retry(lambda: client.encode(req))

use brain_db_sdk::{with_retry, BrainClient, EncodeBuilder, RetryPolicy};

async fn ingest(client: &BrainClient, msg: &KafkaMessage) -> brain_db_sdk::Result<()> {
    // One request_id per message; a retry resends this same object and dedupes
    // server-side. Any that slip through with a fresh id hit content dedup.
    let req = EncodeBuilder::new(&msg.value).context(msg.partition).build();
    with_retry(&RetryPolicy::default(), || client.encode(&req)).await?;
    Ok(())
}

Don't rebuild the request between attempts, and don't mutate the text to "force" a write past dedup — reach for allowDuplicates (with a distinct occurredAt) when repeats are legitimate. Mutating text to dodge the fingerprint corrupts your own data.

Two mechanisms, two jobs

Step-by-step

End-to-end: at-least-once into exactly-once

On this page