The Brain · 03.12
The Benchmark Scorecard
A LoCoMo/BEAM-style eval harness runs as a CI gate on every change to the Brain — single-hop, multi-hop, temporal, knowledge-update, unanswerable, and mid-task recovery, measured, not assumed.
Measured, not assumed
A memory system's claims are cheap until someone runs a benchmark against the real retrieval + cited-ask stack and publishes what came back. Agentis ships a small LoCoMo/BEAM-style eval harness (eval/brain/brainEvalHarness.ts) that does exactly that, and it isn't a one-off report — it's wired into CI as tests/brain/brainEval.test.ts, so a change that quietly regresses recall or lets an unanswerable question get a confident-sounding answer fails the build, not a dashboard nobody checks.
The suite covers six categories: single-hop (one fact, one question), multi-hop (the answer requires connecting two stored facts), temporal ("what's true now" when a fact was updated), knowledge-update (a later statement supersedes an earlier one), unanswerable (nothing in memory should ground an answer — the correct behavior is abstention, not a guess), and recovery (a newer category: does a deliberately-mismatched upfront query correctly abstain, and does the agent's own agentis.brain.search tool then recover the answer with a reformulated query — proving mid-task re-retrieval actually works, not just that it exists).
The floor the CI gate enforces
These are the minimums a pull request cannot drop below — not a one-time measurement that ages the moment it's published, but numbers re-checked on every change:
| Metric | Enforced floor | Why this one is non-negotiable |
|---|---|---|
| Abstention rate (unanswerable cases) | 100% | A confident answer with no grounding is worse than an honest "I don't know" — this is the one metric with zero tolerance. |
| Faithfulness (of answered cases) | ≥ 80% | An answer the Brain DOES commit to must actually be grounded in what it retrieved. |
| Overall accuracy (all categories) | ≥ 70% | The broad recall baseline across single-hop, multi-hop, temporal, and knowledge-update together. |
| Single-hop accuracy | ≥ 90% | The easiest real category — if this regresses, something fundamental broke. |
| Recovery rate (mid-task re-retrieval) | 100% on cases where the upfront pass correctly abstained | Proves an agent can recover from a first search that missed, the way a human re-phrases a query — verified green on the run that added this category. |
These are floors, not the ceiling — the harness scorecard from the last local run typically clears them with room to spare. Regenerate the exact current numbers yourself: pnpm vitest tests/brain/brainEval.test.ts from apps/api.
Continue
The scoring mechanics behind recall: freshness decay, trust weighting, reranking, and MMR diversification, explained with real inputs.
Not a chatbot, not a script, not a SaaS app with an AI feature bolted on. An Agentic App is durable software built from four real pillars — data, interface, orchestration, and its own Brain.