The Formation Pipeline: Extraction, Judge, PACER, Feynman Repair

The wall between 'an agent produced text' and 'the Brain commits a memory'

An earlier version of this pipeline promoted almost anything that looked like a sentence — the only bar was roughly eight tokens. That let digest rows, URLs, ranking keys, and an agent narrating its own process straight into the semantic graph. The current gate is deliberately stricter, in three stages of increasing cost.

Stage 1 — the deterministic gate

extractCandidateStatements is model-free: it flattens output, strips non-prose (code fences, JSON), splits into statements, and drops structural garbage outright — bare URLs, table rows, ranking keys, first-person process narration ("I then decided to…"), and boilerplate framing. Every survivor gets a deterministic 0–1 score, and only statements at or above 0.5 continue. This single stage removes the bulk of the pollution before any model is ever involved.

In parallel, classifyOutputShape asks what kind of thing the output even is:

Stage 2 — the Formation Judge

Deterministic survivors, plus the workspace's existing nearby memories, go to a two-phase LLM step modeled on the Mem0 pattern: extract the genuinely durable, reusable statements, type each one, and decide ADD, UPDATE, or NOOP against what's already known. This is why memory formation is a reconciliation, not a blind append — a restated fact updates the existing atom's confidence instead of creating a near-duplicate. The judge is model-agnostic (any structured completer); when no model is configured, candidates fall back to low-confidence episodic staging rather than being silently dropped.

PACER: what kind of knowledge is this?

Every formed memory gets classified by a separate, deterministic, model-free routing layer — PACER is not a new storage engine, it's a tag that decides how long a memory lives and how hard it resists decay:

PACER classifies before structure is stripped — code refs, paths, and identifiers are exactly where procedural and reference signals live, so the classifier reads the raw-ish candidate, not just the post-strip survivor. It also treats the memory's source surface as a first-class signal: the same sentence means something different coming from operator chat than from a tool dump or a document ingest — that provenance is threaded from the point of capture, never guessed after the fact.

The fallback: staged, decaying, and never silently lost

Without a Formation Judge model wired up, a survivor isn't discarded — it's staged as a low-confidence observation episode tagged unconsolidated, with a PACER-derived TTL: a procedural rule gets roughly 60 days to prove itself useful; bulk evidence decays in about 14. A staged trace that gets retrieved into a real dispatch context at least twice, or reinforced once, graduates into consolidated durable memory before it can expire — "lazy summarization": something earns permanence by proving useful, not merely by having been said once.

Operator-authored memory is treated differently even without a model: because the operator is the authoritative source, it's written as a consolidated, non-expiring atom directly — it never has to survive the staging/graduation gauntlet a machine-generated candidate does.

Continue