Experiments: Measuring Outcomes, Not Guessing

One general primitive covers every A/B question

The Experiment primitive is deliberately domain-neutral: define a key and a list of variants, assign each subject to one sticky variant (a subject keeps the same arm for the life of the experiment), record its terminal outcome, and read back a per-variant success rate. That's the entire substrate — "A/B test the first versus second follow-up message" falls out of it with zero message-specific code, and so does "which model performs better" or "which price point converts," because none of that is baked into the primitive itself.

await agentis.experiment.define({ key: "followup_copy", variants: ["direct", "friendly"] });
await agentis.experiment.assign({ key: "followup_copy", subjectId });
await agentis.experiment.record({ key: "followup_copy", subjectId, outcome: "converted" });
await agentis.experiment.results({ key: "followup_copy" });

Success is a fixed vocabulary, on purpose

An outcome counts toward the success rate only if it's one of a fixed set — won, success, positive, converted — computed as successes divided by everything assigned to that arm. A narrow, fixed vocabulary keeps the aggregate comparable across every experiment in a workspace, rather than each experiment inventing its own ad-hoc notion of "it worked."

Continue