The Workflow Engine · 05.08
The Recovery Ladder: Five Rungs From Reflex to Escalation
How a failed node recovers: deterministic rebind, capability-aware remedy, config-aware surfacing, certified structural repair, then honest escalation.
Cheapest and least invasive first
| Rung | Action |
|---|---|
| 1 · Deterministic | Runtime rebind / reroute to the healer — zero LLM tokens spent. |
| 2 · Capability-aware | A missing binary, provider, or tool is escalated with the exact remedy — never silently re-planned around. |
| 3 · Config-aware | A missing env var or working directory is surfaced as configuration to fix, not treated as a graph fault. |
| 4 · Structural | A certified graph patch, gated by recovery mode and — depending on impact — human approval. |
| 5 · Escalate | An honest stop with a concrete "send to the Agentis team" path; circuit breakers stop the ladder from cycling forever. |
Readiness is checked before work begins
Before a production run enters execution, every explicitly assigned agent is checked for existence, pause state, a healthy runtime, and the capabilities required by its node. A missing or unusable runtime blocks the run before paid work begins, or is resolved through a compatible runtime binding when one is available. The operator sees the actionable reason instead of a run that starts and later fails silently.
Approval is decided by impact, not by mood
Whether a structural repair needs a human's OK isn't a vibe — it's computed. In guarded recovery mode, a proposed repair requires approval unless its assessed impact is internal; anything touching more than the graph's own internals stops for a human. The decision and the impact assessment are kept in one small pure function specifically so the "should a human see this" logic can never quietly drift out of sync between an approval-gated path and an autonomous one.
Repeated failures don't get repeated repair attempts
A stable failure fingerprint combines the failure class, base revision, node and contract location while normalizing volatile ids, timestamps, and raw numbers. Agentis permits only one structural repair proposal for the same fingerprint against the same base revision. Later runs reuse the recorded outcome instead of repeatedly calling self-heal. A new attempt becomes eligible only when the base revision or the materially relevant failure changes.
Repair has a small control surface and a hard budget
The self-heal model can inspect the workflow, environment, runtimes, extensions, and code, but it cannot launch competing runs, replay branches, deliver, patch the stored graph through a second route, or repeatedly invoke the workflow test loop while the engine already owns recovery. Its tool-call budget defaults to 48 through AGENTIS_SELF_HEAL_MAX_TOOL_CALLS, not thousands. Independent build inventory reads run concurrently, and deterministic delivery preflight is reused while the graph hash is unchanged, so recovery spends time on new evidence rather than re-proving identical state.
Continue
Cross-workflow activations are persisted before execution, then claimed, retried, and recovered with idempotent delivery identities.
A proven, blessed workflow graph is never autonomously restructured — repair requires certification and a rollback path.