Huginn Multi-Agent Design

2026-06-11 — synthesized from five reference implementations + the Claude Agent SDK patterns, sized for a self-hosted agent on a local LLM (the GPU server, 4×3090).

Why

Delegation is the bread and butter of working with agents: an orchestrator that can fan work out to specialized children keeps its own context lean (children's verbose tool churn never lands in the parent), parallelizes independent work, and routes cheap work to cheap models. First concrete consumer: the docs-MCP populate flow (resolve + ingest documentation without polluting the coding turn).

What the references taught us

Design (Phase 1 — this implementation)

The one new primitive: delegate

delegate({
  "agent": "explore" | "worker",
  "task":  "self-contained instructions…",   // single
  // or
  "tasks": ["…", "…", "…"]                    // parallel fan-out
})

• Not a Registry tool. It's intercepted in the agent loop (like run_command authorization) because it needs the Agent's client/MCP/policy. Its ToolDef is appended to the defs list only at depth 0.
• Returns one aggregated tool result: per-task status header + each child's final answer only (the SDK's summary-only rule). Children's intermediate tool output never enters the parent context.

Agent kinds (built-in, Phase 1)

Kind = tool subset + prompt + budget (opencode's agent-as-schema, minus the ruleset machinery we don't need yet). User-defined kinds in config are Phase 2.

Isolation & sharing (verified against the code)

Shared via Arc (all hot methods take &self): LlmClient, McpManager, Lsp (already Arc inside ToolContext). Cloned: ToolContext — except children get a fresh empty plan slot (the parent's PLAN.md is not theirs to advance) while keeping the shared spawned list (so the TUI can still reap servers children background). Fresh: messages (subagent system prompt + task), Registry (filtered per kind), fail-streak state. Children get no steering and self_review off (the parent is the verifier — oh-my-pi's "subagents don't verify" rule).

Guardrails

• Depth 1: children's tool defs exclude delegate (the oh-my-pi/opencode trick — absence beats a runtime error).
• Iteration cap 40 per child (config agents.child_max_iterations).
• Concurrency cap default 2 (config agents.max_concurrent) — the GPU server's llama-server is one box; a fan-out of 30 would just queue and starve the parent. Excess tasks queue FIFO (oh-my-opencode's per-provider semaphore, simplified).
• Command policy inherited; approval requests from children flow through the existing Approval event. The TUI queues approvals (previously a second concurrent request clobbered the first — fixed as part of this work).
• Failure isolation: a child error/cap-out becomes a [failed: …] entry in the aggregated result with whatever partial answer exists (oh-my-pi's salvage rule); the parent decides whether to respawn — it never crashes the turn.

Events / UX

Children emit through a relabeling forwarder onto the parent's event channel:

• ToolStart{name} → ToolStart{name: "explore#1·name"} — child activity shows in the transcript and the F2 activity panel with zero TUI schema changes.
• Child streaming (StreamStart/Delta/End) is suppressed — only tools + result.
• Child Done → Highlight("⤷ explore#1 done: <first line>").
• Spawn announced with Highlight("⤷ delegating to explore#1: <task…>").

Prompt guidance (parent)

A short "Delegation" section in the system prompt: delegate broad read-only exploration and independent parallelizable subtasks; write self-contained task descriptions (children share no context — spell out paths, APIs, acceptance criteria); verify children's work yourself; don't delegate trivial single-file edits.

Config

[agents]
enabled = true               # delegate tool offered at all
max_concurrent = 2           # parallel children (the GPU server slots are finite)
child_max_iterations = 40
# explore_model = { base_url = "http://a self-hosted host/v1", model = "...", max_tokens = 8192 }

Phase 2 — SHIPPED 2026-06-11 (headless autonomy platform)

Reframed around the real goal: Huginn is Muninn's headless coding hands, steered from Matrix with zero-to-minimal interactivity. The scarce resource isn't capability, it's trust — an autonomous agent that's wrong about "done" is worse than none. Built in four tracks (commit 1297d0c, Muninn side 121ffff):

A — Trust

• Structured reports (report.rs): every headless job (and every writing delegate child) ends by emitting a strict-JSON JobReport — outcome (done/partial/blocked/failed), summary, files_touched, verification, needs_from_user. Parse is fence/prose-tolerant (brace-scanner, string-aware); one retry; honest fallback that never claims verified. final_report().
• Independent reviewer gate (run_job → job_epilogue): when a job reports done with real file changes, a fresh-context Reviewer child (reads + runs the build/tests itself, cannot edit) verifies the whole-job diff. Reject → one repair round (same agent, full context) → re-review → still bad = downgrade to partial with the unresolved issues in needs_from_user. The quality bar when no human reviews. Toggle [agents] review_gate.
• Soft budget steering: converge warning injected at 70% of max_iterations; exhaustion calls forced_converge() (final no-tools summary) instead of erroring — a headless caller gets an honest partial, not a crash.

B — Durability

• Journal + resume (journal.rs): every message funnels through push_msg, appended JSONL. Worker jobs journal from message one (<id>.inflight.jsonl); a crashed worker's resume_inflight() scan resumes them on restart (acked-on- receipt jobs no longer die with the process). Torn trailing line dropped, not fatal. Finished → renamed .done/.failed as the audit record. TUI sessions auto-journal; huginn --resume restores the last conversation.
• Output spill (tools.rs::clip_or_spill): command output over 8 KB is written IN FULL to state/cmd-output/, represented inline as head+tail+path so the model greps the rest — no truncation loss, no context bloat in long autonomous runs.

C — Scale

• Background delegation: delegate(background=true) — children run detached (shared semaphore), each report injected via drain_bg() at a later parent step; the turn can't end while any are in flight.
• Worktree isolation: delegate(isolated=true) — each writing child gets a detached git worktree (git.rs::worktree_add), so parallel writers never collide; each report carries adopt/discard commands for its diff.
• Built-in reviewer kind + config-defined kinds ([agents.kinds.<name>]: prompt/tools/model/max_iterations — overrides built-ins, enables per-kind model tiering). Shared delegate_sem across all delegate paths.

D — Remote control

• Jobs accept events_subject (Huginn publishes started/highlight/notice/review/ done/result JSON) and steer_subject (token-gated NATS → the agent's existing steering channel). Worker job_events_channel() / spawn_steer_bridge().
• Muninn side: code_task relays progress into the room live (capped 25) and renders the final reply outcome-first from the report; new code_task_steer( job_id, message) redirects a running job from Matrix.

Verified end-to-end on the live worker (a self-hosted host Muninn ↔ NATS a self-hosted host ↔ the workstation worker): a job wrote+ran a file, streamed 4 progress events, returned a structured report, and the reviewer child independently re-ran verification before "done" stood.

Beyond Phase 2 — shipped since (self-improvement + hardening)

The headless platform (Phases 1–2) is the substrate; the work since has been making it trustworthy, learnable, and safe to run more than one instance of. Grouped by theme, each verified against the code.

The four-loop stack — L4 self-improvement

The agent core (L1), the verify loop (L2), and the event-driven worker (L3) all run every job. L4 is the deliberate, operator-driven arm that lets the harness rewrite its own method from its own scorecard. Built bottom-up:

• Run-trace ledger + analyzer (trace.rs): every finished job appends one compact RunRecord to a global runs.jsonl — outcome, stop-reason, iterations, tool-call / tool-failure counts, files touched, the repro/verified flags, and a snapshot of which quality levers were on (and the model id). analyze() reads many records and surfaces failure modes (budget exhaustion, stuck loops, tool-failure hotspots). Append-only and dependency-free like memory.rs. huginn --runs prints it. Config agents.run_trace (default on).
• Weakness mining (failure.rs): the failure-side mirror of the skill loop. After a run that did not finish cleanly, a cheap-model pass tags it with ONE reason from a small fixed vocabulary (forgot-to-validate, went-in-circles, bad-tool-args, wrong-approach, task-too-hard, env-not-persisted, harness-error, other) so the scorecard groups failures by real cause, not the surface stop-reason. The reason rides on the existing RunRecord (no new store). is_addressable() marks which reasons a harness change could actually fix. Best-effort, never changes a job's outcome. Config agents.failure_mining (default OFF until trusted; needs run_trace). From Self-Harness (arXiv 2606.09498).
• Payoff arm — propose / apply / decide (improve.rs): propose() turns the scorecard into ranked, typed config deltas — climbers (raise max_iterations on budget exhaustion, lower model.temperature on high tool-failure/stuck-loop, enable the review gate when unfinished runs burn more iterations than finished ones) and ablations (flip an on-by-default lever — review-gate / static-analysis / repro-check / skill-loop — off purely to MEASURE its contribution). An addressability filter suppresses climbers when mining attributes the failures to non-fixable causes. --improve-apply materialises the top proposal as a CANDIDATE config FILE (the operator's hand-edited config is never rewritten) and opens an experiment in experiments.jsonl. The exam scores baseline vs candidate; --improve-resolve decide()s — keep the candidate (move a one-line baseline pointer) only if it resolves strictly more instances, else revert. Inert until a real exam score is fed back — without that judge, L4 is a random walk.

Procedural memory — the skill loop

Factual memory (memory.rs) remembers WHAT is true about a repo; the skill loop (skill.rs, ported from Muninn / Hermes) remembers HOW a kind of task was done well. After a job finishes cleanly (done + repro + review passed), the run is distilled into a reusable Skill {name, when_to_use, steps, tools_used} and stored in a GLOBAL skills.jsonl; the next job surfaces the best lexically-matching skills as prompt guidance. JSON lines, keyword-overlap recall, append-only — same simplicity as memory.rs. Config agents.skill_loop (default on).

G4 static-analysis verify loop

lint.rs: after self-review, run the language's linter (clippy / ruff / eslint / go vet) over the touched files and fold its diagnostics into the SAME review::Findings the critic emits — so they flow through the existing blocking/auto-fix gate with no new machinery. Bug-class diagnostics map to P1 (block + auto-fix); style nits map to P2 (surfaced, non-blocking). Deterministic ground truth the LLM critic can miss; missing linters skip, never fatal; findings are filtered to the diff's files and capped (blocking first). Config agents.static_analysis (default on).

Reviewer-gate wall-clock budget

The independent reviewer gate now has a hard wall-clock cap across all rounds (agents.review_timeout_secs, default 600) plus a tighter reviewer iteration cap, so a slow reviewer model (e.g. a cloud DeepSeek) can't run the gate for hours re-fetching context. On timeout the gate stops and accepts the self-reviewed result with a note.

scaffold_project — bootstrap a checkpoint-built project

A native tool (tools.rs::ScaffoldProject) that writes a fresh project's README.md, AGENTS.md, design doc, and the first checkpoint spec into a target dir, with the per-stack GATE (build/test/lint, zero warnings, never push) and a runtime-shape smoke note baked in. It refuses to clobber an existing scaffold. The same Hermes process the /new-app Kai skill drives, available to Huginn directly: interview → one tool call.

Multi-endpoint model pool + fallback failover

config.rs: [[model.peers]] adds extra interchangeable hosts serving the same model, round-robined per request (ModelConfig::pool()); a dead one is skipped — for the same Qwen on the GPU server and .78. Separately, [model_fallback] is a different host the agent fails over to when the primary is unreachable (transport / 5xx / 429 / stall) — resilience, not routing. Both opt-in; unset = today's single-endpoint behavior. Named [models.<name>] tables back the TUI /model switch and /add-model wizard.

Per-instance journal + advisory lock (TUI context-bleed fix)

journal.rs: the in-flight session journal is now pid-namespaced (session.<pid>.inflight.jsonl) and each Journal takes an exclusive non-blocking OS flock held for its lifetime. A second TUI opened in another terminal no longer shares one global session.inflight.jsonl and mixes context; the lock is also how a resuming instance tells a live session from an abandoned one (and what gates cleanup of old journals). Convergent compaction also hardened: per-result clamping (clamp_tool_result) means no single message can exceed the window, so context-overflow compaction always terminates.

Hermes feed

hermes_feed.rs: the worker maps its internal AgentEvents to coabai-agent-protocol Envelopes and publishes them on a stable subject (hermes.huginn.events, HUGINN_HERMES_SUBJECT override) for the Hermes fleet console. Purely additive — the per-job thin-JSON feed Muninn parses is untouched, and publishing is best-effort (no subscriber → dropped).

Phase 3+ (deferred)

• Branch summarization for the parent's own long-context (pi's compaction) — Huginn now summarizes the oldest droppable block (not just drops it) on overflow, with per-result clamping guaranteeing convergence, plus the output-spill hygiene; full structured {Goal, Constraints, Progress, …} summaries later.
• Session adoption/revival of finished children (oh-my-pi) — resume a child with its context intact instead of respawning cold (the journal machinery now makes this cheap to add).
• Matrix-driven interactivity (Oz's "for later"): a job reporting blocked could ping for the needs_from_user answer and resume on reply — the events/steer subjects already carry everything required.

Reference repos (cloned, shallow)

~/code/agent-refs/{opencode, hermes-agent, pi, oh-my-pi, oh-my-opencode} — see the four study briefs in the 2026-06-11 session for file:line maps of each.