The Northern Ravens
← Back to Huginn

Huginn — Feature inventory & sources

This is the deliberately complete, slightly boring, very honest catalogue of what Huginn does and where each idea came from. It exists because Huginn was built on a simple rule — borrow the best ideas from the open-source coding-agent community, write the code yourself — and that debt deserves to be itemised in the open.

A few conventions used throughout:

  • Borrowed — the idea came from a specific project or article; the implementation is Huginn's own.
  • Convergent — built independently from first principles, and later found to match (and be validated by) outside work. Called out because it's honest, not because it's a brag.
  • Own — no external antecedent worth crediting; it fell out of building the thing.

The guiding thesis behind all of it — the model is a swappable battery; the harness is the durable layer you own — is itself the community's hard-won consensus, stated most directly in The Harness Problem and proven on Huginn's own SWE-bench numbers (a fixed quantized 27B model climbing 15 → 17 → 20 of 39 on harness changes alone).

A note on scope: every feature below is shipped and in the codebase unless its line says otherwise. A handful of self-improvement pieces are built but inert (they wait on an un-parked benchmark to be the judge) — those are marked explicitly rather than quietly listed as done.

1. The edit tool — how the agent changes code

Anchored ("hashline") edits. Every line the agent reads comes back tagged with a short content-hash; the agent edits by referencing that anchor instead of reproducing the surrounding code, and a hash mismatch (the file moved under it) rejects the edit before it can corrupt anything. This is the single highest-leverage reliability lever for weaker and local models, which are exactly the ones most likely to mangle code when forced to retype it. Borrowed — origin oh-my-pi (github.com/can1357/oh-my-pi), rationale in The Harness Problem (blog.can.ac/2026/02/12/the-harness-problem/); the same write-up reports a weak model going from 6.7% → 68.3% on the edit tool alone.

Syntax-guarded writes. Before an edit lands, a tree-sitter parse checks it didn't regress a file that previously parsed; only regressions are blocked, so partial-language and template files still work. Borrowed — plandex (github.com/plandex-ai/plandex) "validate the edit parses before accepting"; tree-sitter approach shared with aider.

Format-on-edit. If a formatter is on PATH (rustfmt, prettier, black, gofmt), edited files are formatted automatically so the model never spends tokens on whitespace. Convergent — standard practice; aider documents the same auto-lint/format step.

2. Reading the code — context strategy

Tree-sitter repo map. A structural map of the repository (files, definitions, call sites) the agent consults to orient itself instead of blindly grepping. Borrowed — aider's repo-map (aider.chat/docs/repomap.html) is the reference; the deterministic-structure / lazy-semantic split comes from Understand-Anything (github.com/Lum1104/Understand-Anything).

Summarize-don't-dump reads. Large files come back as structural summaries with elision rather than dumped whole, so a single read can't blow the context window. Borrowed — oh-my-pi's summarizing read.

Load-only-what's-relevant. The context is built around where the work is happening rather than front-loading the whole tree — the cross-cutting lesson every serious agent project converges on. Borrowed/Convergent — explicit in oh-my-pi, aider, plandex.

3. Code intelligence — LSP as tools

Language-server tools: lsp_diagnostics, lsp_hover, lsp_goto_definition, lsp_find_references, lsp_rename. The agent gets types, definitions, references and safe renames — information a text search cannot provide — for Rust (rust-analyzer), Python (pyright), TypeScript (typescript-language-server) and Go (gopls). Each degrades gracefully to "not installed." Borrowed — the LSP-as-tool-surface is taken straight from oh-my-openagent (github.com/code-yeongyu/oh-my-openagent).

4. The verify loop — checking its own work

run_command verify loop. The agent runs the build and the tests itself and reads the failures, instead of guessing whether a change worked. This — plus the edit tool — is the pillar every project agrees decides quality. Convergent/Borrowed — aider documents auto build/test/lint-then-fix; the loop framing comes from the loop-engineering reading (below).

Repro-test-first ("done" is a checked fact). For a bug fix, the harness re-runs the bug's own reported failing test before it will honour a "done"; a non-green result downgrades the outcome from done to partial. Saying "fixed" is a machine-verified claim, not the model's opinion. Convergent — built from first principles; later validated by Self-Harness (arXiv 2606.09498), whose self-improving agent independently re-derived "validate-before-conclude" as one of its top edits. Framed as MindStudio's "testable termination" and Lanham's "evaluation checkpoint."

Static-analysis loop. Linters (clippy, ruff, eslint, go vet) run on the touched files and their diagnostics are folded into the review gate as deterministic findings — bug-class issues become blocking and get auto-fixed, style nits are surfaced only. The model no longer has to remember to lint. Borrowed — LLMLOOP (arXiv 2603.23613), "specialized deterministic verify sub-loops"; its compile-fix / static-analysis / test-failure loops are the blueprint.

5. Planning discipline

Phased planning with per-phase commits. A large task is decomposed into phases; each completed phase updates a PLAN.md, is summarised, and is committed to git on its own — so progress is visible (/plan) and reversible. Borrowed/Convergent — plan-before-code is aider's architect/editor split and oh-my-openagent's "Prometheus planner"; per-phase commits are Huginn's own.

Re-anchor to the spec. The task/spec persists to disk and is re-read as a drift anchor across long runs. Borrowed — setkyar, "Building Autonomous AI Agent Loops."

6. Self-review — catching "it compiles but it's wrong"

Self-review critic. After a change passes the tests, a critic stage reviews it against quality rules and auto-fixes blocking findings over up to two rounds (P0–P3 severities; the static-analysis findings above flow through this same gate). Catches the class of bug that compiles and passes but is still wrong — an N+1 query, a missing SAFETY note, an unhandled edge case. Borrowed — code-review-skill (github.com/awesome-skills/code-review-skill) contributed both the self-review stage and its severity vocabulary; "a different agent verifies, so it isn't grading its own homework" is the verification-split principle from the loop-engineering reading.

Best-of-N candidate selection. For hard problems the agent can generate several candidate patches and select among them by test-clustering consensus, rather than polishing one trajectory — selection is its own high-leverage phase. Borrowed — R2E-Gym, which reports 34% → 51% pass@1 from selection alone; top SWE-bench agents do the same.

7. Memory & learned skills

Cross-session memory. Facts learned about a codebase persist across sessions (append-only JSONL, dependency-free to match the static binary). Convergent — every serious agent has this; oh-my-pi's "Hindsight" is the nearest named cousin. Active-memory (extract→transform→commit rather than naive append) is a known upgrade, noted from Loopcraft.

Procedural-skill loop. After a clean job, the run is distilled into a reusable Skill {name, when_to_use, steps, tools_used} via the cheap model and stored; the top keyword-matched skills are surfaced as a hint on the next job. Learning is best-effort and never changes a job's outcome. Borrowed — the Hermes reading / "Skills (reusable knowledge)" as named in Addy Osmani's Loop Engineering; same SKILL.md-style codified-conventions idea.

8. Self-improvement (the outer loops)

These implement the upper layers of LangChain's four-loop stack (L1 agent core → L2 verification → L3 events → L4 hill-climbing). Borrowed framing — LangChain, "The Art of Loop Engineering," and Latent.Space's "Loopcraft."

Run-trace ledger. Every finished job (including wedged ones) appends a compact record — task, outcome, why it stopped, iterations, tool calls and failures, which levers were set — to a global ledger; analyze() turns the ledger into a scorecard with plain-language observations. huginn --runs prints it. Own — built as the scorecard an automated improver needs.

Failure mining (planned, default-off). A cheap post-job pass that tags each failed run with why it really failed (from a short fixed vocabulary), not just the surface symptom — the failure-side mirror of the skill loop. Borrowed — Self-Harness "Weakness Mining." Status: planned; the cheap design (reason-tag + per-model stamp) is specified, not yet built.

L4 hill-climbing (built but inert). improve.rs reads the scorecard and proposes concrete, typed config deltas (raise the iteration budget when runs exhaust it; lower temperature on stuck-loops; enable the review gate when unfinished runs burn more iterations than finished ones), writes each as a separate candidate config, and promotes one only if it resolves strictly more benchmark instances than baseline. Deliberately inert until a real benchmark score is fed back — without that judge, L4 is a random walk, not hill-climbing. Borrowed — LangChain L4 + Loopcraft PostTrainBench; the promote-only-if-no- regression gate matches Self-Harness's validation criterion (built independently).

9. Safety & containment

Command-safety policy. Every run_command line is classified deterministically — each &&/||/|/; segment judged on its own, so cargo build && rm -rf / is caught — into Safe / Dangerous / Unknown, and a configured policy (ask / allowlist / model judge) decides whether it runs. Lets the agent work unattended without the risk of it deleting the repo. Convergent — built from first principles; Self-Harness independently re-derived a "tool-policy" edit, and plandex's controlled-exec / rollback sandbox is the nearest external cousin.

Open containment threads (not yet built): a blast-radius circuit breaker, human approval encoded as run-state (branch + merge request), and a prompt-injection posture that treats file/web/MCP content as data, never instructions. Surfaced by the loop-engineering reading (Lanham's L3 fixes); roadmapped, honestly not done.

10. Reliability & recovery

Session journal / resume. Every conversation message is appended to a JSONL journal as it happens, so a Huginn that crashes, is killed, or swaps its own binary mid-job can resume exactly where it stopped — and the journal doubles as the 3am audit trail. Own.

Context compaction (summarize-on-compact). When the window fills, the oldest block is summarised (via the cheap model) into a compacted note rather than dropped, so the repro and root-cause survive the context wall. Borrowed — Lanham's "silent truncation" failure mode (Oracle, "The Agent Loop Decoded"); the fix is Huginn's own.

Stuck-loop breaker & convergence nudges. A run that repeats identical failures is broken out of; an iteration budget plus a "you're 70% through, start converging" nudge keeps long runs from drifting forever. Convergent — Self-Harness independently re-derived a "loop-breaker"; MindStudio names "no infinite loops" as a core component.

Transient-retry, model peers & fallback. Interchangeable hosts serving the same model are round-robined and a dead one is skipped; a separate fallback host takes over on transport error / 5xx / 429 / stall. Own / Convergent — provider-agnostic routing is a Loopcraft "going-down loop."

11. Multi-agent

delegate tool. The agent can fan work out to subagents (explore / worker / reviewer roles) with depth-bound recursion (children can't themselves delegate) and a concurrency semaphore, governed by the policy judge and reviewer gate. Borrowed — the verification-split (a different agent verifies) is the recurring loop-engineering principle; adversarial/critic-panel shapes noted from oh-my-openagent's "team mode."

12. Extensibility & integration

MCP client. Huginn can use tools from external Model Context Protocol servers (filesystem, git, databases, browsers, your own services) over stdio or HTTP, alongside its native tools. Borrowed — MCP is the open standard; "plugins/connectors wire the loop to real systems" is a named loop-engineering component.

scaffold_project tool. Bootstraps a fresh project for checkpoint-by-checkpoint building against a gate — the shape that built the Workshop's A/B projects. Own — generalised from the in-house "Hermes" bootstrap process.

Four ways to run it: a full-screen TUI cockpit, one-shot CLI, interactive REPL, and a headless NATS worker (the delegation target the personal-agent side hands jobs to). Plus a best-effort event feed for a fleet console. Borrowed — the headless-core-many-frontends shape is oh-my-pi's (TUI / one-shot / RPC / editor); the event-stream decoupling is Huginn's own.

Benchmarks, honestly

  • SWE-bench Verified: 341 / 500 (68.2%) on a local quantized 27B model, scored with the official swebench harness, ~$0 per instance.
  • Controlled A/B vs the reference harness (mini-swe-agent, same model, same 500 instances, same grader): 341/500 vs 351/500 — a statistical tie (McNemar χ²=1.09, n.s.). The two are complementary: union 383/500 (76.6%).
  • Harness-only deltas on a frozen 39-instance subset, model fixed: 15 → 17 (self-review gate) → 20 (iteration budget) — 38.5% → 51.3%.

The honest reading: on a strong model the harness is not the differentiator — the model dominates and a sound harness is just enough to extract its full ability. The layer is real and measurable, but it is not magic.

Sources

Open-source projects (ideas borrowed, code written here):

  • oh-my-pi — github.com/can1357/oh-my-pi (Rust-core coding agent; the nearest blueprint — hashline edits, summarizing reads, role routing, frontends)
  • pi-mono — github.com/badlogic/pi-mono (the leaner upstream of oh-my-pi)
  • oh-my-openagent — github.com/code-yeongyu/oh-my-openagent (LSP-as-tools, model routing, adversarial team mode)
  • aider — github.com/Aider-AI/aider · aider.chat/docs (repo-map, edit-format leaderboard, auto lint/test, architect/editor split)
  • SWE-agent / mini-swe-agent — github.com/SWE-agent/SWE-agent · github.com/SWE-agent/mini-swe-agent (the A/B reference harness and benchmark)
  • plandex — github.com/plandex-ai/plandex (diff-review sandbox, syntax+logic validated edits)
  • Understand-Anything — github.com/Lum1104/Understand-Anything (deterministic + semantic repo-map split)
  • code-review-skill — github.com/awesome-skills/code-review-skill (self-review stage, severity vocabulary, progressive-disclosure-by-language)
  • R2E-Gym — best-of-N candidate selection (34% → 51% pass@1 via selection)

Articles & blog posts (the "loop engineering" reading):

  • The Harness Problem — blog.can.ac/2026/02/12/the-harness-problem/ (the hashline rationale and numbers)
  • What Is Loop Engineering? — MindStudio, mindstudio.ai/blog/what-is-loop-engineering-ai-coding-agents
  • The Agent Loop Decoded: Three Levels — Micheal Lanham / Oracle (three nested loops; silent-truncation and evaluation-checkpoint framing)
  • Building Autonomous AI Agent Loops — setkyar (re-anchor-to-spec)
  • The Art of Loop Engineering — LangChain, langchain.com/blog/the-art-of-loop-engineering (the four-loop stack, L1–L4)
  • Loopcraft: the Art of Stacking — Latent.Space (going-up/going-down loops, active memory, provider-agnostic routing)
  • Loop Engineering — Addy Osmani, addyosmani.com/blog/loop-engineering/ (the five components + memory: automations, worktrees, skills, connectors, verification-split, on-disk state)

Papers:

  • Agent-Computer Interfaces Enable Automated Software Engineering (SWE-agent) — arXiv 2405.15793 (formal evidence that interface/tool design drives performance)
  • LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops — arXiv 2603.23613 (specialized deterministic verify loops; mutation testing to grade test quality)
  • Self-Harness: Harnesses That Improve Themselves — arXiv 2606.09498 (weakness mining, addressability filter, promote-only-if-no-regression — much of which Huginn built independently)

Last reviewed: 2026-06-25. This page is maintained by hand; if a feature ships without landing here, that's a bug in the page, not the catalogue being complete.