The harness is the product

I built a coding agent in a few days that ties the SWE-bench team's own reference harness — running a quantized 27B model on a home GPU, at roughly zero dollars per problem.

The short version

I built a self-hosted coding agent called Huginn. On SWE-bench Verified — the standard benchmark where an agent has to fix real GitHub issues in real repositories — it resolved 341 of 500 problems (68.2%).

Then I ran a controlled experiment against the SWE-bench team's own reference agent, on the same model and the same problems, graded by the same official scorer. The two landed in a statistical tie.

The catch: Huginn wasn't driving a frontier model behind an expensive API. It ran a quantized 27-billion-parameter model on a GPU box in my basement, at roughly zero dollars per problem.

This post is about why that result matters, and the one idea behind it: the model is a swappable battery. The harness is the product.

What I mean by "the harness"

When people talk about AI coding agents, they usually talk about the model — which LLM is under the hood. That's the battery. It's important, and it's also the part you don't own and can't improve. You wait for the next release.

The harness is everything around the model: the tool that lets it edit code without retyping it, the strategy for deciding what code to show it, the loop that runs the tests and feeds back the failures, the planner, the self-review step, the memory, the safety gate. That's the part you do own. Every improvement there is permanent, and it stacks on top of whatever model you plug in next.

My bet from day one was that the harness is where the durable value lives. So I built Huginn harness-first: point it at any OpenAI-compatible endpoint — a local model on my own hardware, or a frontier provider with an API key — and every model improvement becomes a free upgrade.

The result, plainly

Here's the experiment. I took the SWE-bench team's reference agent (mini-swe-agent) — a deliberately minimal harness they ship as a baseline — and ran it against Huginn under identical conditions: same quantized 27B model, same 500 Verified problems, same official grader.

A solo project, written in a few days, landing in a dead heat with the benchmark team's own reference harness — on their benchmark, with their grader — is the result I'm proudest of.

And there's a second, more interesting finding hiding in the numbers. The two harnesses don't solve the same problems. Huginn solved 32 that the reference missed; the reference solved 42 that Huginn missed. Put together, their union is 383 / 500 (76.6%). There's real signal in which problems each approach cracks — not just how many.

The honest part

Here's the thing most "look at my agent" posts won't tell you: on a strong model, the harness is not the differentiator. The model dominates. A good harness is just enough to extract the model's full ability — and a bad one holds it back — but you're not going to out-engineer a weak model into genius with a clever loop.

So where does the harness actually earn its keep? I measured that too. On a frozen 39-problem subset, holding the model fixed and changing only the harness:

• Baseline loop: 15 / 39
• Add a self-review step that catches "it compiles but it's wrong": 17 / 39
• Give it a larger budget to iterate before giving up: 20 / 39

That's 38.5% → 51.3% from harness changes alone, same model throughout. The layer is real. It moves the needle in a measurable way. It just isn't magic, and I'd rather tell you that than sell you a fairy tale.

What's actually in the harness

The pieces that did the work, briefly — each of these probably deserves its own post:

• Anchored edits. The agent edits code by pointing at an anchor, not by retyping the surrounding lines. This is the single biggest reliability win for smaller, local models — they're far more likely to mangle code when forced to reproduce it exactly.
• A verify loop. The agent runs the build and the tests itself and reads the failures, instead of guessing whether its change worked.
• A repo map and real code intelligence. A tree-sitter map for orientation, plus a language server for types, definitions, and references — so it edits with knowledge a text search can't give it.
• Phased planning. Big tasks get broken into phases, each committed to git separately, so progress is visible and reversible.
• A self-review critic. After a change passes the tests, a review step checks it against quality rules — now backed by real static analysis (clippy, ruff, eslint) — and fixes what it flags.
• Repro-test-first. For a bug fix, the agent re-runs the bug's own reported test before it's allowed to declare victory.
• Memory and learned skills. It remembers facts about a codebase across sessions and learns reusable approaches from jobs that went cleanly.
• A safety gate. Every shell command is classified before it runs, and a policy decides whether to allow it, ask, or refuse — so it can run unattended without me worrying it'll delete the repo.

Standing on shoulders

I didn't invent this in a vacuum, and I won't pretend I did. I borrowed ideas (not code) from a handful of open projects and a stack of writing on agent loops — among them aider, the SWE-agent work that put the thesis on a formal footing, and several others I kept a running inventory of as I built. The anchored-edit idea in particular came straight out of that reading, and it was the highest-leverage thing I added.

The point of writing this down is partly to pay that forward.

Why this matters

Two reasons, depending on who you are.

If you care about cost and control: you don't need a frontier API budget to get useful coding-agent work done. A good harness on a model you host yourself gets you into the same neighborhood, and you own the whole thing.

If you care about building agents: the harness is the part worth your time. It's durable, it's learnable, it compounds, and — as the experiment above shows — a sound one is reachable in days, not months. The model will keep getting better on its own. The harness only gets better if someone builds it.

That someone can be you.

Huginn is one of two ravens I'm building. Its brother, Muninn, is the memory to Huginn's thought. More on both in the docs.