Five Qwen Agents Reproduce a Paper’s Coding Result for $36

Memdoor published a team-solve workflow on October 31, 2026 that rebuilds a published multi-agent coding harness with the original authors’ prompts, and reports that five Qwen3.8-27B instances matched it for $35.79 in total. Across 50 runs on ten LiveCodeBench Hard problems the team averaged 9.8 out of 10, against 9.6 for the authors’ own harness and 8.0 for Claude Fable 5 answering each problem in a single call.

What the workflow actually is

It is eight YAML files and no code. A planner reads the problem and writes a plan; two ideators work in parallel, one proposing the most direct provable algorithm and the other a method from a different family; a solver picks between them; a test step runs the samples; a review step writes repair instructions; a repair worker fixes the program; and a verify step grades it. Every agent starts from a fresh context and is told to read the files rather than rely on chat history.

The important design choice is who decides success. The test and verify steps are shell commands, not models. A failing verify exits nonzero and fails the run, so a model’s opinion about its own work cannot pass the workflow. The steps never name a model; a pin in the CLI decides which one answers.

Pass	This workflow	Authors’ harness	Fable 5, one call
1	10 / 10	9	7
2	10 / 10	10	8
3	10 / 10	10	10
4	10 / 10	9	8
5	9 / 10	10	7
Mean	9.8	9.6	8.0

What it does not prove

The repository is unusually direct about this, and the caveats matter more than the headline. Ten problems and five passes is a small sample: the README says 9.8 and 9.6 “are within noise” and describes the claim as parity with the authors’ harness, not an improvement on it.

Contamination is not excluded. The LiveCodeBench problems date from spring 2025, so the model may have seen them, and that applies to every column. The Fable 5 figures come from the authors’ result files with one call per problem — the model was never run inside this harness, so the comparison is between a multi-agent scaffold and a single call, not between two scaffolds.

The full-picture numbers are also less dramatic than the slice. On the authors’ complete 100-problem set the two reported figures are 92.4 and 90.4 — about a point apart, where the ten-problem subset shows a sixteen-point gap. Whatever advantage the harness has, it is concentrated in these ten problems.

Where it lost, and why that is informative

One problem failed. On pass 5, problem abc399_e, the program written in round one was correct. The manager then invented a counterexample with a wrong expected answer, and the team rewrote the program around it. The replacement passed the visible samples — which do not cover that case — and scored 32 of 44 hidden tests.

Two more nearly failed the same way: correct programs were overwritten because the manager never declared the work done, and both were recovered later in the loop, at $3.44 and $4.24 the two most expensive runs. Memdoor identifies both as gaps in the original loop and kept them rather than fixing them, so the published numbers reflect the authors’ method. It also notes both are fixable without changing the method — check a manager’s counterexample against a brute force on small inputs, and keep the last program that passed the samples.

Getting parity took real work

Memdoor ran a byte-for-byte check: it drove the authors’ multiagent_solve and its own workflow against a scripted model and compared every request, across four cases. To make them match it had to add four features to its own engine — answering with the model’s reasoning when it wrote no text, per-model top_p and top_k, marking a reply cut off at the 128K cap, and giving a call its own four-hour clock.

The host differed too. The authors ran their own vLLM in FP8; Memdoor ran on OpenRouter, where one provider slowed to 14 tokens per second and three runs had to switch hosts mid-run with finished steps preserved. That is a realistic picture of reproducing a paper on rented capacity, and it is the kind of detail that usually goes unmentioned.

What This Means

The headline — five small models beating a frontier model — is doing more work than the evidence supports, and Memdoor says so itself. What the release does establish is more useful: a published multi-agent coding harness can be reproduced by an independent team to within a single problem, at $0.39 and 40 minutes a run, with all 50 traces published.

That is a meaningful shift in what a benchmark result costs to check. When a scaffold can be rebuilt from YAML and rerun for tens of dollars, the interesting question stops being which model wins and becomes which harness structure earns its tokens. The finding that a manager can overwrite a correct program twice in fifty runs is worth more to anyone building agents than the 9.8.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

• GitHub — Memdoor team-solve workflow (README, results table, caveats)
• GitHub — team-solve working notes (parity check, host switching, failure analysis, cost)
• GitHub — GVS5H, the harness being reproduced
• GitHub — Memdoor (Apache 2.0, single Go binary)