Weight-Space Programming Works Beyond GPT-6 Astra, Replications Show

Researchers at MIT CSAIL report that a frontier language model can read the raw numerical weights of another neural network, say what it was built to do, and in some cases rewrite those weights to change its behaviour. The report, published November 2, 2026 by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, calls the approach Weight-Space Programming. Its headline result lifted a small open-weight model’s MATH-500 score from 7.9% to 48.1% by editing 0.016% of its weights, with no gradients and no training loop — and within a day, independent attempts had reproduced parts of it on hardware as ordinary as a laptop.

Reading weights without running them

Weights are written by an optimizer, and no human reads them. The report’s first experiments ask whether a model can inspect them instead, without executing the network. Four policies were trained with PPO on four games, all on the same 10×10 grid with identical inputs, actions and dimensions — 17,253 weights each — so only the values distinguish them. Asked which game each was trained for, and not told which games were in play, GPT-6 Astra named all four correctly.

Two harder tests followed. Three diffusion models of 12×12 images, 23,824 weights each, were identified correctly, though Astra drew each shape as an outline when the models generate filled ones — a sign it was reading geometry rather than simulating the network. The third test stripped the input and output layers from three 7.16-million-weight character transformers, leaving no way to map characters to dimensions. Allowed Python but forbidden from running the network, Astra named the right language in both attempts and found every copying head while naming no others, using a technique published by Elhage et al. that the prompt never mentioned.

Writing and repairing

The cleanest repair is Experiment 4. A PPO policy trained for a billion steps to drive a car up a ramp had settled into a local minimum: it drove straight at the ramp, stalled, and failed in all 256 test episodes. Given the 171,664 weights, a scene description and one recorded episode, Astra changed 3.5% of the weights in a single attempt with no tools, adding a path that reads the car’s position and speed and hands control back to the trained network once the car clears the strip. Success rose from 0% to 100% across 1,024 episodes.

Experiment 5 starts from nothing: every weight is zero, and Astra gets only the rules and source code — no data, loss, gradients, tools or rollouts — and writes the weights in one reply. The result outscored PPO trained for 100 million steps on all three MinAtar games. Against 500 million steps it split: 517.4 to 25.3 in Breakout, 53.2 to 55.2 in Asterix, but only 53.1 to 88.9 in Seaquest.

The Qwen edit, and its limits

The most-quoted result is Experiment 8. Qwen3.5-0.8B with thinking enabled repeats itself until it hits the token limit, a failure its own model card warns about — 85 of its first 96 practice answers ran to the 32,768-token cap. Astra traced the looping to a single direction written by the MLPs of layers 15 and 19 and subtracted it from 64 output columns in each: 131,072 weights, 0.016% of the model. Given 50 forward passes to test against practice problems, MATH-500 rose from 7.9% to 48.1%, GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%.

The control is the part worth keeping. Gemma 3 270M rarely loops, reaching its limit on 8.5% of answers, and there Astra could not improve mathematics: edits moved MATH-500 from 6.8% to at best 7.7%, a gain the authors call not established. Their conclusion is narrow — an edit of this kind can remove an obstacle that hid an ability, but it could not add an ability the model lacked. A related experiment found the same boundary on the reading side: shown two Gemma models, one tuned to answer correctly and one to agree with the user, Astra could not tell the sycophant from weights alone, but did pick it in all three pairs once allowed to observe attention on texts it wrote itself.

Who else can do it

The report used one model, leaving open whether the capability belongs to GPT-6 Astra or to frontier models generally. Two replications published on November 3 suggest the latter. Simon Willison ran the paper’s own prompts against Qwen 3.8 27B on a laptop, using a 17 GB quantized build, and it named Pac-Man (72% confidence), Pong (75%) and Breakout (88%) — in 46 minutes, 1 hour 5 minutes and 52 minutes — before misreading Freeway as a Pac-Man variant. Kimi K3.1 Pro got Breakout right in 9 minutes 41 seconds.

A second replication tried something the report does not: steering a behaviour rather than repairing a failure. Valerio Dolci had Claude Opus 5.5 edit Gemma 4 12B-it so that an English past perfect describing an action still in progress renders as Italian imperfetto plus da rather than trapassato plus per — and only in that case. Over about two hours and 27 to 30 forward passes, Claude rewrote four to eight MLP neurons, 92,000 of the model’s 11.9 billion parameters, a 100 KB diff. The share of target sentences in the unwanted form fell from 26% to 12%, and from 32% to 23% on an independently written test set, while 182 of 200 unrelated translations stayed identical word for word. A LoRA trained on the same data shifted as many sentences but changed 80 of those 200.

What This Means

The report is a blog post with released logs, not a peer-reviewed paper, and it says so. Both critiques accept the results while disputing the framing. Matthew Honnibal, creator of spaCy, notes that the setting where the model sees only raw floats, with no tools and no execution, appears in just three experiments, on networks of 17,000 to 172,000 weights; every experiment on a real language model used Python, and the Qwen run also got up to 50 batches of practice problems. Sean Goedecke argues that a competent human researcher could have done the same work, that the game-identification step reduces to a three-line numpy calculation, and that editing weights directly resembles patching a compiled binary when the source is available.

What survives is still notable. Reading structure out of weights without a forward pass is a different instrument from probing activations, and the authors’ motivating case — auditing a model for a backdoor without a test set built to find it — needs exactly that. The 0.016% edit matters for a separate reason: it leaves the rest of the model untouched, avoiding the catastrophic forgetting that fine-tuning risks. The authors are now building a benchmark and harness to make the capability measurable, which is the right next step for a result resting on one model and a few dozen case studies.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

• Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights — Takehi, Teoh, Zhang and Khattab, MIT CSAIL, November 2, 2026
• RikiyaT/lm_understand-logs — prompts, responses, executed code and evaluation logs for all eight experiments
• Qwen3.5-0.8B-weight-space-programming — the edited model from Experiment 8, rebuilt and re-evaluated at Hugging Face
• Qwen 3.8 27B on my laptop can interpret model weights, too — Simon Willison, November 3, 2026
• frontier-weight-edit — Claude Opus 5.5 steering Gemma 4 12B-it through a weight edit, with baselines and results
• The ‘Astra can read and write weights to improve Qwen’ post is misleading — Matthew Honnibal, November 3, 2026
• LLMs are not superhuman at reading and writing neural network weights — Sean Goedecke, November 3, 2026
• Qwen3.5-0.8B model card — documents the reasoning-loop behaviour behind Experiment 8