A 763B Model in 2 Bits: What ISTA-DASLab’s GGUFs Give Up

ISTA-DASLab published non-uniform GGUF quantizations of DeepSeek-V4.1-Flash-1017 on October 23, 2026, compressing a 763.2-billion-parameter mixture-of-experts model to 1.75, 2.00 and 2.25 bits per weight under an MIT licence. The lab’s benchmark table says the middle build gives up nothing: 100.0% of the base model’s task average across AIME25, GPQA-Diamond and LiveCodeBench v6. The runtime built to serve those same files reports a different number for the same weights — perplexity 15.5% above the original at 2.00 bits, and 7.3% at 2.25. Both measurements are defensible. The gap between them is what a quantization actually costs, and why a three-benchmark table cannot show it.

What the Files Actually Are

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with 384 routed experts per layer, six active per token, and a one-million-token context window. The backbone holds 551.9 billion parameters; a separate Engram file holds two lookup tables totalling 196.6 billion. Because the routed experts dominate both the parameter count and the size budget, the lab’s search spends most of its freedom there — assigning a quantization type per layer rather than per expert, which GGUF cannot express.

Two methods produced the files, both from IST Austria’s Distributed Algorithms and Systems Lab. GSQ (arXiv 2604.18556) learns each coordinate’s grid assignment and each group’s scale jointly through a Gumbel-Softmax relaxation, closing most of the accuracy gap between scalar and vector quantization at two to three bits while staying readable by ordinary kernels. RCO (arXiv 2605.00649) then picks one of K types for each of N tensors under an exact size budget, reframing the constraint as a Riemannian manifold in logit space so a gradient step optimizes real loss rather than a proxy.

The result is three builds differing only in how much precision the search could afford, each a four-shard backbone plus the same shared Engram file, held at IQ4_NL in every build because it is a lookup table read one row at a time rather than a matmul weight.

Build	Bits per weight	Backbone	Total with Engram
IQ1_M	1.75	120.72 GB	231.32 GB
IQ2_XXS	2.00	138.24 GB	248.84 GB
IQ2_XS	2.25	155.23 GB	265.83 GB

Two Ways of Measuring the Same Weights

ISTA-DASLab reports three benchmarks, all at reasoning effort 50: AIME25, GPQA-Diamond and LiveCodeBench v6. The base model scores 100.00, 90.91 and 89.71, a task average of 93.54. IQ2_XXS reproduces all three exactly. IQ2_XS scores 90.40 on GPQA-Diamond for 99.8% recovery, and IQ1_M drops to 88.38 on GPQA-Diamond and 88.57 on LiveCodeBench for 98.7%.

Read that table as an accuracy test and it is close to a clean sweep. Read it as proof that nothing was lost, and the ceiling gets in the way: AIME25 is already at 100.00 for the base model, so no build can score lower without failing a problem the original got right, and every remaining difference is one or two problems out of a short set. The lab says as much on its own discussion board. Asked why the 2.25-bit build scored below the 2.00-bit one, a maintainer replied: “No, that’s just benchmark noise. We can’t afford to run each benchmark multiple times, so it’s totally possible that in one run, a larger model just solves one fewer problem and ends up with a lower score.” The lab is, in its own words, still looking for benchmarks “that can better show the differences between the models.”

Strata, the open-source engine that added DeepSeek-V4.1 support in version 0.2.4 on October 24, measures something else. Its documentation reports quality against the original model across 4,995 text positions using KL divergence and perplexity — distribution-level metrics that score every token the model predicts, including the ones no benchmark question ever reaches. On that test the 2.00-bit build has KL 0.170 and perplexity 15.5% above the original, with code up 8.6% and agent work up 5.3%. The 2.25-bit build has KL 0.107 and perplexity up 7.3%, with code up 3.2%. The 1.75-bit build is a different animal: perplexity up 21%, top-1 agreement 82.5%, documentation up 24%, multilingual text up 14%, and plain English prose up 53% — which is why Strata recommends it only where 2.00 bits will not fit.

A second engine project reached a similar conclusion independently. Xolotl, a Strata fork for running these models from one gaming PC, loaded IQ2_XS and measured perplexity 3.93 against 3.60 for its own build at the same flags — one run on a small sample, so a data point rather than a verdict, but pointing the same way. Xolotl also notes the ISTA files are text-only there, because they do not carry the router’s image bias.

Where the Rest of the Loss Comes From

Not all of the gap between a benchmark pass and a served answer belongs to the weights. Strata’s own defaults move the number. On a PC whose graphics cards total 16 GiB of VRAM or less — the 12 GB class these builds target — the engine defaults to its “faster” setting, which uses five of the six routed experts DeepSeek selects per token. Its documentation calls that “a small quality change, usually not noticeable” and makes full quality an explicit choice. The default context is 8,192 tokens against a model card recommending one million, the dense half of the network is re-quantized for speed, and the Engram tables page in from NVMe on every token.

Those are documented, reversible trade-offs, and they are how a 763B model answers at all on a card that cannot hold 1% of it. Measured on the 2.00-bit set: an RTX 5070 with 64 GB of RAM produces 4 to 6 tokens per second and reads a 4K prompt at about 110; an RTX 3060 with 46 GB of RAM reaches 9 to 11; four Radeon AI PRO R9700 cards reach 74 to 77.

Runtime support is still settling. The model card stated on October 23 that llama.cpp did not support DeepSeek-V4.1, so the GGUFs ran only in Strata; within two days a llama.cpp contributor reported on the card’s discussion page that support had merged, and the lab has since replaced a chat template carried over from V4.

What This Means

The useful lesson is not that one measurement is wrong. It is that “we recovered 100% of the base model” is a claim about three benchmarks, and a benchmark the original already aces has no room to register a regression. Perplexity and KL divergence answer a different question, about how far the whole next-token distribution moved, and they are the ones that notice a model going vaguer or less fluent in prose while still solving what it solved before.

For anyone choosing a build, the reading is narrow. If your disk allows 274 GB, IQ2_XS is the honest operating point for coding and agent work: 7.3% perplexity drift and 3.2% on code, against 15.5% and 8.6% at 2.00 bits. IQ1_M is for machines that otherwise cannot run the model at all, and its 53% prose penalty is the price of the 17.5 GB you did not spend.

The broader point concerns defaults rather than files. A local stack serving five experts instead of six, an eighth of the context the model was trained for, and a re-quantized dense half will feel worse than the table promised, and none of that is the quantizer’s fault. Before concluding that two bits ruined a model, check which knobs the engine turned for you.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

• ISTA-DASLab — DeepSeek-V4.1-Flash-1017 GSQ-RCO GGUF model card and discussion threads (October 23, 2026)
• Dadgarnia et al. — GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (arXiv 2604.18556)
• Helcig and Alistarh — Model Compression with Exact Budget Constraints via Riemannian Manifolds (arXiv 2605.00649)
• Strata — DeepSeek-V4.1-Flash guide: file sizes, perplexity and KL measurements, runtime defaults
• Strata — open-source local inference engine (version 0.2.4 added DeepSeek-V4.1 support)
• Indras-Mirror/Xolotl — Strata fork, notes on loading the ISTA-DASLab GGUFs and independent perplexity figures
• DeepSeek — DeepSeek-V4.1-Flash-1017 model card