AdvancedAI
EUROPA Tests a 951.9B-Parameter Model on the JUPITER Supercomputer
- Domyn
- EuroHPC
- JUPITER
- mixture of experts
- open weights
The EUROPA consortium published an account on November 12, 2026 of running a 951.9-billion-parameter mixture-of-experts training setup on JUPITER, Europe’s first exascale supercomputer, and of the work it took to make a shared, publicly funded machine carry a workload it was not configured for. The headline number is throughput: 90 tokens per second per GPU when the team started, about 1,300 when it finished tuning. The document is also explicit about what it does not yet show, and the model’s weights, licence, training data and release date remain unpublished.
A science machine, used from the outside
JUPITER’s Booster module holds 5,884 accelerated compute nodes, each with four NVIDIA GH200 Grace Hopper superchips — 23,536 superchips in total. Every one pairs a 72-core Arm Grace CPU with 120 GB of LPDDR5X at 512 GB/s to a Hopper GPU with 96 GB of HBM3 at 4 TB/s, joined by a 900 GB/s NVLink-C2C link. The consortium had user-level access only, so it could not change the kernel, the interconnect, the node memory configuration or the system software, and hundreds of other projects run on the same machine.
That access level turns an ordinary parallelism choice into a bandwidth problem. Within a node, GPUs talk to each other at 150 GB/s per direction over NVLink 4, but all traffic leaving the node squeezes through four InfiniBand NDR200 adapters, about 25 GB/s per GPU. Measured against the GPU-to-own-CPU link, the inter-node fabric is 36 times slower.
| Link | Bandwidth | Relative to InfiniBand |
|---|---|---|
| Grace to Hopper (NVLink-C2C) | 900 GB/s | 36× |
| GPU to GPU, same node (NVLink 4) | 150 GB/s per direction | 6× |
| GPU to outside the node (InfiniBand) | ~25 GB/s per GPU | 1× |
The layout follows from that: 384-way fully sharded data parallelism, with 4-way expert parallelism confined inside each node, and no pipeline parallelism. The mixture-of-experts token all-to-all is hard to overlap with computation, so it stays on NVLink — one expert-parallel group per node. Steps are kept short to shorten the feedback loop at scale, and fixed costs are amortised with a large batch of 384 sequences of 65,536 tokens, roughly 25 million tokens per step. The reported runs use a single data-parallel replica; four replicas across 1,536 GPUs keep 89.4% of the throughput of one.
From 90 to 1,293 tokens per second
The baseline profiled step on rank 0 lasted 83.6 seconds. Computation took 14.3 seconds of it; 68.3 seconds went to waiting for communication. Four changes account for most of what follows.
Longer sequences with full activation recomputation. The volume of weights gathered per layer does not depend on sequence length, so longer sequences buy compute per byte of communication. At 8,192 tokens the GPU sat idle through most of the step; storing activations only at layer boundaries and recomputing inside each layer moved throughput from 90 to 370 tokens per second at 32,768 tokens.
Offloading activations to Grace. At 65,536 tokens the boundary activations alone are 74 layers × 65,536 tokens × 6,144 × 2 bytes, about 55 GiB per GPU, which does not fit in HBM beside the weights. Each boundary activation is copied to the CPU’s LPDDR5X after the forward pass produces it and brought back one layer before the backward pass recomputes it. Throughput rose from 370 to 508, up 37%, and the sequence length could grow to 49,152 and then 65,536.
Running Muon’s orthogonalisation on the Hopper. Muon holds one fp32 optimiser state per parameter where Adam holds two, and that is what makes a 65,536-token sequence fit at all; it covers every weight matrix except the router and embeddings, about 99.5% of parameters. Muon orthogonalises each update with Newton-Schulz iterations over whole matrices, which FSDP had scattered across 384 shards. The consortium’s FlexShard, now part of TorchTitan, shards experts by whole expert so each rank owns complete matrices and runs Newton-Schulz with no communication, while dense matrices are gathered on an owning rank and other ranks keep working. On the 72 Arm cores at roughly 35 seconds per step the optimiser dominated; moving it to the GPU while parameters stayed offloaded cut the step from 41.7 to 2.2 seconds and lifted throughput 35%, from 441 to 597.
Fixing where FSDP allocated its buffers. After a software upgrade the CUDA caching allocator began retrying allocations on every one of the 384 ranks at around 70% memory utilisation, with 25 to 30 GiB of HBM nominally free. The allocator keeps a separate memory pool per stream, and FSDP’s side streams were sitting on 27.3 GiB of which only 6.4 GiB was live; the remaining 20.9 GiB was out of reach of the compute stream, which then could not find a 4 GiB block for a mixture-of-experts buffer. A private FSDP2 flag, unshard_async_op, runs the all-gather setup on the compute stream so buffers are allocated from and returned to its pool. Allocator retries fell from 384 ranks to zero, peak reserved memory from 88.1 to 74.6 GiB, and throughput rose from 996 to 1,210.

What went back upstream
The remaining gains in the waterfall came from load balancing across ranks, sending expert weights in FP8 and gradients in bf16 to cut bytes on the wire, node-local resharding, a dynamic GEMM scheduler and capping NCCL at 16 CTAs. Almost all of it was contributed back, on the argument that other users of JUPITER and similar EuroHPC systems should be able to reproduce it: FlexShard, activation-checkpoint offloading hooks, the unshard_async_op option now moving into a public FSDP API, a shared RoPE cache worth 2.56 GiB per GPU at this sequence length, skipping the embedding re-gather in the backward pass, and configurable seed-offset mesh axes. In PyTorch itself, fixes for CPU initialisation resetting the RNG state and for trunc_normal_ all-gathering DTensors. The team also reported unattributed host memory at job start to the Jülich Supercomputing Centre, because it affects every user of the machine.
What the Document Says
The Frontier AI Grand Challenge was launched by the European Commission with the EuroHPC Joint Undertaking on 13 February 2026 under the Apply AI Strategy. The call invited proposals to “develop frontier AI models with a computational capacity equivalent to at least 400 billion parameters”, with one proposal selected. The selected project receives “up to 2.5% of the overall EuroHPC computing capacity for one year on one or more AI-optimised EuroHPC supercomputers”, and the challenge is “expected to deliver open models that will be made widely available to public authorities, scientific communities and businesses across Europe”.
On 19 June 2026 the Commission named EUROPA, a consortium led by the Milan-based company Domyn with the Fraunhofer institutes IAIS and IIS, as the winner. The consortium’s own post records the three conditions it was selected on: an open European mixture-of-experts model of at least 400 billion parameters, support for all 24 official EU languages, and strict compliance with the EU AI Act. Henna Virkkunen, Executive Vice-President for Tech Sovereignty, Security and Democracy, said in the Commission’s release: “Europe can lead in advanced AI on its own terms. EUROPA will build a frontier European AI model in all 24 EU languages, showing that we can match the best while staying true to our values.”
On the model itself the post is deliberately provisional. It opens by saying “we have recently started training a frontier-scale model”, while the configuration it documents is the one “we currently use for experiments”, with a note that “some values may change before the training run starts” and a scope statement that “the setup is still changing”. The figures are steady-state medians over steps 4 to 10. At about 1,300 tokens per second per GPU, one 384-GPU replica works out to roughly 43 billion tokens a day, and four replicas to roughly 150 billion.
Neither the Commission’s release nor the consortium’s post states EUROPA’s licence, training data, language mix, benchmark results or release date. What has shipped is a smaller model: Domyn Edge, a 4-billion-parameter dense transformer trained from scratch on 13 trillion tokens, whose first bundle was released on 20 October 2026 under Apache 2.0 with intermediate checkpoints, pre-training data and training code, and which is natively trained to a 1M-token context.
This post was drafted with AI assistance and reviewed by RITS staff.
Sources
- PyTorch: Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan — EUROPA consortium, 12 November 2026
- Domyn: Training a 1T-Parameter MoE on EU public infrastructure (full write-up) — 12 November 2026
- European Commission: Commission selects EUROPA consortium as the winner of the Frontier AI Grand Challenge — press release, 19 June 2026
- European Commission: Turning strategy into action — Commission launches Frontier AI Grand Challenge — call, 13 February 2026
- Forschungszentrum Jülich: JUPITER technical overview
- Jülich Supercomputing Centre: JUPITER configuration documentation
- EuroHPC Joint Undertaking: Our supercomputers
- Hugging Face: domyn/Domyn-Edge-Base model card
- GitHub: pytorch/torchtitan






