AdvancedAI

EUROPA Tests a 951.9B-Parameter Model on the JUPITER Supercomputer

The EUROPA consortium published an account on November 12, 2026 of running a 951.9-billion-parameter mixture-of-experts training setup on JUPITER, Europe’s first exascale supercomputer, and of the work it took to make a shared, publicly funded machine carry a workload it was not configured for. The headline number is throughput: 90 tokens per second per GPU when the team started, about 1,300 when it finished tuning. The document is also explicit about what it does not yet show, and the model’s weights, licence, training data and release date remain unpublished.

Title card reading Enabling 1T MoE training with PyTorch and TorchTitan on EU public infrastructure, with the Domyn logo on a dark blue geometric background
Image credit: The EUROPA consortium, published by PyTorch

A science machine, used from the outside

JUPITER’s Booster module holds 5,884 accelerated compute nodes, each with four NVIDIA GH200 Grace Hopper superchips — 23,536 superchips in total. Every one pairs a 72-core Arm Grace CPU with 120 GB of LPDDR5X at 512 GB/s to a Hopper GPU with 96 GB of HBM3 at 4 TB/s, joined by a 900 GB/s NVLink-C2C link. The consortium had user-level access only, so it could not change the kernel, the interconnect, the node memory configuration or the system software, and hundreds of other projects run on the same machine.

That access level turns an ordinary parallelism choice into a bandwidth problem. Within a node, GPUs talk to each other at 150 GB/s per direction over NVLink 4, but all traffic leaving the node squeezes through four InfiniBand NDR200 adapters, about 25 GB/s per GPU. Measured against the GPU-to-own-CPU link, the inter-node fabric is 36 times slower.

Diagram of one JUPITER node showing four Grace CPUs each linked to a Hopper GPU by a 900 GB/s NVLink-C2C connection, GPUs interconnected by NVLink 4, Grace CPUs interconnected at 100 GB/s, and thin InfiniBand links leaving the node
Node bandwidths drawn to line thickness; all inter-node traffic leaves through the thin InfiniBand links. Image credit: The EUROPA consortium, published by PyTorch
JUPITER bandwidth hierarchy, as tabulated by the consortium
Link Bandwidth Relative to InfiniBand
Grace to Hopper (NVLink-C2C) 900 GB/s 36×
GPU to GPU, same node (NVLink 4) 150 GB/s per direction 6×
GPU to outside the node (InfiniBand) ~25 GB/s per GPU 1×

The layout follows from that: 384-way fully sharded data parallelism, with 4-way expert parallelism confined inside each node, and no pipeline parallelism. The mixture-of-experts token all-to-all is hard to overlap with computation, so it stays on NVLink — one expert-parallel group per node. Steps are kept short to shorten the feedback loop at scale, and fixed costs are amortised with a large batch of 384 sequences of 65,536 tokens, roughly 25 million tokens per step. The reported runs use a single data-parallel replica; four replicas across 1,536 GPUs keep 89.4% of the throughput of one.

From 90 to 1,293 tokens per second

The baseline profiled step on rank 0 lasted 83.6 seconds. Computation took 14.3 seconds of it; 68.3 seconds went to waiting for communication. Four changes account for most of what follows.

Waterfall chart titled Throughput per change, 90 to 1,293 tokens per second per GPU, showing fourteen cumulative optimisations from baseline FSDP at 90 through full recompute at 370, activation offload at 508 and newer Hopper at 597 up to 1,293 tokens per second
Steady-state throughput, median of steps 4 to 10 on rank 0, for each cumulative change. Image credit: The EUROPA consortium, published by PyTorch

Longer sequences with full activation recomputation. The volume of weights gathered per layer does not depend on sequence length, so longer sequences buy compute per byte of communication. At 8,192 tokens the GPU sat idle through most of the step; storing activations only at layer boundaries and recomputing inside each layer moved throughput from 90 to 370 tokens per second at 32,768 tokens.

Offloading activations to Grace. At 65,536 tokens the boundary activations alone are 74 layers × 65,536 tokens × 6,144 × 2 bytes, about 55 GiB per GPU, which does not fit in HBM beside the weights. Each boundary activation is copied to the CPU’s LPDDR5X after the forward pass produces it and brought back one layer before the backward pass recomputes it. Throughput rose from 370 to 508, up 37%, and the sequence length could grow to 49,152 and then 65,536.

Running Muon’s orthogonalisation on the Hopper. Muon holds one fp32 optimiser state per parameter where Adam holds two, and that is what makes a 65,536-token sequence fit at all; it covers every weight matrix except the router and embeddings, about 99.5% of parameters. Muon orthogonalises each update with Newton-Schulz iterations over whole matrices, which FSDP had scattered across 384 shards. The consortium’s FlexShard, now part of TorchTitan, shards experts by whole expert so each rank owns complete matrices and runs Newton-Schulz with no communication, while dense matrices are gathered on an owning rank and other ranks keep working. On the 72 Arm cores at roughly 35 seconds per step the optimiser dominated; moving it to the GPU while parameters stayed offloaded cut the step from 41.7 to 2.2 seconds and lifted throughput 35%, from 441 to 597.

Fixing where FSDP allocated its buffers. After a software upgrade the CUDA caching allocator began retrying allocations on every one of the 384 ranks at around 70% memory utilisation, with 25 to 30 GiB of HBM nominally free. The allocator keeps a separate memory pool per stream, and FSDP’s side streams were sitting on 27.3 GiB of which only 6.4 GiB was live; the remaining 20.9 GiB was out of reach of the compute stream, which then could not find a 4 GiB block for a mixture-of-experts buffer. A private FSDP2 flag, unshard_async_op, runs the all-gather setup on the compute stream so buffers are allocated from and returned to its pool. Allocator retries fell from 384 ranks to zero, peak reserved memory from 88.1 to 74.6 GiB, and throughput rose from 996 to 1,210.

Two profiler traces of rank 0. The upper trace shows a 144.7 second step with 34.3 seconds idle as Newton-Schulz runs on Grace; the lower shows a 110.9 second step with 2.3 seconds idle after moving the optimiser to the Hopper
Rank 0’s step with Newton-Schulz on Grace, top, and on the Hopper, bottom. Image credit: The EUROPA consortium, published by PyTorch

What went back upstream

The remaining gains in the waterfall came from load balancing across ranks, sending expert weights in FP8 and gradients in bf16 to cut bytes on the wire, node-local resharding, a dynamic GEMM scheduler and capping NCCL at 16 CTAs. Almost all of it was contributed back, on the argument that other users of JUPITER and similar EuroHPC systems should be able to reproduce it: FlexShard, activation-checkpoint offloading hooks, the unshard_async_op option now moving into a public FSDP API, a shared RoPE cache worth 2.56 GiB per GPU at this sequence length, skipping the embedding re-gather in the backward pass, and configurable seed-offset mesh axes. In PyTorch itself, fixes for CPU initialisation resetting the RNG state and for trunc_normal_ all-gathering DTensors. The team also reported unattributed host memory at job start to the Jülich Supercomputing Centre, because it affects every user of the machine.

What the Document Says

The Frontier AI Grand Challenge was launched by the European Commission with the EuroHPC Joint Undertaking on 13 February 2026 under the Apply AI Strategy. The call invited proposals to “develop frontier AI models with a computational capacity equivalent to at least 400 billion parameters”, with one proposal selected. The selected project receives “up to 2.5% of the overall EuroHPC computing capacity for one year on one or more AI-optimised EuroHPC supercomputers”, and the challenge is “expected to deliver open models that will be made widely available to public authorities, scientific communities and businesses across Europe”.

On 19 June 2026 the Commission named EUROPA, a consortium led by the Milan-based company Domyn with the Fraunhofer institutes IAIS and IIS, as the winner. The consortium’s own post records the three conditions it was selected on: an open European mixture-of-experts model of at least 400 billion parameters, support for all 24 official EU languages, and strict compliance with the EU AI Act. Henna Virkkunen, Executive Vice-President for Tech Sovereignty, Security and Democracy, said in the Commission’s release: “Europe can lead in advanced AI on its own terms. EUROPA will build a frontier European AI model in all 24 EU languages, showing that we can match the best while staying true to our values.”

On the model itself the post is deliberately provisional. It opens by saying “we have recently started training a frontier-scale model”, while the configuration it documents is the one “we currently use for experiments”, with a note that “some values may change before the training run starts” and a scope statement that “the setup is still changing”. The figures are steady-state medians over steps 4 to 10. At about 1,300 tokens per second per GPU, one 384-GPU replica works out to roughly 43 billion tokens a day, and four replicas to roughly 150 billion.

Neither the Commission’s release nor the consortium’s post states EUROPA’s licence, training data, language mix, benchmark results or release date. What has shipped is a smaller model: Domyn Edge, a 4-billion-parameter dense transformer trained from scratch on 13 trillion tokens, whose first bundle was released on 20 October 2026 under Apache 2.0 with intermediate checkpoints, pre-training data and training code, and which is natively trained to a 1M-token context.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

Related stories

cybersecurity

Mistral Releases Le Chonk Weights: 1T Parameters, Open

Mistral published the weights of Mistral Large 4, nicknamed “Le Chonk,” on October 29, 2026, three weeks after the model entered public preview. It is a 1-trillion-parameter multimodal mixture-of-experts model with 52 billion active parameters, a 1M-token context, and a licence that keeps one commercial use — hosting it as a service — under restriction. […]

agentic ai

Mistral Large 4: Europe’s 1T Open-Weight Model Enters Preview

On October 6, 2026, Mistral AI opened a public preview of Mistral Large 4 — a 1.05-trillion-parameter, natively multimodal model with 52 billion active parameters. Nicknamed “le Chonk” and unofficially ML4, it is the Paris-based lab’s largest model to date and, on Mistral’s own numbers, the strongest open-weight model developed outside China. The preview API […]

Never miss a RITS update

Subscribe to the RSS feed for AI, emerging technology, and digital scholarship news curated for the NYU Shanghai community, or get in touch to suggest a story.