AdvancedAI
miHoYo’s Gamer Plays Star Rail From Pixels and Keystrokes
- agentic ai
- benchmark
- game agents
- mihoyo
miHoYo’s Honkai AI R&D Team posted Gamer to arXiv on 13 November 2026: a vision-language-action model that plays commercial games the way a person does, turning 5 Hz screen captures into keyboard and mouse events with no access to game memory or a scripting interface. Its best checkpoints clear 80% and 83.6% of two held-out Honkai: Star Rail side quests and average 49.7% on a 40-instance minigame benchmark. The paper also carries a negative result worth as much as the headline: extra gameplay footage stops helping once the target game drifts from the training mixture.
Pixels in, keystrokes out
Most celebrated game-playing systems of the past decade had privileged access: Atari emulators that expose RAM, Minecraft APIs, StarCraft interfaces built for bots. A live commercial title offers none of that. Gamer starts from Qwen3-VL-8B and reuses the language model’s own output vocabulary, so an action is just text. Each 200 ms interval yields one observation and six action blocks of about 33 ms, an effective 30 Hz control rate; every block carries accumulated mouse movement plus the keys and buttons held, so held keys, taps, drags and simultaneous inputs all fall out of the same representation, with no extra output head.
Two choices in that stage matter. More than 85% of action steps repeat the preceding one, so consecutive repeats are down-weighted exponentially at γ = 0.9, following Game-TARS, which shifts supervision toward transitions such as stopping and turning. And because bindings differ between titles, each trajectory gets a random one-to-one remapping of rebindable keys, applied in both the control specification and the action labels, so the model learns “move left” rather than “the A key”.
Pre-training uses about 2.4K hours of curated gameplay from VPT, D2E, OpenP2P and miHoYo’s own recordings, at a ratio of 85% gameplay to 15% general vision-language data; the widest of the five scaling mixtures spans 80 distinct game worlds.
Three stages: control, instruction, self-play
Instruction tuning is where the annotation budget goes. The Star Rail recordings hold about 68K sessions and 11K hours; filtering corrupt, duplicated and idle material leaves 3.2K sessions and 1K hours, some 149K aligned clips and 332 hours inside the pre-training corpus. GPT-5.6-Luna then annotates sparse “Think” notes at decision points (about 1.2M), task segments with instructions and scene descriptions (about 63K) and first-person memory summaries roughly every 16 seconds (about 223K), split across operations, subgoals and complete tasks.
Memory is a summary budget, not a longer context: when the window overflows, the policy writes a first-person note about what it has completed before those rounds are evicted. Reasoning is sparse by design — routine movement emits an action directly, and only decisions get an explicit Think block, decided inside the same generation pass.
The reinforcement stage is where the engineering gets unusual. Temporal-Aware GRPO slices long rollouts into segments linked by those summaries, so a 100-minute quest never has to fit in one context, and scores them with binary rewards verified by the game engine — no value critic, no learned reward model. That required instrumenting a full game client, with game-master commands to restore initial states and per-milestone verifiers to judge progress; rollouts run on 32 parallel game instances served by 64 NVIDIA H200s.
The scoreboard
Evaluations use held-out quests and minigames inside Star Rail itself, so this measures transfer to new objectives in a familiar game, not zero-shot play. Each task runs three times, and accuracy counts verified milestone progress.
| Model | Opportunity Makes the Thief (I) 30 milestones, 30 min |
Bestial Ferocity 55 milestones, 100 min |
|---|---|---|
| Best general-purpose baseline | 23.3 (DeepSeek-V4-Flash-Vision) | 45.5 (Gemini-3.8-Flash) |
| Qwen3-VL-8B base | 0.0 | 13.9 |
| Gamer-CPT | 34.4 | 47.9 |
| Gamer-SFT | 72.2 | 69.1 |
| Gamer-RL | 80.0 | 83.6 |
Combat and interface interaction reach 100% on both quests after reinforcement learning; dialogue reaches 88.1% and 100%, navigation 69.1% and 80.2%. On minigames the seven-type macro-average climbs from 3.2% to 15.0%, 39.2% and 49.7%, against 17.1% for GPT-6.1-Sol, the best of seven general-purpose models compared. Navigation Compass alone goes from 50.0% to 95.8% after reinforcement learning.
On four public benchmarks the comparison turns on how fast actions arrive. The two pre-trained checkpoints — 8B dense and a 35B-A3B mixture-of-experts — average 29.5 and 31.5 under real-time evaluation, against 17.6 for GPT-5.6-Sol. The gap between pausing the game during inference and letting it run is the telling number: general-purpose models lose 17 to 35 points on average, with GPT-5.6-Sol dropping 73.3 points on MCU, while the mixture-of-experts checkpoint gains 3.4.
Latency as a training variable
That gap is why the serving stack is part of the method. Streaming-vLLM keeps each environment’s KV cache across turns with incremental rotary-embedding correction after compaction, verifies structural action tokens with a finite-state machine, and dispatches the mouse fields and first verified key block before decoding finishes. On an H200 with the 35B-A3B model, end-to-end latency falls from 1,306 ms to 65.12 ms — a 20.1× reduction, with a fused megakernel alone taking step latency from 537 to 88 ms — and step rate rises from 0.77 to 15.36 Hz.
Accuracy holds: with a Qwen3-VL-8B backbone, Streaming-vLLM scores 83.29 on StreamingBench against 83.21 for the DSCache baseline and 62.3 on OVO-Bench against 58.2, at 0.08 s versus 0.53 s per query. The same machinery feeds training. Collected the ordinary way at 0.6 frames per second, reward curves across eight navigation tasks are lower, and several collapse in later iterations.
What holds the headline back
The scaling study deserves first read. Across 120 configurations — four backbones from 0.8B to 27B, six budgets from 50 to 1,600 hours, five mixtures varying Minecraft’s share from 100% to zero, roughly 45K H200 GPU-hours — a composition-aware fit explains training loss well, reaching a within-mixture R² of 0.959 on the training-game distribution. On distant unseen games the fitted data exponent lands near zero and the within-mixture R² falls to 0.079: hours stop buying transfer, and coverage matters more than volume.
Caveats are the authors’ own: a selected set of environments, deliberation still a bottleneck (a System 2 planner over a System 1 controller is sketched), little gain for pre-trained checkpoints on the most distant benchmark, and one minigame where reinforcement learning hurts — Janus’s Maze falls from 52.4% to 23.8%. All baselines ran on the authors’ harness, in one game.
Release scope is the sharper limit. The GitHub repository is Apache-2.0 and includes the system prompt and action-format documentation, and the Hugging Face collection holds eight GameScaling checkpoints from 0.8B to 27B — the controlled scaling study published in October, not the Star Rail-tuned model. The project page states plainly that the Star Rail data and any checkpoint trained or fine-tuned on it will not be public “for data security reasons”. Streaming-vLLM appears in the paper’s own comparison table as open-source, yet still sits on the repository’s to-do list.
What This Means
Games are one of the few training environments that ship their own verifier: nothing in Gamer’s rewards comes from a model judging another model. Verified binary rewards, no privileged state and a universal keyboard-and-mouse action space are what make a commercial title usable as a reinforcement-learning gym, and they point at automated QA and regression testing before anything a player would notice.
The second lesson is that latency belongs to the capability. A policy planning over a screen it saw 1.3 seconds ago is solving a different, harder problem than one that acts within 65 ms, and the real-time spreads show how much of the baselines’ apparent game skill was an artefact of a frozen game. The third is that self-reported negatives are useful, and that independent verification is what remains missing while the weights behind the headline numbers stay inside miHoYo.
This post was drafted with AI assistance and reviewed by RITS staff.
Sources
- arXiv:2611.18397 — Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent — miHoYo Honkai AI R&D Team with NUS, Tsinghua, UCAS, USTC and Xi’an Jiaotong University, 13 November 2026
- Gamer project page — miHoYo Honkai AI R&D Team
- GitHub: RainBowLuoCS/Gamer — Apache-2.0 repository, system prompt and action format
- Hugging Face: THUSI-Lab GameScaling model collection — the eight released scaling checkpoints
- AI Modeling: miHoYo Trains an 8B Model to Play Star Rail in Real Time — 16 November 2026 analysis
- AGI Hunt: miHoYo’s Gamer, a pixel-to-keystroke agent — summary of method and results
- TechNode: MiHoYo team open-sources GameScaling models — the October scaling release behind the 120-configuration study






