AdvancedAI

miHoYo’s Gamer Plays Star Rail From Pixels and Keystrokes

miHoYo’s Honkai AI R&D Team posted Gamer to arXiv on 13 November 2026: a vision-language-action model that plays commercial games the way a person does, turning 5 Hz screen captures into keyboard and mouse events with no access to game memory or a scripting interface. Its best checkpoints clear 80% and 83.6% of two held-out Honkai: Star Rail side quests and average 49.7% on a 40-instance minigame benchmark. The paper also carries a negative result worth as much as the headline: extra gameplay footage stops helping once the target game drifts from the training mixture.

Grid of forty game screenshots spanning Genshin Impact, Honkai: Star Rail, Zenless Zone Zero, PUBG, Minecraft, Cyberpunk 2077, GTA V, DOOM Eternal, Counter-Strike 2 and farming and racing simulations, surrounding a centre panel labelled A Generalist Game Agent for Diverse Games that shows a screen observation feeding a unified game-playing model with short-term history, long-term memory and reasoning, which outputs keyboard and mouse actions
The overview figure released with Gamer shows forty titles across shooters, role-playing games, racing and simulation, with one observation-to-action loop in the middle. Image credit: miHoYo Honkai AI R&D Team, Gamer project page

Pixels in, keystrokes out

Most celebrated game-playing systems of the past decade had privileged access: Atari emulators that expose RAM, Minecraft APIs, StarCraft interfaces built for bots. A live commercial title offers none of that. Gamer starts from Qwen3-VL-8B and reuses the language model’s own output vocabulary, so an action is just text. Each 200 ms interval yields one observation and six action blocks of about 33 ms, an effective 30 Hz control rate; every block carries accumulated mouse movement plus the keys and buttons held, so held keys, taps, drags and simultaneous inputs all fall out of the same representation, with no extra output head.

Diagram of Gamer's continued pre-training pass: a row of system, vision and action token blocks passes through a central Gamer model, with callouts for random key remapping, which turns a move-left intent bound to A into the same intent bound to D, and repeat-aware loss decay, which down-weights long runs of identical held keys
Two de-biasing tricks from the continued pre-training stage. Image credit: Gamer technical report, arXiv:2611.18397

Two choices in that stage matter. More than 85% of action steps repeat the preceding one, so consecutive repeats are down-weighted exponentially at γ = 0.9, following Game-TARS, which shifts supervision toward transitions such as stopping and turning. And because bindings differ between titles, each trajectory gets a random one-to-one remapping of rebindable keys, applied in both the control specification and the action labels, so the model learns “move left” rather than “the A key”.

Pre-training uses about 2.4K hours of curated gameplay from VPT, D2E, OpenP2P and miHoYo’s own recordings, at a ratio of 85% gameplay to 15% general vision-language data; the widest of the five scaling mixtures spans 80 distinct game worlds.

Three stages: control, instruction, self-play

Instruction tuning is where the annotation budget goes. The Star Rail recordings hold about 68K sessions and 11K hours; filtering corrupt, duplicated and idle material leaves 3.2K sessions and 1K hours, some 149K aligned clips and 332 hours inside the pre-training corpus. GPT-5.6-Luna then annotates sparse “Think” notes at decision points (about 1.2M), task segments with instructions and scene descriptions (about 63K) and first-person memory summaries roughly every 16 seconds (about 223K), split across operations, subgoals and complete tasks.

Diagram of Gamer's supervised fine-tuning sequence, showing system, memory, vision, reasoning and action tokens passing through the model, with context memory compaction summarising history before eviction and adaptive thinking emitting reasoning only at selected decision points
Reasoning and memory updates share the same autoregressive sequence as the key and mouse actions. Image credit: miHoYo Honkai AI R&D Team, Gamer project page

Memory is a summary budget, not a longer context: when the window overflows, the policy writes a first-person note about what it has completed before those rounds are evicted. Reasoning is sparse by design — routine movement emits an action directly, and only decisions get an explicit Think block, decided inside the same generation pass.

The reinforcement stage is where the engineering gets unusual. Temporal-Aware GRPO slices long rollouts into segments linked by those summaries, so a 100-minute quest never has to fit in one context, and scores them with binary rewards verified by the game engine — no value critic, no learned reward model. That required instrumenting a full game client, with game-master commands to restore initial states and per-milestone verifiers to judge progress; rollouts run on 32 parallel game instances served by 64 NVIDIA H200s.

The scoreboard

Evaluations use held-out quests and minigames inside Star Rail itself, so this measures transfer to new objectives in a familiar game, not zero-shot play. Each task runs three times, and accuracy counts verified milestone progress.

Verified task-execution accuracy (%): two held-out Star Rail side quests
Model Opportunity Makes the Thief (I)
30 milestones, 30 min
Bestial Ferocity
55 milestones, 100 min
Best general-purpose baseline 23.3 (DeepSeek-V4-Flash-Vision) 45.5 (Gemini-3.8-Flash)
Qwen3-VL-8B base 0.0 13.9
Gamer-CPT 34.4 47.9
Gamer-SFT 72.2 69.1
Gamer-RL 80.0 83.6

Combat and interface interaction reach 100% on both quests after reinforcement learning; dialogue reaches 88.1% and 100%, navigation 69.1% and 80.2%. On minigames the seven-type macro-average climbs from 3.2% to 15.0%, 39.2% and 49.7%, against 17.1% for GPT-6.1-Sol, the best of seven general-purpose models compared. Navigation Compass alone goes from 50.0% to 95.8% after reinforcement learning.

Benchmark overview card for Honkai: Star Rail minigames listing seven game types with mechanics and task counts, bar charts of task instances per type and game types grouped by logical reasoning, spatial reasoning, temporal planning and timing and control
The minigame benchmark: 40 held-out instances across seven mechanics, scored as an unweighted mean so no single type dominates. Image credit: Gamer technical report, arXiv:2611.18397

On four public benchmarks the comparison turns on how fast actions arrive. The two pre-trained checkpoints — 8B dense and a 35B-A3B mixture-of-experts — average 29.5 and 31.5 under real-time evaluation, against 17.6 for GPT-5.6-Sol. The gap between pausing the game during inference and letting it run is the telling number: general-purpose models lose 17 to 35 points on average, with GPT-5.6-Sol dropping 73.3 points on MCU, while the mixture-of-experts checkpoint gains 3.4.

Latency as a training variable

That gap is why the serving stack is part of the method. Streaming-vLLM keeps each environment’s KV cache across turns with incremental rotary-embedding correction after compaction, verifies structural action tokens with a finite-state machine, and dispatches the mouse fields and first verified key block before decoding finishes. On an H200 with the 35B-A3B model, end-to-end latency falls from 1,306 ms to 65.12 ms — a 20.1× reduction, with a fused megakernel alone taking step latency from 537 to 88 ms — and step rate rises from 0.77 to 15.36 Hz.

Accuracy holds: with a Qwen3-VL-8B backbone, Streaming-vLLM scores 83.29 on StreamingBench against 83.21 for the DSCache baseline and 62.3 on OVO-Bench against 58.2, at 0.08 s versus 0.53 s per query. The same machinery feeds training. Collected the ordinary way at 0.6 frames per second, reward curves across eight navigation tasks are lower, and several collapse in later iterations.

What holds the headline back

The scaling study deserves first read. Across 120 configurations — four backbones from 0.8B to 27B, six budgets from 50 to 1,600 hours, five mixtures varying Minecraft’s share from 100% to zero, roughly 45K H200 GPU-hours — a composition-aware fit explains training loss well, reaching a within-mixture R² of 0.959 on the training-game distribution. On distant unseen games the fitted data exponent lands near zero and the within-mixture R² falls to 0.079: hours stop buying transfer, and coverage matters more than volume.

Caveats are the authors’ own: a selected set of environments, deliberation still a bottleneck (a System 2 planner over a System 1 controller is sketched), little gain for pre-trained checkpoints on the most distant benchmark, and one minigame where reinforcement learning hurts — Janus’s Maze falls from 52.4% to 23.8%. All baselines ran on the authors’ harness, in one game.

Release scope is the sharper limit. The GitHub repository is Apache-2.0 and includes the system prompt and action-format documentation, and the Hugging Face collection holds eight GameScaling checkpoints from 0.8B to 27B — the controlled scaling study published in October, not the Star Rail-tuned model. The project page states plainly that the Star Rail data and any checkpoint trained or fine-tuned on it will not be public “for data security reasons”. Streaming-vLLM appears in the paper’s own comparison table as open-source, yet still sits on the repository’s to-do list.

What This Means

Games are one of the few training environments that ship their own verifier: nothing in Gamer’s rewards comes from a model judging another model. Verified binary rewards, no privileged state and a universal keyboard-and-mouse action space are what make a commercial title usable as a reinforcement-learning gym, and they point at automated QA and regression testing before anything a player would notice.

The second lesson is that latency belongs to the capability. A policy planning over a screen it saw 1.3 seconds ago is solving a different, harder problem than one that acts within 65 ms, and the real-time spreads show how much of the baselines’ apparent game skill was an artefact of a frozen game. The third is that self-reported negatives are useful, and that independent verification is what remains missing while the weights behind the headline numbers stay inside miHoYo.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

Related stories

AI

Qwen3-VL: The Next Generation Multimodal LLM from Qwen / Alibaba Cloud

Introduction Qwen3-VL is the new flagship multimodal large language model (LLM) series developed by the Qwen team at Alibaba Cloud. (GitHub)It advances both text and vision capabilities, enabling richer understanding and generation across mixed modalities (images, video, text). Key points Qwen3-VL is the flagship multimodal large language model series from the Qwen team at Alibaba […]

agentic ai

Google Open Sources Harness2 for Self-Improving AI Agents

On October 9, 2026, researchers from Google Cloud AI Research and Northwestern University posted Harness²: Recursive Agent Harnessing for an Open World and released the accompanying code on GitHub under the Apache 2.0 licence. The framework improves an AI agent’s harness — the prompts, tools, memory, skills and orchestration wrapped around a model — while […]

Never miss a RITS update

Subscribe to the RSS feed for AI, emerging technology, and digital scholarship news curated for the NYU Shanghai community, or get in touch to suggest a story.