Number 2 of 12 · November 17, 2026

Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Gamer is an SI agent trained on tens of thousands of hours of keyboard and mouse play. The authors say it plays 3D open-world games with efficiency comparable to skilled players.

What they found

The authors introduce Gamer, a model that sees the screen, keeps a memory, and reasons over many steps to play games. They first continue pre-training it on tens of thousands of hours of keyboard and mouse recordings from simulated games and embodied environments. Then they use sparse-thinking tuning, which cuts redundant computation while keeping the reasoning needed for hard decisions. They also use a multi-turn reinforcement learning method meant to handle limited context windows, accumulated errors and sparse rewards. The authors report solid generalization and efficiency comparable to skilled players across several challenging 3D open-world games. They also report competitive results on conventional 2D game benchmarks. The abstract gives no specific scores, so the size of these results is unclear.

Why it matters

Long tasks in interactive worlds are hard for SI systems. They must remember, plan and act in real time. The authors suggest that even repetitive keyboard and mouse data can give large gains when data and learning methods are scaled up. They also build a cloud-edge inference engine that runs many game environments at once with low latency. This kind of setup is needed to train and test agents at scale.

How it changes the race

If the results hold, this points to a data route for general computer and game agents: large logs of raw keyboard and mouse input plus reinforcement learning. It could help teams that have data and serving infrastructure, but the abstract does not compare Gamer with other named systems.

Safety and control

The abstract does not discuss safety. Agents that operate keyboard and mouse across digital and physical-style worlds could later be used on real software, which would call for careful testing and human oversight.

The abstract, by the authors

We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.

Quoted from arXiv:2611.18397. The research and its words belong to its authors.

The authors

Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.