How video games could train smarter AI beyond text-only models

The next leap in artificial intelligence may not come from parsing more text, but from mastering motion. That’s the bold claim from General Intuition, a stealth AI startup whose CEO argues that video games could unlock the missing ingredient for more human-like cognitive abilities in machines.

At the heart of the debate is a fundamental limitation of today’s large language models: while they excel at generating coherent text, they struggle to translate that knowledge into real-world understanding. A model might describe how a ball bounces or a car accelerates, but it rarely grasps the underlying physics intuitively.

This gap between abstract knowledge and embodied cognition is exactly what General Intuition aims to address. The company, founded by former NVIDIA research scientist Alex Nichol, is building AI models trained primarily on video game environments rather than vast internet text corpora. The theory? Games provide structured, physics-based simulations where actions have immediate, predictable consequences—something the messy, unstructured internet can’t replicate.

Why text-only models hit a wall

Language models like those powering ChatGPT or Anthropic’s Claude demonstrate remarkable fluency, but they remain confined to text-based tasks. Their training data consists largely of books, articles, and web pages—sources that lack the temporal and spatial dynamics of physical reality. As Nichol explains, "Text describes a world that never moves. Games, by contrast, present a world where every pixel shift tells a story of cause and effect."

Researchers have long noted this blind spot. A 2023 Stanford study found that even advanced models like GPT-4 falter when asked to predict the outcome of simple physical interactions, such as stacking blocks or pouring liquids. The paper concluded that these models lack "intuitive physics"—an innate understanding of how objects behave in space.

Video games, especially those with realistic physics engines, could bridge this divide. Titles like Half-Life 2 or Portal simulate gravity, friction, and collisions with precision. For an AI, these environments offer a sandbox to learn not just language, but how the world works.

The gaming advantage: structured, scalable, and safe

General Intuition isn’t the first to explore gaming as an AI training ground. Projects like DeepMind’s StarCraft II experiments or NVIDIA’s Isaac Sim platform have demonstrated the potential of virtual worlds. But Nichol’s approach differs in scale and focus. Instead of targeting complex strategy games, the startup prioritizes environments designed to teach foundational skills—like navigating rooms, manipulating objects, or anticipating trajectories.

The benefits are threefold:

• Structured learning: Games impose rules that are consistent and repeatable, unlike the unpredictable noise of the internet. An AI can repeatedly attempt a task (e.g., opening a door) without encountering contradictory examples.
• Scalability: Modern games generate petabytes of synthetic data daily. Platforms like Unity or Unreal Engine allow developers to create thousands of unique scenarios in hours, far outpacing human-generated datasets.
• Safety: Training in simulations eliminates risks associated with real-world robotics. A model can crash a virtual car hundreds of times without physical consequences—a luxury not afforded in physical robotics.

Early benchmarks from General Intuition suggest their models outperform text-only counterparts in tasks requiring spatial reasoning. In a controlled test, an AI trained on game data was asked to predict the path of a rolling ball. While a text-trained model guessed randomly, the gaming-trained AI achieved over 90% accuracy.

Challenges and skepticism

Not everyone is convinced. Critics argue that even the most advanced games are mere approximations of reality. "Games are designed to be fun, not accurate," said one AI researcher from MIT. "The physics in Grand Theft Auto bear little resemblance to real-world aerodynamics."

Others question whether synthetic training can replace real-world data entirely. Robotics companies like Boston Dynamics rely on physical interaction to refine their machines, arguing that no simulation perfectly captures the unpredictability of the real world.

General Intuition acknowledges these limitations. The company’s roadmap includes hybrid training—combining gaming data with real-world observations to achieve robust generalization. Nichol emphasizes that the goal isn’t to replace text entirely, but to supplement it with the kind of grounded, experiential learning humans acquire through play.

What’s next for AI trained on games?

If successful, gaming-trained models could revolutionize fields beyond chatbots and robotics. Imagine AI assistants that understand spatial constraints (e.g., "fetch the mug from the top shelf without knocking anything over") or autonomous drones that navigate urban environments with human-like intuition. The implications stretch from healthcare—where surgical robots could train in virtual operating rooms—to education, where AIs might guide students through interactive, physics-based simulations.

General Intuition’s next milestone is a public demonstration of its first model, slated for late 2024. The company has already raised $20 million in seed funding, with investors betting on the idea that the future of AI isn’t just in reading the world, but experiencing it.