It’s 2016, and you’re wandering around the neighborhood at dusk. On the prowl, you’re swiping on a smartphone to catch that elusive Snorlax and grow your collection in Pokémon Go. You, alongside hundreds of millions of other players, are enjoying a mixed-reality game built around lovable characters. What you don’t realize is that these scans of your environment will one day help robots navigate obstacles and interact with objects.
Niantic, the creator of Pokémon Go, sold the game last year and pivoted hard into one of Silicon Valley’s buzziest AI topics right now: world models. “We’re very much about reconstructing the real world,” says Brian McClendon, Niantic Spatial’s chief technology officer. He hopes to leverage its trove of world data to build replicas. Unlike the text-based nature of large language models, world models use visual inputs and can generate digital environments that simulate physics.
This software is poised to seep into everyday life, without you necessarily ever interacting directly with world models. 2027 could be a major year for the expansion of home robotics and driverless cars, both of which are bolstered by advances in world models. It’s time to better understand how these bots think.
“So with a language model, you have a sentence, and it will predict the next word in the sentence,” says Oliver Cameron, the CEO of Odyssey, an AI lab focused on world models. “With a world model, you give it a visual state of the world, an image, and it will then predict the next visual state of the world.” These 3D generations can be used as digital training grounds. “They’re simulations, but they look and feel real,” he says.
McClendon sees these world models as helping robots practice without physical movement or being in intended environments. He says someone could take photos of where the robot is intended to function, generate a 3D model, and then let the software run “millions of simulations” to train the robot on the space’s parameters.
It’s not just humanoid robots that could use simulated practice; driverless cars also improve using world models. Without a human behind the wheel, split-second software reactions are paramount.
“Importantly, a world model must be able to react to the actions that we take in the world, and so it must understand physics, cause and effect, and how other objects and people change their actions in response to our own,” says Jamie Shotton, chief scientist at Wayve, which builds world models designed for cars.