Google-owned DeepMind and NVIDIA have released DreamX-Phi 1.0, a world model for robot control. They demonstrated its ability to predict in video the events that will occur after a robot performs an action.
DreamX-Phi 1.0 has been released as a world model for robot control. Based on images and language instructions, as well as information on a robot arm’s position and orientation and whether its gripper is open or closed, it generates videos predicting what will happen after the action is completed. This suggests that its applications could expand from video generation to testing the behavior of the real world in advance.
DreamX-Phi incorporates the robot’s three-dimensional movements into the AI’s internal representation and also includes training on depth information and the spatial relationships between objects. As of the fixed snapshot used for WorldArena 2.0, it ranked first in Track 1 for video prediction and tied for second in Track 2, which evaluates policies using world models.
Video-generation AI is expanding beyond AI that “creates works” to AI that “tests what will happen in the real world,” and is said to have the potential to become a new tool for robot development.
Some observers say that the implications of advances in world models for Japanese manufacturing companies will be significant, and practical developments are attracting attention.