Jim Fan, senior research manager at NVIDIA and head of the Generalist Embodied Agency Research (Gear) lab at Generalized AI Division, highlights how future artificial intelligence (AI) robots are trained and operated. Executives claim that AI agents that will soon be embodied will be born in simulations and they will learn and gain expertise on specific tasks. He also stressed that soon, the entire city, houses and factories will be shipped to the simulation, allowing AI to train in real-life situations. He also speculated that future AI robots can share the ideas of the hive.
Jim Fan of NVIDIA shares how to train AI robots in the future
Fan shared his thoughts in an article on X (formerly known as Twitter) as he stressed that the city of Tokyo released 3D digital twins across the city with high resolution point clouds and is available for free downloads. . NVIDIA directors shared the development, saying that this trend of digital simulation of the real world will only grow in the future.
“It’s an inevitable trend, with more and more cities, houses and factories going to be shipped to the simulation,” Fan said.
Fan shared the examples, adding that in the future, robots will not be trained in isolation. Currently, robots are trained in controlled environments where they can learn how to move around, identify objects, pick up and discard things, and complete tasks. However, many claim that this training method is not prepared for the various challenges that may be faced in real-life situations.
NVIDIA research managers have provided solutions to the problem, claiming that the embodied AI agents will soon be simulated as Iron Fleet. It is worth noting that the embodied AI agent is an AI system integrated with the body or simulation embodiment, allowing them to interact with and perceive the world like humans or animals.
Fan said that future AI robots will be deployed in a real-time graphics engine and will be scaled in a huge cluster. He added that this training approach will also generate trillions of high-quality training tokens. “Most of the manifested agents will be born in SIM and transfer zero shots to our real world when they are ready,” he said.
Once these AI robots are deployed into the real world, such as assistants in factories or offices and houses, they will also share the idea of the hive. This hive idea can enable embodied agents to learn from thousands of use cases and coordinate difficult tasks with multi-institutional efforts.
Although this seems to be science fiction, FAN, who leads the AI department embodied by NVIDIA, is confident that this will be the direction of training efficient robots in the near future. It seems that Nvidia has moved in that direction. He shared that the company’s Santa Clara headquarters building is “designed and rendered in Omniverse (in the Omniverse of the GPU-accelerated graphics platform) before atoms can be realized.”
