Researchers at the Moscow Institute of Physics and Technology (MIPT) have developed a modular control system for humanoid warehouse robots. The technology, named SG-RAPL, enables robots to map their surroundings using dynamic 3D scene graphs and execute tasks based on natural language voice commands.
The system integrates large language models with 3D mapping to track objects like shelves, containers, and people in real time. By utilizing camera data, depth maps, and ArUco markers, the robot can decompose complex instructions, such as retrieving a box from a high shelf, into a specific sequence of physical actions.
The team created a specialized, manually labeled dataset to train and test the system. Benchmarking conducted in both virtual and physical environments demonstrated that the graph-based approach outperforms traditional systems that lack 3D spatial awareness.
The project was supported by the Russian Ministry of Economic Development. The source code and the dataset are now available to the public on GitHub, with future plans to adapt the technology for various industrial environments.
