NVIDIA has released IsaacTeleop, a framework designed for robot teleoperation using hand and controller tracking data. The system utilizes a graph-based retargeting engine that processes inputs from devices like OpenXR and CloudXR to drive robot actions.
The core of the framework is the TensorGroup structure, which manages typed data slots for hand and controller inputs. It supports 26 OpenXR hand joints and 14 controller slots, including buttons, axes, and poses, ensuring strict type validation during data processing.
Developers can implement custom retargeters by subclassing BaseRetargeter. This allows for the creation of logic that processes tracking data, such as measuring thumb-to-index distance, with support for live-tunable parameters that update in real-time.
IsaacTeleop includes built-in retargeters for specific tasks, such as a gripper controller with hysteresis settings and SE(3) end-effector pose estimation. These tools allow for complex control schemes, including prioritizing controller inputs over hand tracking.
