Inside NVIDIA’s IsaacTeleop: From Hand and Controller Tracking to Robot Actions with the Graph-Based Retargeting Engine

In this tutorial, we work through the retargeting engine at the core of NVIDIA IsaacTeleop, the framework that turns XR hand tracking and motion-controller input into commands for simulated and real robots. Rather than plugging in a headset, we build every input ourselves in NumPy, so each step runs on a plain Colab CPU and prints what it computes. We start with the type system every node speaks, generate synthetic hand and controller data, write our own retargeter with live-tunable parameters, and then drive the built-in gripper and SE(3) retargeters with it. From there, we compose a full graph that emits one action vector per step, apply a world-frame transform, step through the run, pause and kill the state machine, and finish with a controller-to-dexterous-hand mapping and parameter tuning that persists across restarts. Copy CodeCopiedUse a different Browser We install the stable isaacteleop wheel from PyPI with the retargeters-lite extra, which adds only SciPy, and report the version, interpreter, and NumPy build. The package splits into device I/O modules that wrap OpenXR and CloudXR, a schema module holding the FlatBuffer message types every tracker emits, and the pure-Python retargeting engine we work with here. Listing the schema types shows the vocabulary of the data layer, but nothing below opens a headset session; from here on, every input is a tensor we build by hand. Copy CodeCopiedUse a different Browser The engines contract is a TensorGroupType, an ordered list of typed slots, and a TensorGroup, the runtime container that holds one value per slot and validates each write. HandInput carries four NumPy arrays for the 26 OpenXR hand joints, and ControllerInput carries fourteen slots for poses, buttons, and axes, addressed through generated IntEnum indices rather than magic numbers. Writing a float64 array where float32 is declared fails at the write, and reading a slot nobody wrote raises instead of returning stale data. OptionalType marks inputs a tracker may not deliver; the matching OptionalTensorGroup starts absent and becomes present on its first write, which is how every downstream node learns that a hand has left the tracking volume. Copy CodeCopiedUse a different Browser We build the tracking data that a headset would normally supply. make_hand lays out 26 joints in OpenXR order around a wrist position, runs four finger chains and a thumb chain away from the palm, and places the thumb tip a chosen pinch distance from the index tip, so the one number the later steps depend on is under our control. make_controller fills every ControllerInput slot: grip and aim poses with validity flags, four buttons, the thumbstick axes, and the analog squeeze and trigger values. Both return ordinary TensorGroups, which is all a retargeter ever sees, whether the numbers came from OpenXR or from NumPy. Copy CodeCopiedUse a different Browser A retargeter is a BaseRetargeter subclass that declares input_spec and output_spec and implements _compute_fn; the framework fills missing optional inputs with absent groups, validates types, syncs parameters, and only then calls our code. PinchRetargeter measures the thumb-to-index distance and emits a float and a bool, reporting -1 when no hand is tracked instead of guessing. Its two parameters live in a ParameterState whose sync functions write onto the instance before every compute, so a value set from another thread by the tuning UI takes effect on the next frame. We reproduce that by calling set directly: the same 2.8 cm pinch flips between not pinching and pinching as the threshold moves, and switching the measurement to the distal joints changes the distance itself. Copy CodeCopiedUse a different Browser We drive two of the built-in retargeters with the same synthetic groups. GripperRetargeter turns pinch distance into the -1/+1 gripper command Isaac Lab expects, with hysteresis: the gripper closes below 3 cm, opens above 5 c,m and holds its state in between, so 4.0 cm stays closed after a close and the sweep back through 5.2 cm reopens it. A present controller takes priority, and a trigger above the threshold closes an open hand. Se3AbsRetargeter maps the controller grip pose to a 7-D end-effector target, applying the configured roll offset and keeping only yaw when zero_out_xy_rotation is set. It holds the last pose when the grip becomes invalid rather than passing a zero quaternion downstream. Se3RelRetargeter emits deltas instead: a constant 2 cm step per frame appears scaled by ten and smoothed by the EMA, converging toward 0.2. Copy CodeCopiedUse a different Browser This is the shape of a real Isaac Teleop pipeline. ValueInput nodes stand in for the DeviceIO source nodes as graph leaves, connect wires each retargeters inputs to upstream outputs with type checking at connect time, TensorReorderer flattens the 7-D pose and the gripper scalar into the action layout an environment expects, and OutputCombiner exposes the result under a single action key. execute_pipeline takes inputs keyed by leaf name and runs the DAG once per step against a shared ExecutionCache, so the controller leaf, although wired into both the SE(3) and the gripper retargeter, computes exactly once per frame, which our counting subclass confirms. Swapping the leaves for HandsSource and ControllersSource is the only change needed to run this graph against a headset. Copy CodeCopiedUse a different Browser Headsets report poses in the runtimes anchor frame and the robot lives in the world frame, so every pipeline that talks to a simulator applies a world_T_anchor transform first. ControllerTransform takes the two optional controller groups plus a TransformMatrix group holding a 44 homogeneous matrix, rewrites the grip and aim positions as R p + t and the orientations by the rotation part, and passes buttons, axes and validity flags through untouched. We verify the position against the same matrix product in NumPy and see the absent left controller propagate as absent, which is what lets a single-controller session flow through a two-controller graph. Copy CodeCopiedUse a different Browser Teleoperation needs a way to arm, pause, and kill the robot that doesnt depend on the retargeters behaving. DefaultTeleopStateManager is itself a retargeter: three optional bool inputs go in, a one-hot teleop_state and a reset_event pulse come out. A rising edge on the run toggle walks STOPPED to PAUSED to RUNNING and back to PAUSED, the reset button emits a one-frame pulse without changing state, kill forces STOPPED and pulses reset, and losing the kill or run signal fails safe to STOPPED. Those events reach every other node through ComputeContext.execution_events; LocomotionRootCmdRetargeter integrates the right thumbstick into a hip height each frame and, on the frame carrying reset=True, snaps back to its initial height before integrating again. Copy CodeCopiedUse a different Browser Two production concerns close the tutorial. TriHandMotionControllerRetargeter is the controller-only route to a dexterous hand: trigger drives the index finger, squeeze drives the middle finger, the larger of the two curls the thumb and their difference rotates it, giving seven named joint angles with no hand tracking and no optimization library. Then we make tuning stick: setting the rotation and position offsets on Se3AbsRetargeters ParameterState and calling save_to_file writes a JSON file, and a fresh instance constructed with the same parameter_config_path loads it on startup and produces an identical end-effector pose, which is how a calibration tuned in the UI survives a restart of the session. Copy CodeCopiedUse a different Browser The summary collects the headline of every section from the RESULTS dictionary that the section decorator filled in, so a skipped or failed step shows up here with its error instead of silently disappearing from the run. In conclusion, we built a working Isaac Teleop pipeline without any of the hardware the framework is designed