Humanoids and Legged Robots
Teleoperation: How Humanoid Training Data Is Collected
Behind every humanoid demonstration is a human wearing a rig. Here are the capture methods, their data rates, their costs per hour and the retargeting problem nobody shows.
Almost all humanoid manipulation policies are trained on human demonstrations, captured by a person driving the robot or wearing a rig. A skilled operator produces roughly 60 to 200 usable episodes per hour, a policy for a single task typically needs hundreds to a few thousand, and the binding constraint is not the algorithm but the cost of an operator hour.
The capture methods
| Method | Operator wears or uses | Fidelity | Throughput | Weakness |
|---|---|---|---|---|
| Direct kinaesthetic teaching | Their hands, on the robot | High | low | Robot must be backdrivable and present |
| VR headset and controllers | Headset, hand controllers | Medium to high | medium | No force feedback, latency sensitivity |
| Exoskeleton rig | Arm and hand exoskeleton | High | medium | Expensive, fits few body sizes |
| Motion capture suit | Optical or inertial suit | Medium | high | Retargeting error, no contact forces |
| Handheld gripper recorder | A gripper on a stick with cameras | Medium | very high | No robot dynamics in the data |
| Video of humans working | Nothing | Low | enormous | No actions, only observations |
The bottom two rows are where the field is pushing hardest, because they decouple data collection from robot availability. A handheld recorder lets a warehouse worker generate demonstrations during their normal shift without a robot in the room, which changes the cost per episode by an order of magnitude.
What makes an episode useful
- Synchronised streams. Robot joint states, camera frames, gripper state and often force or tactile data, timestamped consistently. Drift of 50 ms between video and action is enough to degrade a learned policy noticeably.
- Consistent action space. Recording end effector poses is portable across robots; recording joint angles is not. Most large open datasets settle on end effector deltas for exactly this reason.
- Failure and recovery included. Datasets of only successful, clean demonstrations produce policies that cannot recover, because they have never seen the state that follows a mistake.
- Deliberate variation. Lighting, object pose, background and distractors have to vary, or the policy learns the room rather than the task.
The pipeline from rig to policy
Collection is one stage of five, and the later ones consume more engineering time than the recording itself.
| Stage | Work | Share of effort | Failure mode |
|---|---|---|---|
| 1 | Capture episodes with synchronised streams | 20 to 30 % | Timestamp drift above 50 ms |
| 2 | Filter and label, discard unusable takes | 10 to 20 % | Silent inclusion of failed episodes |
| 3 | Retarget to the robot embodiment | 15 to 25 % | Workspace and joint limit mismatch |
| 4 | Train and evaluate the policy | 20 to 30 % | Overfitting to the capture room |
| 5 | Deploy, monitor, collect corrections | 10 to 20 % | No mechanism to feed failures back |
Stage five is the one most often missing. A deployment that cannot record its own failures and feed them into the next training round improves only when someone schedules another collection campaign, which in practice means it does not improve at all.
Storage and bandwidth
Multi-camera capture at 30 Hz with three to six views produces roughly 1 GB to 20 GB per hour depending on resolution and compression. A campaign of 200 operator hours therefore lands between 200 GB and 4 TB before augmentation, and the cost of moving and versioning that data is a genuine line item rather than an afterthought.
The economics
Take an operator cost of 30 currency units per hour, a rig amortisation of 10, and 120 usable episodes per hour. That is roughly 0.33 per episode. A task needing 1,500 episodes therefore costs about 500 in direct capture, which sounds cheap until the number of tasks is counted. A hundred tasks is 50,000, and each new gripper, workcell or object family restarts a share of the work.
This arithmetic is why so much effort goes into three multipliers: simulation to expand a small real dataset, cross-embodiment datasets so data collected on one robot helps another, and passive capture from humans working normally rather than demonstrating deliberately.
Frequently asked questions
How is humanoid training data collected?
Overwhelmingly from human demonstrations: an operator drives the robot through VR controllers, an exoskeleton or direct guidance, while joint states, camera frames and gripper state are recorded together at 30 Hz to 100 Hz.
How many demonstrations does one task need?
Hundreds to a few thousand episodes for a single well-defined task, depending on variation. A skilled operator produces roughly 60 to 200 usable episodes an hour, so a task is typically a few operator days.
What is the retargeting problem?
Human and robot limbs differ in length, joint limits and workspace, so a recorded human motion has to be mapped onto the robot. The result is a trajectory the robot can execute rather than the one the human performed, and the error is largest where tolerances are tightest.
Why include failures in the dataset?
Because a policy trained only on clean successes has never observed the state that follows a mistake and therefore cannot recover from one. Recovery behaviour has to be demonstrated to be learned.
Can video of humans working replace teleoperation?
Not on its own, because video contains observations without actions. It is valuable for pretraining representations, but a policy that outputs robot actions still needs action-labelled data from teleoperation, handheld recorders or simulation.
Sources
- arXiv robotics preprints, imitation learning and robot datasetsPrimary literature on demonstration collection and retargeting
- Open X-Embodiment datasetCross-embodiment robot demonstration dataset and its action space conventions
- ROBOTS guide, humanoid profilesIEEE Spectrum, hardware context for teleoperation rigs