AI and Robot Learning
How Much Data Does a Robot Policy Need?
Roughly 100 demonstrations for a fixed, single-object task and several thousand for a policy that generalises. The scaling depends on variation, not on task difficulty.
For a fixed task with one object in one place, 50 to 150 demonstrations often produce a working policy. Add object variety, pose variation, lighting change and clutter, and the requirement rises to 1,000 to 5,000. The driver is variation, not difficulty: a hard task with no variation needs less data than an easy task with a lot of it.
What drives the number
| Scope | Episodes | Operator hours | Example |
|---|---|---|---|
| One object, one pose, one background | 50 to 150 | 0.5 to 2 | Pick a fixture-located part and place it |
| One object, varied pose | 150 to 400 | 1 to 5 | Pick a part from anywhere on a tray |
| Object family, varied pose | 400 to 1,200 | 3 to 15 | Pick any of 12 similar parts |
| Open object set, cluttered scene | 1,500 to 5,000 | 10 to 60 | Pick any item from a mixed tote |
| Multi-step task with dependencies | 2,000 to 10,000 | 15 to 120 | Unpack, inspect, sort, pack |
| New embodiment, existing task | 100 to 600 | 1 to 8 | Same task on a different robot, with pretraining |
The last row is the reason cross-embodiment pretraining matters commercially. A policy fine-tuned from a model trained across many robots and tasks needs a fraction of the data that training from scratch would require, which changes the cost of adding the second and third deployment.
What the dataset should contain
- Deliberate variation across every dimension the deployment will see: object pose, lighting, background, distractor objects, and the operator performing the demonstration.
- Failures and recoveries, roughly 10 % to 30 % of episodes. A policy that has never seen a missed grasp cannot recover from one.
- Consistent action representation. End effector deltas transfer across robots; joint angles do not.
- Synchronised streams with timestamp drift under about 50 ms between video and action.
- Honest metadata. Which robot, which gripper, which operator, which day. Without it, diagnosing a performance drop is guesswork.
Storage, versioning and the cost of a campaign
Data collection is a capital project in miniature, and the running costs surprise teams that budgeted only for operator time.
| Item | Small campaign, 300 episodes | Large campaign, 3,000 episodes |
|---|---|---|
| Operator hours at 120 episodes per hour | 2.5 h | 25 h |
| Rig setup and teardown | 4 to 8 h | 8 to 16 h |
| Filtering and labelling | 2 to 6 h | 20 to 60 h |
| Raw data volume, 4 cameras at 30 Hz | 15 to 60 GB | 150 to 600 GB |
| Storage and versioning per year | minor | a real line item |
| Total engineering days | 1 to 2 | 7 to 13 |
Filtering is the item that scales worst. At 3,000 episodes, deciding which takes are usable is a larger job than recording them, and teams without a fast review tool end up shipping unusable episodes into training, where they cost far more than they saved.
Three ways to need less
- Pretraining. Starting from a model trained on a large multi-robot dataset can cut task-specific requirements by a factor of three to ten.
- Simulation augmentation. Using a small real dataset to calibrate a simulator, then generating variation there. Effective for pose and lighting variation, less so for contact behaviour.
- Reducing the variation instead. A fixture that presents parts consistently converts a 3,000-episode problem into a 150-episode one, and the fixture usually costs less than the data collection.
The third option is routinely overlooked by teams whose instinct is to solve the problem in software. Mechanical part presentation is the oldest trick in automation and it remains the cheapest way to remove variation from a learning problem.
Frequently asked questions
How many demonstrations does a robot policy need?
Roughly 50 to 150 for a fixed task with one object in one place, and 1,000 to 5,000 for a policy that must handle varied objects, poses and clutter. The requirement scales with variation rather than with task difficulty.
How long does that take to collect?
A skilled operator produces 60 to 200 usable episodes per hour, so a fixed task is under two hours and a generalising policy is 10 to 60 operator hours, plus filtering and labelling.
Should the dataset include failures?
Yes, roughly 10 % to 30 % of episodes. A policy trained only on clean successes has never observed the state that follows a mistake and therefore cannot recover from one.
Does pretraining reduce the requirement?
Substantially. Fine-tuning from a model trained across many robots and tasks typically cuts task-specific data needs by a factor of three to ten, which is what makes the second and third deployment cheaper than the first.
Is there a cheaper alternative to more data?
Removing the variation. A fixture that presents parts consistently can turn a 3,000-episode problem into a 150-episode one, and the fixture usually costs less than the collection campaign it replaces.
Sources
- Open X-Embodiment: robotic learning datasets and RT-X modelsCross-embodiment dataset scale and transfer results
- arXiv robotics preprints, imitation learning data scalingPrimary literature on demonstration counts and dataset composition
- arXiv machine learning preprints, data scalingScaling behaviour relevant to policy pretraining and fine-tuning