AI and Robot Learning
Imitation Learning vs Reinforcement Learning for Robots
Imitation needs demonstrations and gets working behaviour in days. Reinforcement needs a reward and a simulator and can exceed human performance. Which to use, and why most projects use imitation.
Use imitation learning when a human can demonstrate the task, which is almost always in manipulation. Use reinforcement learning when demonstrations are impossible or when you need behaviour beyond human ability, which in practice means locomotion and dynamic control. Imitation needs hundreds to thousands of demonstrations; reinforcement needs millions to billions of simulated steps and a reward function that is far harder to write than it looks.
The practical difference
| Aspect | Imitation learning | Reinforcement learning |
|---|---|---|
| Needs | Demonstrations | Reward function and an environment |
| Data scale | 100s to 1,000s of episodes | millions to billions of steps |
| Where data comes from | Teleoperation on real hardware | Simulation, almost always |
| Time to first result | days | weeks |
| Performance ceiling | The demonstrator | Can exceed the demonstrator |
| Failure mode | Drifts out of the demonstrated distribution | Exploits the reward, ignores intent |
| Typical use | Manipulation | Locomotion, dynamic balance |
| Sim-to-real burden | Low, trained on real data | High, trained in simulation |
The bottom two rows explain the field's division of labour. Manipulation is contact-rich and hard to simulate accurately, but easy for a human to demonstrate, so imitation dominates. Locomotion is expensive and dangerous to demonstrate but simulates comparatively well and benefits from millions of falls, so reinforcement dominates.
Where imitation breaks
The core weakness is distribution shift. A policy trained on states a competent demonstrator visited has never seen the state that follows its own small mistake, and because a small error moves it slightly off-distribution, the next action is slightly worse, and the error compounds. This is why a policy can look excellent for three seconds and then diverge.
Three remedies are standard, and all three are about data rather than architecture.
- Demonstrate failures and recoveries deliberately. Take the robot off-course and demonstrate getting back. Policies cannot learn recovery they never saw.
- Collect corrections during deployment. An operator intervening produces exactly the data the policy most needs, and a pipeline that captures interventions improves faster than one that collects new campaigns.
- Chunk actions. Predicting a short sequence rather than a single step reduces compounding, because the policy commits to a coherent motion instead of reacting to its own noise.
What each approach costs to run
| Item | Imitation | Reinforcement in simulation |
|---|---|---|
| Human effort to produce data | 8 to 25 operator hours | 0, after the reward is written |
| Engineering effort before training | 1 to 3 days rig setup | 1 to 4 weeks environment and reward |
| Episodes or steps needed | 300 to 3,000 episodes | 10 million to 1 billion steps |
| Wall clock to first working policy | 2 to 10 days | 2 to 8 weeks |
| Compute for training | 1 to 4 GPU days | 5 to 200 GPU days |
| Parallel environments used | 1 | 1,000 to 8,000 |
| Hardware risk during training | low, teleoperated | none in simulation, high on hardware |
| Cost to add a second task | 60 to 100 % again | 40 to 90 % again |
The last row is the one that shapes programme plans. Neither method amortises well across tasks on its own, which is exactly the gap that cross-embodiment datasets and pretrained action models are trying to close: a shared backbone makes the second task cheaper than the first.
The combination that actually wins
- Pretrain on demonstrations to get a policy that is roughly correct, which removes the exploration problem entirely.
- Fine-tune with reinforcement against a reward that measures task success, in simulation or carefully on hardware.
- Use demonstrations as a constraint so the policy does not drift into a reward-exploiting behaviour that a human would never perform.
This ordering matters because random exploration in a high-dimensional action space almost never stumbles on a successful manipulation, so pure reinforcement from scratch on a real manipulator is impractical. Starting from demonstrations converts an intractable search into a local improvement problem.
Frequently asked questions
Which should I use for a manipulation task?
Imitation learning. Manipulation is easy to demonstrate and hard to simulate accurately, so a few hundred to a few thousand teleoperated demonstrations reach a working policy far faster than reinforcement from scratch.
Why is reinforcement learning used for walking?
Because demonstrating a fall is impractical and a simulator can produce millions of them safely. Locomotion also simulates more faithfully than contact-rich manipulation, so the transfer to hardware is more reliable.
What is distribution shift?
The compounding failure of an imitation policy that has only seen states a competent demonstrator visited. A small error moves the robot slightly off that distribution, the next action is slightly worse, and the deviation grows.
How much data does each approach need?
Imitation typically needs hundreds to thousands of episodes per task. Reinforcement needs millions to billions of environment steps, which is why it lives in simulation with thousands of parallel environments.
Can the two be combined?
Yes, and that is the strongest recipe. Pretrain on demonstrations to get roughly correct behaviour, then fine-tune with reinforcement against a success reward while constraining the policy to stay near demonstrated behaviour.
Sources
- arXiv machine learning preprints, imitation and reinforcement learningPrimary literature on distribution shift, action chunking and reward design
- Open X-Embodiment: robotic learning datasetsScale of demonstration data used for imitation across robot types
- MuJoCo documentationSimulation performance characteristics relevant to reinforcement learning throughput