AI and Robot Learning
Robot Foundation Models: What Works Today
A single pretrained model that transfers across robots and tasks is real but partial. What generalises now, what does not, and how to read a demonstration video honestly.
Generalisation across objects works, generalisation across robots works with the right action representation, and generalisation across tasks is partial. A model pretrained on a large multi-robot dataset then fine-tuned on a few hundred task demonstrations reliably beats training from scratch, typically cutting task data by a factor of three to ten. Zero-shot performance on a genuinely new task remains poor.
Three axes of generalisation
| Axis | Status | Why |
|---|---|---|
| New object, known task | Works well | Vision backbone carries object knowledge from internet-scale pretraining |
| New scene or lighting | Works with randomisation | Visual variation is well covered by augmentation |
| New robot, known task | Works with end effector action space | Cross-embodiment data teaches a shared action representation |
| New task, similar skills | Needs fine-tuning | Hundreds of demonstrations rather than thousands |
| New task, new skill | Poor | The required behaviour is not in the training distribution |
| New force regime | Poor | Contact force is not observable in the training signal |
The pattern is consistent: what the backbone saw during pretraining generalises, what only the robot data could teach does not. That is why object generalisation is strong, since the internet is full of objects, and force behaviour is weak, since nobody publishes force traces at scale.
The numbers behind the claim
| Property | Typical value | Consequence |
|---|---|---|
| Backbone parameters | 1 to 55 billion | Larger generalises better, may not fit on the robot |
| Pretraining trajectories | 100,000 to 1,000,000+ | Pooled from dozens of laboratory datasets |
| Embodiments in pretraining | 10 to 25 | Enables cross-robot transfer |
| Distinct skills covered | 200 to 600 | Breadth, not depth, per skill |
| Fine-tuning demonstrations | 100 to 600 | Against 1,000 to 5,000 from scratch |
| Inference latency | 100 to 300 ms | Requires action chunking for smooth motion |
| Action output rate | 3 to 50 Hz | Below scripted motion control rates |
| Reported task success | 50 to 90 % | Varies enormously with evaluation protocol |
The success rate row deserves scepticism rather than dismissal. Reported figures between 50 % and 90 % are not comparable across papers, because the number of attempts, the reset protocol, the object set and whether a human intervened all vary. Comparing two published percentages is usually comparing two different experiments.
Does scaling solve it
Partly, and the constraint is different from language. Language models scaled because text was already on the internet in quantity. Robot action data is not, and it must be produced one episode at a time on physical hardware. The largest open collections gather roughly a million trajectories across more than twenty embodiments by pooling dozens of laboratory datasets, which is a substantial achievement and still tiny beside a text corpus.
Three routes are being pursued to break that constraint, and each has a visible weakness.
- Pool everyone's data. Effective, and it introduces heterogeneity in camera setup, action rate and quality that has to be normalised.
- Generate in simulation. Cheap and abundant, and it inherits the sim-to-real gap, particularly on contact.
- Learn from human video. Enormous supply, and it contains observations without actions, so it helps representations more than control.
What this means for a deployment decision
| Situation | Approach | Reason |
|---|---|---|
| One task, one part, high volume | Taught program | Faster, cheaper, analysable, certifiable |
| Many parts, one task, moderate volume | Pretrained policy, fine-tuned | Handles variation that fixturing cannot remove |
| Frequently changing tasks | Pretrained policy plus language interface | Task selection without reprogramming |
| Tight tolerance assembly | Scripted motion with force control | Learned policies remain coarse at this scale |
| Unstructured picking from mixed containers | Pretrained policy | The case the technology is genuinely best at |
The middle three rows describe the real commercial opening. Conventional automation has always been weak exactly where variation is high and volume per variant is low, and that is where a fine-tuned pretrained policy earns its cost today.
Frequently asked questions
Do robot foundation models generalise to new tasks?
Partially. Tasks composed of skills present in the training distribution can be reached with a few hundred fine-tuning demonstrations. Genuinely new skills, and anything depending on contact forces, still perform poorly zero-shot.
How much does pretraining help?
It typically cuts task-specific demonstration requirements by a factor of three to ten compared with training from scratch, which is what makes the second and third deployment substantially cheaper than the first.
Why is there so little robot data?
Because it has to be produced on physical hardware, one episode at a time. The largest open collections pool dozens of laboratory datasets into roughly a million trajectories across more than twenty embodiments, which is small beside any text corpus.
Can these models transfer between different robots?
Yes, when actions are expressed in end effector space rather than joint angles and the training data spans multiple embodiments. Joint-space policies do not transfer, because the kinematics differ.
How should I read a demonstration video?
Ask for the success rate across attempts, whether the object appeared in training, and how many operator resets occurred. A single successful take demonstrates capability once, not reliability.
Sources
- Open X-Embodiment: robotic learning datasets and RT-X modelsDataset scale, embodiment count and cross-embodiment transfer results
- arXiv robotics preprints, robot foundation modelsPrimary literature on pretraining and generalisation evaluation
- ROBOTS guideIEEE Spectrum, hardware context for deployed learned systems