AI and Robot Learning
Open Robot Datasets Compared: Size, Embodiments, Licence
Which open robot learning datasets are worth using, what they actually contain, and the three properties that decide whether a dataset helps your robot at all.
Check three properties before anything else: the action space, the observation setup and the licence. A dataset recorded in joint space on a different arm cannot transfer to yours, a single-camera dataset will not train a policy that needs a wrist view, and a research-only licence blocks commercial use however good the data is. Size matters least of the four.
The three checks, in order
| Property | What to look for | Why it disqualifies |
|---|---|---|
| Action space | End effector deltas, ideally with gripper state | Joint angles are specific to one kinematic chain |
| Observation setup | Number of views, resolution, whether a wrist camera exists | A policy cannot learn from a view it never had |
| Licence | Commercial use, redistribution, attribution | Research-only data cannot ship in a product |
| Control rate | Stated, and consistent across episodes | Mixed rates need resampling and introduce error |
| Failure content | Whether failed and recovery episodes are included | Clean-only data produces policies that cannot recover |
| Metadata quality | Robot, gripper, operator, date, task label | Without it, diagnosing a performance drop is guesswork |
The landscape
Open robot data falls into four groups, and they serve different purposes.
- Pooled cross-embodiment collections. Dozens of laboratory datasets normalised into one format, spanning more than twenty robot types and over a million trajectories. These are what pretrained action models are built on, and their value is breadth rather than depth per task.
- Single-laboratory task collections. Hundreds to a few thousand episodes on one robot for a defined task set, usually with consistent setup and better metadata. Good for fine-tuning and for benchmarking.
- Low-cost hardware datasets. Collected on inexpensive bimanual platforms, which makes them reproducible by other groups. Valuable precisely because the hardware is obtainable.
- Simulation datasets. Effectively unlimited and free of collection cost, carrying the sim-to-real gap in exchange, especially on contact behaviour.
What the sizes actually mean
| Class | Episodes | Raw size | Embodiments | Training GPU days |
|---|---|---|---|---|
| Single task, single laboratory | 200 to 2,000 | 10 to 200 GB | 1 | 0.5 to 4 |
| Task suite, single laboratory | 2,000 to 20,000 | 200 GB to 2 TB | 1 to 3 | 4 to 30 |
| Low-cost bimanual platform | 1,000 to 10,000 | 50 GB to 1 TB | 1 to 2 | 2 to 20 |
| Pooled cross-embodiment | 500,000 to 1,000,000+ | 5 to 50 TB | 20 to 25 | 50 to 500 |
| Simulation generated | unbounded | generated on demand | any | 10 to 200 |
The storage column is a practical filter. A 30 TB download is not a casual experiment; it is an infrastructure decision involving bandwidth, storage cost and a data pipeline. Most teams should start from a published pretrained checkpoint rather than from the raw data that produced it.
Using a public dataset properly
| Use | Works | Typical benefit |
|---|---|---|
| Pretraining a policy backbone | Yes | 3 to 10x less task data needed |
| Benchmarking your own method | Yes | comparability, if the protocol matches |
| Learning your specific task directly | Rarely | near zero without fine-tuning |
| Replacing your own data collection | No | none, your gripper and cell are unique |
| Validating a perception model | Partly | depends on sensor similarity |
Row three and four are the expectation-setting rows. Public data pretrains, it does not deploy. The task-specific fine-tuning set still has to be collected on the actual robot with the actual gripper in the actual cell, and budgeting otherwise is the most common planning error in a robot learning project.
If you release data
- Record end effector poses alongside joint states, so the data is portable.
- State the control rate explicitly and keep it constant.
- Include failures and label them, rather than deleting them.
- Publish camera intrinsics, extrinsics and mounting geometry.
- Choose a licence that permits commercial use if you want the data to matter beyond academia.
Frequently asked questions
Which open robot dataset should I use?
For pretraining, a pooled cross-embodiment collection with end effector action space, currently over a million trajectories across more than twenty embodiments. For fine-tuning, your own data on your own robot.
What makes a dataset transferable?
An action space expressed as end effector deltas rather than joint angles, a consistent control rate, and observation setups similar to your own. Without those, the size of the dataset is irrelevant.
Can public data replace my own collection?
No. It pretrains a backbone and reduces task-specific data needs by roughly three to ten times, but the fine-tuning set must come from your robot, your gripper and your cell.
Do licences matter in practice?
Yes. Research-only terms prevent shipping a model trained on that data in a commercial product, and the constraint propagates to the trained weights. Check the licence before training, not after.
Should failures be included in released data?
Yes, and labelled. Policies trained only on clean successes cannot recover from mistakes, so failure and recovery episodes are among the most valuable content a dataset can carry.
Sources
- Open X-Embodiment: robotic learning datasets and RT-X modelsScale, embodiment coverage and action space conventions
- arXiv robotics preprints, robot learning datasetsPrimary literature on dataset construction and evaluation protocols
- MuJoCo documentationContext for simulation-generated datasets and their limitations