AI and Robot Learning
Vision Language Action Models: What They Actually Do
A VLA model takes a camera image and a text instruction and outputs robot actions. What that buys, what it costs in compute and data, and where it still fails.
A vision language action model takes an image and a natural language instruction and outputs robot actions directly, usually end effector deltas at 3 to 50 Hz. The architecture is a vision language model with an action head, and the appeal is generalisation: one model handles many tasks and objects instead of one policy per task. The cost is data, compute and a failure mode that is hard to predict.
How they are built
The recipe is consistent across published models. Take a vision language model pretrained on internet-scale image and text data, which already knows what a mug is and what "pick up" means. Attach an action representation, usually discretised tokens or a continuous head. Fine-tune on robot demonstration data where each step pairs an observation and an instruction with the action a human took.
The pretraining is the point. A policy trained only on robot data has never seen a mug it was not shown; a model built on a vision language backbone has seen millions of images of mugs and can carry that knowledge into a grasp it was never demonstrated.
| Choice | Options | Trade |
|---|---|---|
| Action representation | Discrete tokens or continuous head | Tokens reuse the language head; continuous is smoother |
| Action horizon | Single step or a chunk of 4 to 50 | Chunks give smoother motion and tolerate latency |
| Control frequency | 3 to 50 Hz | Higher rates need smaller models or chunking |
| Backbone size | 1 to 55 B parameters | Bigger generalises better and may not fit on the robot |
| Observation | One camera or several plus proprioception | More views help occlusion, cost bandwidth and compute |
| Output frame | Joint space or end effector space | End effector transfers across robots, joint space does not |
What genuinely works now
- Language-conditioned task selection. Telling a robot which of several known tasks to perform, in plain words, works reliably and removes a programming step.
- Object generalisation within a task. Picking an object type the model has seen in its pretraining but not in its robot demonstrations succeeds far more often than with a task-specific policy.
- Recovery behaviour, when the training data contained failures and recoveries rather than only clean successes.
- Cross-embodiment transfer when training data spans multiple robots and actions are expressed in end effector space.
Where they still fail
- Precision. Tasks needing better than roughly 1 mm placement remain hard, because the action representation and the visual feedback loop are both coarse relative to that.
- Long horizons. Multi-minute tasks with many dependent steps accumulate error, and there is no reliable mechanism for the model to notice it is off-plan.
- Force-critical work. Insertion, wiping and assembly depend on contact forces that a camera cannot observe. Models without force input guess.
- Predictable failure. A learned policy has an empirical rather than provable competence region, which is a genuine obstacle to any safety argument.
- Speed. Published demonstrations often run well below the speed a scripted robot would use for the same motion.
What this means for a factory today
For a high-volume cell doing one task on one part, a taught program remains faster, cheaper and analysable. The case for a learned policy strengthens as variety rises and volume per variant falls, which is exactly the region conventional automation has always struggled with: mixed depalletising, kitting from varied bins, handling of items that arrive in unpredictable poses.
The honest deployment pattern is hybrid. Use a learned policy for the perception-heavy, variable part of the task and scripted motion for the parts with tight tolerance, and keep a deterministic fallback for when the policy declines to act.
Frequently asked questions
What is a vision language action model?
A model that takes camera images and a natural language instruction and outputs robot actions directly, typically end effector deltas at 3 Hz to 50 Hz. It is a vision language model with an action head, fine-tuned on robot demonstrations.
Why use language at all?
Because the pretrained backbone carries knowledge about objects and verbs from internet-scale data, which lets the policy generalise to objects it never saw in robot demonstrations. Language also becomes the interface for selecting among tasks.
How fast can these models run?
Inference typically takes 100 ms to 300 ms, so models predict a chunk of 8 to 50 future actions that a lower-level controller executes. That is what makes a large model compatible with smooth motion.
What tasks do they still fail at?
Sub-millimetre precision, long multi-step tasks where error accumulates, and force-critical operations such as insertion where a camera cannot observe the relevant signal. Speed also remains below scripted motion for the same task.
Should a factory deploy one today?
Where part variety is high and volume per variant is low, yes, as part of a hybrid with scripted motion for tolerance-critical steps and a deterministic fallback. For a single high-volume task a taught program is still faster, cheaper and easier to certify.
Sources
- Open X-Embodiment: robotic learning datasets and RT-X modelsDataset scale, embodiment coverage and cross-embodiment results
- arXiv robotics preprints, vision language action modelsPrimary literature on architectures, action chunking and evaluation
- ROBOTS guideIEEE Spectrum, hardware context for deployed learned policies