Large language models have benefited from enormous collections of existing internet content. Robotics faces a different challenge: much of the information a machine needs to learn physical actions and their consequences has never been recorded.
A useful robot demonstration captures more than what an action looks like. Depending on the equipment, it can connect visual observations with joint positions, control commands, and force or other sensor readings throughout an interaction. These records help a model learn the relationship between what a robot observes, what it does, and what happens next. Producing them requires equipment, human effort, and carefully managed environments.
Recently, while learning how the robotics industry collects training data, I kept returning to a question about life sciences: how much of the information AI needs to understand biology has yet to be generated?
Four ways to teach a robot
Four sources offer a useful way to think about robot training data: real-robot demonstrations, simulation, human motion capture, and internet video. They can be pictured as a pyramid, with expensive, directly actionable records near the top and abundant observational material near the base. This is an illustration of tradeoffs, rather than a fixed ranking: the value of each source depends on the task and the information it contains.
1. Real-robot demonstrations: learning through direct interaction
In teleoperation, a human controls a robot while its actions and available sensor readings are recorded. This provides direct examples of how a particular machine interacts with its surroundings.
It is skilled, demanding work. Operators need coordination, spatial awareness, and patience, particularly when the robot’s structure differs from their own body. Preparing a scene, resetting objects, checking recordings, and recovering from unsuccessful attempts all add to the work of collection.
Scaling this approach therefore means building the capacity to collect consistent, useful demonstrations. Shared datasets offer another way to extend their value. The Open X-Embodiment project brought together demonstrations from different robots and institutions, showing that training across these sources could improve performance on multiple robotic platforms.[1]
2. Simulation: creating experience in a virtual world
Simulation allows virtual robots to practice in parallel, repeat tasks, and encounter varied conditions without a person resetting physical equipment after every attempt.
The difficulty is transferring what they learn to the real world. Contact, friction, soft materials, and liquids can make this especially challenging. Reproducing the path of a robotic arm is a different problem from reproducing all the interactions involved in folding a sweater.
One approach is to vary simulated conditions so that a model encounters a wider range of possibilities during training. Research on domain randomization demonstrated this principle for transferring visual perception from simulated images to real-world robotic applications.[2] It does not eliminate every transfer problem, but illustrates why diversity within a simulation can matter alongside realism.
3. Human motion capture: adapting movement to a different body
Human motion capture records how people move and provides information that can be mapped onto a robot. The challenge is that human and robotic bodies differ in joint structure, reach, dexterity, and sensory feedback.
A robot might reproduce the outline of a movement without achieving its purpose. Human hands continually adjust their grip through touch; a movement recording may not capture the feedback behind those adjustments.
The boundaries between data sources are also becoming less distinct. EgoMimic, for example, combines first-person human video and three-dimensional hand tracking with robot demonstrations, using alignment methods and a shared learning approach.[3] The relevant question is how to preserve useful information about an action while adapting it to the machine that must perform it.
4. Internet video: observing actions at scale
Internet video provides abundant examples of objects, activities, and task sequences. But conventional video generally lacks the robot-specific commands and force measurements that would directly connect an observation to physical control.
Watching a table tennis match can reveal the sequence of play. It does not supply the coordinated movements and continuous adjustments needed to return a serve.
Instrumented first-person recordings can add information that ordinary video lacks, as approaches such as EgoMimic illustrate.[3] The distinction is therefore not simply between video and robot data. It is between what was observed, what was measured, and what must still be inferred.
Each source contributes something different. The challenge is to combine them in ways that improve reliable performance in the intended setting.
Can more data produce more capable robots?
A promising idea is a data flywheel: deployed robots generate experience, that experience supports learning, and improved performance enables wider deployment.
For me, the important question is what makes that cycle informative. Much of the data could document situations a robot already handles well. Failures and unfamiliar conditions may reveal more about its limitations, but only if the records contain enough context to distinguish a perception error, an unsuitable action, a hardware constraint, or an unexpected environment.
This leads to the question of scaling. Language-model research has documented predictable relationships between training loss, model size, data, and compute within studied conditions.[4] That evidence does not establish a universal rule that more robot data will produce proportionate gains in every kind of physical capability.
It also helps to distinguish object generalization from task generalization. Handling a new cup or working in a different room tests transfer across objects or settings. Accomplishing an unfamiliar task can require a different kind of adaptation. Better performance on the first does not, by itself, demonstrate the second.
The question I would ask of a growing robotics dataset is therefore: what new capability does each additional category of experience help the robot acquire?
What might life sciences learn from robotics?
Beyond the pace of progress in robotics, I was struck by the effort required to create information that did not previously exist.
Our biological and clinical data reflect the experiments we have conducted, the patients we have observed, and the outcomes we have chosen to measure. They also reflect the questions, methods, and practical constraints that shaped their collection. Some of the evidence needed for the next advance may be absent because the relevant intervention or measurement was never made.
In a November 2025 Cheeky Pint interview, Eli Lilly CEO Dave Ricks argued that AI needs a more complete repository of biological knowledge to tackle difficult problems. He estimated that we understand only 10–15% of human biology and suggested that building the necessary datasets could require “robotic 24/7 experiments just to create training data sets.”[5] The percentage expresses his assessment of the knowledge gap, rather than an established measure of how much biology we understand.
The connection to robotics is compelling. Both fields can benefit from data that connect an intervention to its consequences, with enough context to examine when and why an outcome occurs. Both face a transfer problem: performance in a simulation or controlled experimental setting may not carry over to the conditions that ultimately matter.
The analogy has limits. Biological responses can vary across experimental systems and individuals, and their consequences may unfold over much longer periods than a robot’s interaction with an object. More experiments alone would not resolve those differences.
That is why I see the potential lesson as a question about the design of learning itself. Which experiments would be most informative? How would we distinguish an uninformative result from a meaningful negative finding or a technical failure? What conditions, measurements, and provenance would need to accompany each result? And how would we test whether an updated model predicts something beyond the conditions used to train it?
For those of us working in RWE and HEOR, this also invites a closer look at the relationship between existing records and new evidence generation. Existing data can help identify gaps and shape the next question. Deliberately collected evidence can then test assumptions that those records cannot resolve. Neither source automatically establishes what will happen in a different patient population or care setting.
I do not yet have an answer. But robotics leaves me with one question for life sciences:
What would it take to build a biological data flywheel in which AI helps identify the most informative experiments, those experiments generate trustworthy new evidence, and that evidence improves our ability to predict biology beyond the conditions we have already studied?
References
- Open X-Embodiment Collaboration. Open X-Embodiment: robotic learning datasets and RT-X models. arXiv [preprint]. 2023. doi: 10.48550/arXiv.2310.08864.
- Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. arXiv [preprint]. 2017. doi: 10.48550/arXiv.1703.06907.
- Kareer S, Patel D, Punamiya R, Mathur P, Cheng S, Wang C, et al. EgoMimic: scaling imitation learning via egocentric video. arXiv [preprint]. 2024. doi: 10.48550/arXiv.2410.24221.
- Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, et al. Scaling laws for neural language models. arXiv [preprint]. 2020. doi: 10.48550/arXiv.2001.08361.
- Cheeky Pint. Dave Ricks, CEO of Eli Lilly, on GLP-1s and the business of pharma [interview and transcript]. 2025 Nov 11 [cited 2026 Sep 6]. Available from: Interview transcript.
Other Posts You Might Like
