MotionOrigin Research

MotionOrigin Research · November 19, 2026

Expert-paced egocentric demonstrations for dexterous robot learning

A tomato-sorting study on generating expert-paced robot-view demonstrations without building a full physics sandbox for each task.

The concept robot's hands turn a blemished tomato over the moving conveyor.
Robot’s head cameraGenerated demonstration
Abstract

Can generated first-person video teach expert pace?

A robot can learn visual cues from a recorded demonstration. The environment that produced the frames does not have to run inside the training loop. We propose a vision-language model that generates expert-paced first-person episodes as files, with compatible robot actions supplied on the same clock. In bimanual tomato sorting, an episode must show support, regrasp, inspection and release while the belt moves.

We audited four short synthetic human episodes, then built a three-camera robot-view capture and a reduced simulator with learned motion and visual classification. We compared native timing, uniform 2× timing, phase-aware timing, and revisions for moving-grasp tracking and inspection. All three studies were rerun from source, covering 13,320 policy–case records. In the matched motion study, tracking reduced missed grasps from 151 to 3 across 360 shifted cases. In the final independent inspection comparison, the system completed 351 / 360 expanded-stress sorts correctly by the 7.55 s model deadline and correctly routed 174 / 180 blemished fruit. It ran at a median 6.97 s among successful trials in that condition.

We also authored and rendered synchronized wrist and overhead episodes for tomato sorting and sample-pod packing. Both play as SVGs and export as timestamped camera frames with authored pose targets. The detailed renders and the numerical simulator serve different purposes. The renders demonstrate how to produce the visual input, and the numerical study tests timing and feedback in a reduced model. Physical transfer and maximum repeatable expert pace remain hardware questions.

Findings at a glance

The robot had to move with the tomato

Phase timing helped the controller work faster. Under greater delay and variation, the arm still arrived too late to grasp a fruit moving away from its planned target. The strongest intervention was to update the target from the measured belt and move the hand with the fruit until pickup. The next improvement came from giving the detector enough pixels to see the blemish.

351 / 360correct and on time in the final expanded simulation holdout
174 / 180blemished tomatoes correctly rejected end to end in those trials
6.97 smedian successful sort against the 7.55 s model deadline

The visual dataset did not require a full physics sandbox. We authored the robot’s motion, rendered its cameras and saved the resulting episodes as reusable files. A separate reduced model tested whether timing and feedback held up when conditions changed.

MotionOrigin’s reported physical result

The team reports that a trained robot completed the tomato task correctly at the fastest expert pace used as its reference. This motivated the study. The physical trial count, checkpoint, timing and unedited hardware footage need verification before the reported outcome can be compared with the calculations here.

Methods · Source observations

Which human movements are worth teaching?

The prior manuscript inspected 24 frame samples across four approximately 20.042-second clips at 24 fps and 854 × 480 pixels. The source labels record visible events and flag occlusion. We use these clips to decide what an expert demonstration should show and which details need additional sensing or physical verification.

What we observed in the four source episodes
EpisodeVisible skill and measurement boundary
WP18 · Bottle packingThe left hand braces the case while the right lowers and releases three bottles into different cells. The case hides side clearance and parts of the landing. The fixture makes this task a practical first physical comparison.
WP17 · Produce replenishmentThe right hand places three sealed packs and six remain in two rows. A long supported hold precedes the third lift. The fingertips conceal part of the pack edge during lowering.
WP23 · Tomato inspectionThe left palm supports the fruit while the right thumb and fingers turn and regrasp it. The final split is two accepted, one rejected. The camera establishes a visible turn and sort, with parts of the hand and tomato occluded.
WP25 · Seedling insertionThe left hand braces the pot and the right guides the soil plug. An earlier clip showed ambiguous changes at the soil boundary. The later review found no obvious recurrence in inspected windows. Depth, compression and contact force remain unmeasured.
Observable timing changes at uniform 2×
Source eventNative frames1× duration2× duration
WP18 · First bottle settles121–134583 ms292 ms
WP18 · First visible release135–140250 ms125 ms
WP17 · Supported hold249–3283.333 s1.667 s
WP23 · First support handoff93–95125 ms62.5 ms

A 20.042-second clip plays in about 10.021 seconds at 2×. The WP17 hold remains long enough to see. The WP23 handoff lasts only 1.5 frame intervals in a 24 Hz retimed export. To keep that evidence inspectable, we preserve native frames, immutable source timestamps and a separate monotone map to the candidate training clock. Versions of the same clip and near-duplicate repeats stay in the same dataset split. Several clips start with the demonstrator already near the object, which is why the proposed benchmark times an agreed pre-grasp start on the real workcell.

Methods · Human-view reference

Follow one blemished tomato

We generated a first-person clip of a person sorting tomatoes. Follow the blemished one. The left palm supports it. The right thumb and fingers roll it to inspect the skin. Then the right hand lowers it into the yellow tray and opens after the tomato is resting.

We use the visible sequence to label support, turning and release, with separate hand roles. The brief changes in contact are the cues a generated example must preserve. Use the wrist views below to see what these cues look like after retargeting.

The left palm cradles the tomato while the right thumb braces its lower edge.
8.50 s Support and roll
The right fingertips shift around the blemish while the left palm supports the tomato.
9.00 s Change finger contact
The right hand lowers the tomato into the yellow tray.
12.08 s Lower into the tray
The right hand starts opening as the tomato rests in the tray.
12.33 s Open after placement

Frames from MotionOrigin’s generated first-person tomato clip. Times refer to the original clip.

Methods · Offline demonstration generation

Generate the robot’s view without building a full simulator

For tomato sorting, the lesson can be a short first-person episode. Show the left hand supporting the tomato, the right fingertips rotating it to reveal the blemish, and the fingers opening after the tray takes the fruit. Encode the intended expert timing in the episode. A visual model can use those frames to learn what each phase looks like and which cue should come next.

What we built. We set the arm, fingers and object to authored poses, rendered the cameras along that motion, and saved the episode for offline playback. The tomato reconstruction borrows its phase clock from a successful reduced-model trial. The detailed hand and object motion is authored. The pod-packing episode uses the same camera layout with an authored clock. We verified that all 2,196 camera images embedded in the two SVGs match the image files in the downloadable archive byte for byte. The archive adds 730 timestamped next-pose records. A model can read the individual frames and their timestamps directly. Export audit · Play the two examples

How a saved episode becomes supervision. Frames alone can train visual predictions, phase recognition or goal conditioning. To train the robot to issue commands, pair frames with targets exported from a checked robot trajectory, or estimate the commands with a model calibrated on real robot recordings. Keep those sources in the dataset. An SVG is convenient to inspect in the browser. A frame sequence or video with a separate timestamped record is convenient for training.

Where the physics still matters. A saved episode gives one history. If the tomato starts to slip earlier, the next frame still shows the original grasp. An interactive simulator can calculate a different outcome when the policy changes its action. In the offline approach, train with explicit recovery examples and use live sensing to decide when to hold, retry or release. Verify on hardware that the tray supports the tomato before the hand opens.

Evidence that the approach can transfer

DreamGen generated new robot demonstrations from images and instructions, attached estimated actions, and used those records in policy training. Its humanoid experiments reported 22 new behaviors with real pick-and-place teleoperation used to fit the video and action models. RoboTryOn translated human videos into robot-view clips and labeled them with an action model trained on robot demonstrations. Its synthetic-only policy fine-tuning condition reached 48.5% mean success over seven held-out OpenArm tasks, compared with 3.1% for its teleoperation-only baseline. Both studies establish that generated episodes can contribute to physical control when paired with a way to obtain robot actions. DreamGen · RoboTryOn, §4 and Table 1

The MotionOrigin hypothesis is that for tasks with clear visual cues and a reliable way to check actions, this route can lower the effort to create useful data for a new workstation while preserving quality and expert pace. The deciding comparison is time and cost to reach the same physical performance with generated episodes, interactive simulation or real demonstrations. Read the proposed comparison

Figure 2 · Articulated task reconstruction

Retarget the grasp and release to a robot hand

The saved animation recreates the demonstration with the robot’s own hands, camera and work area. From the head camera, watch the fingers roll the tomato and find a new grip. Good tomatoes go back on the belt and blemished ones go into the yellow tray.

Play the recording to see the head camera and the factory floor. See the synchronized wrist and overhead views of the autonomous run.

Authored simulation with head-camera and factory-floor views.

Watch the fingers at 8.90 s. The left palm keeps the tomato steady as the right index lifts and finds a new hold.

Watch the release at 12.08 s. The fruit touches the tray first. Then the thumb lets go, the fingers open, and the hand lifts away.

Figure 3 · Authored reconstruction of an autonomous rollout

What the robot sees while it sorts on its own

Watch the left wrist support the tomato, the right wrist take it to the yellow tray, and the fixed overhead view follow the whole work area. All three views show the same moments. There is no operator in the scene.

The 7.03-second run uses phase timing from a successful held-out trial of the trained reduced model. The detailed arm and finger motion is an authored reconstruction on the MotionOrigin rig. Expert pace on a physical workcell still needs a measured human benchmark.

4K UHD SVG Left wrist · Right wrist · Fixed overhead
Two wrist-camera close-ups of both hands rolling the tomato, with the synchronized overhead image below.
0.00 / 7.06 s
353 synchronized samples at 50 Hz. Each view records 640 × 360 pixels. The SVG arranges them on a 4K canvas for inspection.

Why keep the overhead view? The wrist cameras move with the hand and can lose the target when it opens or turns away. During placement and release, the right wrist saw the tomato in 83 of 83 frames and the left in 12. At least one camera saw it in 353 of 353 frames. We counted a view when at least three of five rays to the modeled object were unobstructed.

Why share one clock? We measured how much the wrist positions would disagree if a camera frame were paired with a pose 100 ms later. The 95th-percentile difference was 49.64 mm in this reconstruction. Writing images and pose targets from the same sampled trajectory keeps the references aligned.

The same camera setup on another task

We kept the two wrist mounts, overhead view and data format. Then we replaced the moving belt and tomato with a stationary pick surface, a capped sample pod and a marked packing cell. Watch the cap turn during the lift and the right hand release the pod upright in the case.

At least one camera saw the pod in 379 of 379 samples. The 95th-percentile wrist displacement under the same 100 ms pairing error was 44.05 mm. The two episodes reuse the camera and record format. The packing motion comes from an authored trajectory.

Play the sample-pod packing episode
4K UHD SVG Left wrist · Right wrist · Fixed overhead
Left wrist, right wrist and overhead views of the robot holding a capped sample pod beside the packing case.
0.00 / 7.58 s
379 synchronized samples and authored next-pose targets.

What makes this a useful way to generate data

Flash ALOHA’s fast demonstrations came from operators directly guiding backdrivable arms. The robot recorded two wrist views, an overhead view and encoder positions. The authors masked parts of the operator that would disappear at deployment. Their slower puppet-controlled system was a separate baseline. Flash ALOHA, Figure 6 and Appendix V · Capture code

The paper’s four ARX task datasets use 50 Hz recording at 640 × 360, resized to 320 × 180 for policy input. The repository’s generic bimanual configuration gives a 100 Hz example. Our visual export follows the ARX paper’s sampling rate and camera roles with original wrist mounts. The archive pairs every 20 ms observation with the next authored kinematic target, with no target after the terminal frame. Bimanual configuration

MotionOrigin’s proposal. Author the motion once, render every camera from it and save the pose targets on the same clock. For a new task, change the object, fixture, contact constraints and trajectory. Keep the cameras and observation format tied to the target robot. Use a short set of real robot recordings to calibrate the optics, verify the action mapping and test tracking around contact on held-out layouts. How the saved episodes work

Download tomato-sorting dataset · Download pod-packing datasetView capture and visibility checksMethods and reproduction

Embodiment, precision and touch

What a worker’s head camera leaves out

Imagine a skilled worker packing a bottle into a divided case. The head camera shows the left hand bracing the case and the right hand lowering the bottle. It gives us the sequence, the object relationships and an example of expert timing. The case lip hides the last millimetres of the descent. The worker can feel when the bottle catches and when the floor takes its weight. The camera alone leaves both events ambiguous.

Robot-view synthesis can make that demonstration easier to use. The camera shows the robot’s own fingers at their real scale and angle. Joint state tells the policy where those fingers actually are. Contact sensing or a validated feedback signal helps establish whether the object has seated, whether the pads are slipping, and when opening the hand is appropriate.

Different observations answer different control questions
SignalWhat it can tell the learner
Human egocentric RGBThe order of the work, the human’s hand roles, visible approach and separation, and rough timing. Around an occluded landing, several contact histories can produce similar frames.
Calibrated robot-view RGBHow the same task looks from the robot’s head and wrists, where its fingers enter the image, and what its hand obscures. Exact geometry and camera calibration help align what the policy sees in training and deployment.
Robot joint state and action logsJoint and digit position, how quickly they are moving, which targets were issued, and how much motion lagged the command. These channels connect images to a realizable action on that body.
Contact feedbackTactile, force, calibrated current, or another validated contact signal can reveal slip, load transfer or support. The exact signal and whether it is measured or simulated belong in the record.

Published comparisons separate visual resemblance from usable actions

EgoEngine reports 3% → 51% mean success on four physical tasks. Its direct-retargeted human-video condition achieved 3% and its visual-only conversion condition 5%. Refining the action generation alone reached 43%, and combining both branches reached 51%. In that experiment the action branch accounted for the larger improvement. The result helps explain why adding a convincing robot skin to a human video can leave grasping and placement errors intact. EgoEngine, Tables 3–4

RoboTryOn measures the two gaps separately. On its image-feature measure, translated videos were closer to real robot observations, reducing Fréchet SigLIP distance from 125.5 to 32.7. In a separate open-loop action replay experiment in simulated replicas, simple retargeting succeeded on 34.8% across three tasks. The full action-labeling pipeline reached 69.7%. Appearance similarity and action executability are different outcomes, measured by different tests. RoboTryOn, Appendix D, Tables 7–8

Worker footage still has real value. EgoMimic found that human egocentric data helped when co-trained with robot demonstrations and aligned across embodiments. Phantom also reports physical tasks learned from human demonstrations after hand-pose processing and robot overlays. These studies support collecting human examples for coverage and intent, then testing how to align them to a particular policy. Geometry-guided VLA adaptation offers another route. Use generated observations for the visual geometry they preserve and fit the action head with measured robot data. EgoMimic, §V · Phantom, §4 · Supervise What Survives

Touch supplies a different kind of evidence. On a physical cup experiment, Calandra et al. report 95 successes in 100 attempts for a tactile-plus-vision grasping system optimizing success. The authors report 76% over 50 attempts for the analogous vision-only evaluation. They also use feedback to find lower-force grasps. These figures belong to the specified object, sensors and controller. A 100× learning-efficiency claim for MotionOrigin would require a matched study of how much training data each system needs to reach the same quality and speed target. Calandra et al., physical grasp and force experiments

Original experiment · Same visual cue, different grip

We constructed an analytic pinch-and-lift test with paired, visually indistinguishable packages. Each pair has the same known mass, requested upward acceleration and actuator gain. One contact surface can be grippy and easy to damage, and the other slick and mechanically sturdier. We give both non-contact policies the load in advance. This is a deliberately favorable assumption for vision and joint feedback. The differing friction and damage limits are hidden from both at the point where one force must be chosen.

For a symmetric two-pad pinch in this model, the minimum normal force per pad is Nmin = m(g + a)/(2μ). The upper limit is the force at which the package is damaged. Each individual package has a feasible interval. If two intervals do not overlap and the policy receives identical evidence, one force can handle at most one member of the pair. In an actual held-out model pair, the fragile package needs 1.91–3.09 N per pad and the slick one needs 4.65–6.31 N per pad.

Successful simulated lifts, without slip or damage
Test conditionVisual priorJoint feedbackJoint + contact
Force windows mostly overlap498 / 720416 / 720719 / 720
Disjoint force windows163 / 720285 / 720707 / 720
Disjoint, noisier contact signal160 / 720272 / 720592 / 720

In the disjoint-grip condition, every pair has a different safe force window. Even perfect execution of a shared force can therefore succeed on at most 360 of 720 attempts. A noisy pre-lift probe supplied enough extra information for 707 of 720 successes in the first test and 592 of 720 when probe noise increased. The overlap test is an important control. Its theoretical ceiling for an exact shared force is 711 of 720, so the information gap depends on contact physics and the task.

The joint-feedback baseline improves the execution of a chosen grip. In the overlap condition its pooled force prior is conservative and produces more slips than the visual prior. In the disjoint condition, even tighter execution cannot tell the controller which of two force windows applies. This is why the policy needs separate evidence about the hand’s state and the object’s contact state.

Contact test assumptions, code and trial records

The fixed prior chooses a grip command from the visible-task load. The calibrated joint-feedback arm also receives a simulated estimate of actuator response. The contact arm adds a quantized, noisy signal from a low-load tangential probe while a fixture still supports the object. A small fit maps its signal to a friction estimate. We selected the controllers on 480 training pairs and used 360 unseen matched pairs per test condition. Reported successes require avoiding both slip and damage at the modeled lift load.

These results isolate a pre-lift decision with identical visible evidence by construction. The friction and damage ranges are assumptions, and the simulator omits full fingers, deformation and visual slip correction after the lift starts. It measures the value of an added observation in this model. Policy training time, real-hardware success and the schedule cost of probing require independent tests. A real robot can also gather new visual or contact evidence before deciding, changing the information it has available.

Simulation codeContact results (JSON)All 6,480 policy–case records (CSV)

Prompt engineering for an exact robot point of view

The useful advance is to prompt for a structured task plan, then use robot geometry, calibrated cameras and a feasible trajectory as controls for generation. A single prose prompt can describe the robot. Fixed measurement and geometry conditions give the generator a shared reference it can follow across frames and viewpoints.

  1. Lock the embodiment and optics

    Condition on the target robot’s approved mesh or URDF, joint limits, gripper or finger configuration, and measured camera intrinsics and mount transforms. Provide reference renders from the actual optics. Fix hand identity and handedness across keyframes. Render changed light and backgrounds as controlled variations around those calibrated views.

  2. Prompt the contact sequence as events

    Ask the VLM to return object and fixture IDs, supporting and moving hands, visibility, pre-contact alignment, closure, support transfer and release. Provide measured phase durations with permitted variation. Supply approach, grasp and settle keyframes plus hand skeleton, rendered depth or segmentation controls from the same scene geometry. Recompute the trajectory and render after changing the phase timing.

  3. Validate before assigning labels

    Reject clips with joint-limit violations, discontinuous object motion, changing finger counts, loss of contact during a required hold, or inconsistent camera projections. Join all views at the same timestamp to authored targets, estimated actions or measured robot state and commands. Keep uncertain inputs explicitly unknown. A generated contact image can cue a grasp. Sensor evidence or a calibrated simulator must supply any touch or force channel.

  4. Train visual cues and close the loop on the real hand

    Use generated views to learn where and when to act. Check target feasibility with synchronized state and action labels. If only image geometry has been validated, use that geometry for visual supervision and learn executable control from calibrated robot data. During execution, measure the hand and object again before tightening, lifting or opening.

RoboTryOn implements parts of this recipe through autoregressive robot keyframes, hand-skeleton flow, phase retiming before generation, and an inverse-dynamics model trained on teleoperated data. Its physical ablations showed a drop from 75.9% to 42.0% on its OpenArm in-domain average when hand-skeleton flow was removed from the full system. That comparison supports structured motion conditioning on this pipeline and motivates explicit reprojection and contact tests in ours. RoboTryOn, §3 and Table 3

Example prompt for an expert-paced packaging episode
Goal
Pack the upright sample pod into its marked cell. Time the full
episode from the agreed start through stable placement.

Required inputs
Use the supplied robot geometry, camera calibration records, pod
and case geometry, joint limits and expert phase measurements.
Mark any missing measurement as unknown.

Return an episode specification
For each phase, give the hand role, object ID, visible cues, target
pose reference, timing range, and condition required to advance.
Use approach, grasp, lift, align, seat, release and retreat.
Keep the pod upright. Begin release only after stable support is
verified. Flag the episode if the expert deadline is infeasible.

Generation controls
Use the same joint and object trajectory for the wrist and overview
views. Condition the renderer with calibrated keyframes and geometry.
Log RGB, robot state and action targets with a shared clock. Label
contact observations by source and attach uncertainty to estimates.

Acceptance check
Check identity, camera reprojection, collisions, action timing,
stable seating, and visible release. If a check fails, return the
failing segment and the evidence required to repair it.
Methods · Dexterity and geometry

The finger motion behind the sort

The task is bimanual and asymmetric. One hand keeps the fruit supported while the other changes opposition and turns it. During release, the thumb frees the fruit first and the other digits open as the wrist lifts. A wrist path alone leaves out the evidence of which surfaces touch the object and how they separate.

The animation revision reduced peak index-pad distance from the center of the fruit from about 51.5 to 46.5 mm. During the inspections, two or more pads per hand remained in the modeled 29–36 mm envelope. In the authored layout, the fruit reaches the tray at 12.08 s. The thumb first gains 7 mm of radial separation at the 12.375 s sample, and the other fingers reach the same modeled threshold between 12.438 and 12.542 s. The geometry audit checks 1,920 poses at 96 Hz, finds no intersections within the tested 2 mm tray-clearance allowance, and records stable modeled fruit position after settling. These are geometric properties of the reconstruction. Contact forces and deformation need a validated contact model or real sensing.

Explore the robot’s finger motion

The supporting palm, index regrasp and thumb opening are individual movements. This hand study makes that independence easier to see.

4K UHD SVG 3840 × 2160
Hand close-up and full arm assembly in the MotionOrigin vector study.
0.00 / 15.70 s
Eight reference-timed gestures · Scalable vector paths
Results · Controlled simulation

Timing, tracking and inspection each solve a different failure

The visual demonstrations show what the task looks like. The numerical experiment below asks which control and perception changes make a correct sort possible within 7.55 s. This is an engineering deadline taken from the authored concept from approach at 5.23 s to release at 12.78 s. It gives all conditions a common clock.

Experimental design

We built a reduced closed-loop simulator with horizontal reach, height and roll as its three motion coordinates. A scripted teacher supplies position observations and command targets at fixed phases. A ridge-regression controller learns action targets from ten observations per phase, and a regularized logistic classifier learns routing from rendered RGB crop features. The simulated plant applies speed limits and a first-order velocity response. It can delay observations, drift from the planned belt intercept, miss a grasp or fail to settle before release.

For each comparison, three independent data seeds each use 50 development or training episodes for the model fits and 120 fresh test cases per condition. The reference cases share task geometry and near-nominal conditions. The shifted cases vary fruit appearance, light, actuator response, sensing delay and belt motion. The expanded cases also include belt acceleration, camera yaw error and partial camera obstruction. Test cases are matched across conditions within an experiment. Seeds and cases change between the motion and final inspection comparisons.

A correct sort needs a grasp within tolerance, at least 123° of roll, the right destination and placement within 12 mm after the modeled settling interval. The control cadence is fixed at 30 Hz. Timing refers to the commanded phase schedule and completed rollout, while the camera reconstruction elsewhere in the paper is sampled at 50 Hz. Every case remains in the denominator. Paired 95% simulation intervals resample matched cases within the three fixed training seeds. They describe uncertainty from the modeled cases.

Experiment 1 · Faster schedules can spend the contact margin

Reference cases, 360 matched trials per condition
Training conditionCorrect sortsCorrect by deadlineMedian time¹
Native timing343 / 3600 / 3609.37 s
Uniform 2×113 / 360113 / 3605.03 s
Phase-aware279 / 360279 / 3606.97 s
Phase-aware, head camera only267 / 360267 / 3606.97 s

¹ Median among correct sorts only.

Phase-aware timing added 46.1 percentage points of correct, on-time sorts over uniform 2× (paired 95% interval 40.6 to 51.4 points). Successful runs were a median 25.6% shorter than at native timing. The phase-aware arm lost 17.8 percentage points of correct sorts overall against native timing, exceeding the proposed ten-point hardware limit. Adding the second camera improved correct, on-time sorting by 3.3 points in this model.

Under the first combined stress test, phase-aware timing missed all 360 grasps. A fresh stress batch isolated a simple feedback change using the two most recent delayed encoder readings to extrapolate arm position.

Fresh shifted cases, 360 matched trials per condition
Training conditionCorrect sortsCorrect by deadlineMedian time¹
Phase-aware0 / 3600 / 360—
Phase-aware + delay compensation126 / 360126 / 3606.97 s

¹ Median among correct sorts only.

That correction recovered 126 / 360 correct and on-time sorts (35.0%). The remaining first failures included 146 missed grasps, 42 unstable releases and 46 wrong routes. A faster clock was useful only when the controller could follow the target through contact.

Experiment 2 · Track the moving fruit while the hand closes

Moving target and estimated arm state

Use the first camera sighting and timestamped belt measurements to predict where the fruit is now. Advance the delayed arm estimate through the commands already issued. During pickup, match the hand’s velocity to the fruit’s velocity and check sustained alignment before lifting. At the tray, check position and velocity before release.

On the same shifted cases, this combined change reduced missed grasps from 151 to 3 and raised on-time sorts from 140 / 360 to 270 / 360. The intervention changes both the moving pickup target and the arm estimator. It does not identify either effect alone. Physical conveyor replanning precedent

Color normalization and half-turn

Measure the color of a candidate patch relative to the same fruit and require adjacent pixels to agree. Under combined shifts, normalization raised correct, on-time sorting from 270 / 360 to 321 / 360. Adding a half-turn brought it to 329 / 360. The half-turn’s incremental effect was smaller and uncertain in those shifted cases.

Aggregate scores obscured a trade-off. In expanded stress, the tracking-only condition sorted 172 of 180 blemished fruit correctly and accepted 109 of 180 sound fruit. The normalization plus half-turn condition improved overall sorting, but correctly routed only 137 of 180 blemished fruit. That made a separate inspection experiment necessary.

In matched reference cases, the full motion design produced 340 / 360 correct sorts against 342 / 360 at native timing, inside the proposed ten-point accuracy margin. It reached 340 / 360 on time against 94 / 360 at uniform 2×.

Experiment 3 · Give the classifier enough pixels

In a fresh holdout, we held motion, the two viewing angles, control cadence and scoring fixed. We compared 24×24 and 40×40 crops of the same class of rendered fruit, then added a lower rejection threshold selected on development cases. The smallest front-facing mark spans roughly two pixels in the original crop and three to four in the denser crop. A coherent patch is easier to distinguish after sampling it across multiple adjacent pixels.

Expanded stress · same fruit and controller
InspectionCorrect by deadlineBlemished fruit sorted correctly
24 px crop324 / 360144 / 180
40 px crop335 / 360155 / 180
40 px + lower reject threshold351 / 360174 / 180

On the expanded holdout the denser crop and lower threshold added 7.5 percentage points of correct, on-time sorts (paired 95% interval 4.4 to 10.6 points) over the 24-pixel crop. Blemished-fruit recall improved from 144/180 to 174 / 180. Correct sound-fruit routing moved from 180/180 to 177/180. The result is a trade between missed defects and false rejects that the workcell owner can price explicitly.

360 / 360correct and on time in reference conditions
349 / 360correct and on time under combined shifts
351 / 360correct and on time with added belt and camera disturbances
Final independent inspection holdout
Test conditionsCorrect by deadlineBlemished fruit sorted correctlySound fruit sorted correctly
Reference360 / 360180 / 180180 / 180
Shifted349 / 360174 / 180175 / 180
Expanded stress351 / 360174 / 180177 / 180

All correct sorts in this final comparison met the 7.55 s model deadline. The expanded holdout’s median successful sort lasted 6.97 s. The finite reference result of 360/360 comes from these synthetic cases. The simulator assumes idealized object crops, an approximate servo model, a limited blemish family and simplified optical disturbances. Results do not establish fingertip force, contact compliance, recognition of real defects and lookalike marks, or policy performance on a physical robot.

Run the experiments and inspect every trial
Model schedule by phase, in seconds
PhaseNativeUniform 2×Phase-aware
Approach1.550.780.98
Grasp0.660.330.54
Lift1.050.530.81
Roll2.151.071.77
Transfer1.650.821.02
Place1.300.650.94
Release0.700.350.57

Across the three studies we generated 3,600, 6,480 and 3,240 condition–case records, respectively. Matched rows share test scenarios across arms. The counts describe evaluated policy executions and should not be treated as that many distinct physical trials. The scripts and outcome tables below reproduce each study independently.

Original timing codeOriginal results (JSON)Original trials (CSV)Moving-grasp experimentMotion results (JSON)Motion trials (CSV)Inspection experimentInspection results (JSON)Inspection trials (CSV)Sources and reproduction

Proposed data pipeline

A vision-language model for expert-paced demonstrations

We propose a vision-language model that can take a task instruction, a workcell specification and examples of expert motion, then produce first-person episodes at the fastest repeatable pace for that task. The model would specify what to show and when. An offline renderer or video generator would save the images. A robot trajectory or calibrated action model would supply targets for policy training. The deployed policy would use live observations to correct its motion.

  1. Describe the task and its evidence

    Start from language, source video, object and fixture definitions, and task-specific pass criteria. For tomato sorting, label approach, pickup, support handoff, roll, route decision, placement and release. Record what is visible, which hand carries the load, the condition that permits the next phase, and how certain the annotation is. Include missed pickups and unclear views as candidates to reject or recover from.

  2. Set speed from measured expert work

    Specify the upper end of repeatable expert pace using independent, quality-qualified human blocks on the same geometry. Keep the raw event clock and attach a phase duration distribution. Use the overall expert deadline to allocate time where a safe trajectory allows it. A loaded hold can follow a faster free-space move, and an uncertain blemish can still require a second view.

  3. Generate a robot-feasible version of the motion

    Map fingertips, hand roles and object motion into the target arm’s reachable coordinates. Check joint, speed and acceleration limits and collision geometry along the trajectory. Use targeted contact checks or instrumented replay for grasp feasibility and object stability. Store edits, rejects and failure reasons. At the tray, ensure support before the fingers open. On the belt, maintain hand–fruit alignment while the grasp settles.

  4. Render the episode and label it on one clock

    Render the wrist and overview views from the same motion and calibrated cameras. Save RGB alongside timestamped pose or action targets and their provenance. If a generative renderer alters lighting, texture or backgrounds, check object identity, geometry and visible contact events across frames and views. Use measured signals or a validated contact model for force or touch targets.

  5. Train the policy and let live feedback govern transitions

    Use the generated episodes to supervise timing, contact-relevant observations and robot actions. In deployment, the policy can advance through the reference with live camera and joint feedback. Allocate an explicit latency budget for each observation and command. Hold or replan when the object has slipped, a cue arrives late, or a placement has yet to settle. Count all those holds on the actual cycle clock.

Why a good-looking video can still fail as training data

Suppose q is a joint path and we replace its time with a faster reference clock. A twofold speed-up of the same path demands twice the velocity and four times the acceleration. Changes in the rate introduce additional acceleration at phase boundaries. A correct retiming tool therefore warps a continuous trajectory, checks it against motor limits and preserves the minimum duration of verified contact stages. At 30 Hz in the reduced study, three to six frames of observation lag are 0.10–0.20 s. Subtract 0.28 s of settling from the uniform 2× release budget of 0.35 s and there is no nominal allowance for that full delay. The phase-aware schedule allocates 0.57 s to the release. These are scheduling bounds before the common grace window and the actual dynamics.

Frame rate, playback rate, reference pace and controller cadence are different controls. A display can play a recording at 2× without changing the signal reaching the policy. A robot can also act faster with the same command rate by tracking a faster feasible trajectory. The deployment logs should retain source-frame time, hardware capture time, observation arrival, action issue time and achieved motion. Flash ALOHA’s chunk scheduler makes the timing stakes concrete. Ten overlapping actions give 100 ms at 100 Hz but only 50 ms at 200 Hz. A controller receiving stale targets can lose that margin even when average inference looks fast. Flash ALOHA inference code

What constitutes a useful episode

Task and geometry
Task ID, instruction, object instance and outcome, fixture revision, robot embodiment, safety envelope and allowed recovery modes.
Clock and views
Episode identity, start and reset events, native timestamps, rate map, synchronized view names, intrinsics and extrinsics, and missing-frame or occlusion flags.
Motion and supervision
Action convention and units, target versus observed state, phase boundaries, hand role, object support, fingertip visibility, and provenance for authored labels, estimated actions and any sensor or simulator measurements.
Validity
Replay checks, rejected segments, cases used for training or evaluation, policy lineage, and which target-task teleoperation records, if any, informed the checkpoint.

RGB can support proposals about visible geometry and timing. Physical quantities such as metric joint motion, force, pressure and torque require calibrated measurements or explicitly labeled simulation estimates. A generated observation should never be silently paired with a pose that comes from a different rollout. Our two released examples illustrate the synchronized format. Their next-pose labels are authored kinematic targets for the concept rig.

Validation plan

How we would prove it on hardware

The physical claim needs three comparisons. Can the policy learn the skill from generated robot-view data? Which pace in the training set helps it perform reliably? Does feeding a faster reference to the same deployed policy change the achieved speed? A separate comparison identifies each effect.

Measure the human target before running the robot

Use the same fixture geometry, object class, start signal and definition of completion for people and robots. Include repeated blocks from skilled operators. Specify the maximum repeatable expert pace in advance as the fastest quality-qualified block under a minimum trial count, and report the distribution of individual cycles and per-phase timing. A time from a single best attempt does not characterize a sustainable pace. Keep a separately measured expert target for each task and layout class, with held-out human trials where possible.

On a conveyor, record fruit spacing and speed, and include the whole arrival stream in the denominator. For one cell that handles one fruit at a time, the nominal spacing must be at least belt speed × the planned cycle budget. The figure study’s 45 mm/s belt and 7.55 s model deadline imply about 0.34 m per fruit, and 70 mm/s implies 0.53 m. That arithmetic gives a starting spec. Sustained correct throughput must also account for failed pickups, variable arrival spacing, retries and clearing the reject tray.

Test the training data on a fixed robot and policy class

Proposed hardware comparisons
ConditionWhat the comparison isolates
Scripted simulator baselineTrain on ordinary scripted robot examples in the same geometry, with a fixed measurement and calibration budget.
Same generated clips at native paceUse retargeted robot-view demonstrations with finger-level cues and their source timing.
The same clips at uniform 2×Compress every phase with the same ratio while keeping the control cadence, labels and task fixed.
Expert phases with finger-level cuesUse measured expert phase budgets, feasible trajectories, verified contact intervals and live feedback at the same control cadence.
Expert phases with coarse labelsKeep expert pace and withhold digit-level targets and auxiliary labels to test the contribution of visible finger–object cues.
Synthetic-only target-task conditionKeep target-task teleoperation out of the training lineage and disclose reused calibration, general robot data and pretrained weights. Compare against mixed and real-data baselines with the same evaluation.

Start with upright bottle or pod placement in a divided fixture. Measure final position, tilt and a stable interval directly. The same setup can grade a grab, move and release without relying on a blemish classifier. Then test tomatoes under shifted ripeness, defects and cosmetic lookalikes, jams and varying belt speed. Seedling insertion is a follow-on with occluded geometry and different compliance. Match layouts and object instances across arms and withhold whole groups of near-duplicate episodes and retimed derivatives from evaluation. Randomize policy order within physical reset blocks.

Test whether the saved episodes can replace sandbox setup

On the same physical packing task, fix the policy architecture, target camera setup, control cadence, expert timing and real robot calibration budget. Compare that real-data base on its own, with saved generated episodes and checked action targets, and with an equal number of labeled episodes from an interactive physics simulator. Keep the episode conditions and target action convention comparable. Within the generated arm, also compare visual pretraining alone against training with the added action targets.

Evaluate each policy on unseen placements, including shifted objects or catches at the fixture lip. Score stable placement by the expert deadline, damage, drops and interventions. Record person-hours for scene setup, authoring and verification, robot collection time and compute, including rejected clips. The hypothesis is supported if the offline route meets a prespecified quality and speed margin with less setup effort against the simulator baseline. Recovery performance will show where a fixed library of demonstrations needs to be expanded.

Isolate the live playback effect

Freeze the learned checkpoint and its control cadence. Compare a 1× reference, a continuously accelerated reference, and a mid-run change after verified grasp on the same object. Include a display control that plays at 2× on the operator screen while the policy still receives the 1× reference. When changing speed online, advance from the current reference time. Preserve continuity and log exactly which cue reached the policy. In the native bottle example, the first visible release occurs around 5.625–5.875 s. The same cue would be requested around 2.813–2.938 s at 2×. Physical timing still comes from measured release and confirmed stable placement.

Define the decision before the trials

For every fruit or bottle, score correct completion by the expert deadline, overall success, placement error and the full cycle-time distribution. Log rejected or missed fruit, false rejects, drops, manual resets, failed grasps and task timeouts. Report defect recall and sound-fruit acceptance separately. Use a pilot to select sufficient trial counts and report paired differences and uncertainty across reset blocks, object instances and training seeds. Archive the model checkpoint, data lineage, unsped videos, camera and controller configuration, and timestamped commands and measured state.

The phase-paced policy passes the timing hypothesis if it improves correct and on-time sorting over uniform 2× while retaining overall success within the proposed ten-percentage-point margin against native timing. A lower success rate outside that margin, a contact failure at the required pace, or a missed-defect rate outside the task specification each rules out that operating point. The model targets and numerical margin are test criteria, with the production acceptance thresholds to be set from the real workcell’s costs and tolerances.

To test the embodiment and sensing hypothesis on the same packaging task, hold the expert demonstrations, observation timing, policy backbone and physical reset blocks fixed. Compare direct retargeting from the worker view, calibrated robot-view rendering with the same approximate targets, verified action labels on the calibrated views, and the same system with contact feedback added. Use real robot encoders for the state channel and measure the cost of extra observations and probing in the complete cycle. Estimate learning efficiency with curves of robot-data hours, total collection cost and interventions required to reach the same quality-qualified expert deadline. This is the comparison that could substantiate a large learning multiplier.

Discussion and conclusion

Why the research is worth pursuing

The initial MotionOrigin insight holds up as a research direction. Expert video offers a way to specify the sequence, the hand–object interaction and the pace of a task. The external hardware literature shows that robot-view synthesis and phase-specific tempo can improve learning or execution in specific settings. Our tomato experiments add a local result with visible failure modes. A phase schedule made fast sorts possible, then failed on moving fruit under stress. Live tracking recovered the grasp. Better sampling and classification recovered many of the remaining missed defects. The useful lesson is what each intervention repaired.

The camera export demonstrates the first step across two tasks. A short, saved robot-view episode can reproduce the visual cues a training pipeline needs, using authored motion and synchronized cameras without a full physics sandbox. The next step is to test how much engineering and robot recording this saves once the action targets and recovery behavior are checked on hardware. Expert pace is achieved only when the physical robot repeatedly finishes on time at acceptable quality.

Sources and data availability

Inspect the evidence and reproduce the results

The cited literature was checked for this edition on November 19, 2026. The experiment files include individual case outcomes, the paired analyses and the original camera exports. The two case-study disclosures retain the source limitations, including the generated origin of WP23 and the difference between camera rendering and hardware measurements.

Source register, code and episode audit

All simulated numerical comparisons can be rebuilt with the scripts linked from the results section. The video and geometry audits retain the source rates, clocks and distinctions between measured model state and authored targets.