BEYOND MOTION EXECUTION

Beyond Motion Execution: Benchmarking Embodied Agents on Wet-Lab Operations and Procedures

Benchmark overview. Four operation families and six zero-shot tasks surround a summary wheel of 14 atomic skills, the dual-arm Lift2, 14 demonstrated tasks, 500 training episodes each, and four generalization settings. The right side shows a titration that scores 0.4 and fails by overshooting, plus capability profiles and success-rate and stage-score summaries for InternW0-Δ, OpenWAM, π0.5, and Qwen+π0.5.
Benchmark overview. Fourteen demonstrated tasks and six zero-shot tasks span four operation families. Stage-wise scoring keeps partial progress visible: the titration example earns a process score of 0.4 and fails because it overshoots the endpoint. Results compare InternW0-Δ, OpenWAM, π0.5, and Qwen+π0.5 on demonstrated tasks, zero-shot tasks, and all 20 tasks.Open full figure

Benchmark at a Glance

20Laboratory tasks14 demonstrated and 6 zero-shot, across four operation families
14Atomic skillsComposed into 12 basic operations and 8 experimental procedures
7,000Task demonstrations500 per demonstrated task, varying task configuration, background and instruction
680Evaluation episodes per model40 per demonstrated task over four test settings, 20 per zero-shot task

Key features

  • Dynamic liquid transfer

    Pouring moves simulated liquid between vessels. Metrics check how much reaches the target and how much is retained, such as ≥80% transferred or ≥95% kept in the beaker.

  • Instruments and readouts

    The centrifuge, drying oven, burette and analytical balance respond to buttons, doors, knobs and stopcocks. The balance shows a mass readout, and two tasks require reaching a target mass.

  • Reaction observation

    Appearance changes with the procedure. Five tasks require observing the result, such as a pale-pink titration endpoint, the blue-bottle color cycle, a brick-red precipitate or a silver mirror.

  • Skill-aligned stage scores

    Each episode receives binary success and a stage-wise score whose metrics are tagged with atomic skills, so partial progress and the point of failure stay visible.

  • Two generalization tests

    Demonstrated tasks are tested under task-configuration, background and instruction shifts, one at a time and mixed. Six zero-shot tasks have no demonstrations and test new operation compositions and experimental conditions.

  • Dual-arm robot, real-to-sim assets

    All tasks use the dual-arm Lift2 robot. Laboratory assets come from Lab2Sim, an agentic pipeline that builds them from object specifications and admits them only after geometry, articulation and physics checks.

What is WET-OP-Bench?

WET-OP-Bench is a dual-arm simulation benchmark of 20 wet-lab tasks. In a laboratory, completing the right motions is not enough: an agent can still spill a reagent, miss a target mass, or overshoot a titration endpoint. The simulator therefore couples manipulation with evolving experimental states, including dynamic liquid transfer, quantitative instrument readouts, and procedure-conditioned color changes, and the same states drive both what the agent observes and how success is checked. Tasks are built from a shared set of atomic manipulation skills and range from basic operations to multi-step experimental procedures. Fourteen tasks come with demonstrations and are tested under controlled task-configuration, background, and instruction variations; the other six have no demonstrations and test zero-shot generalization to new operation compositions and experimental conditions. Every episode is scored by binary success and by a stage-wise score whose metrics are aligned with atomic skills, so partial progress and failures remain visible.

Overview of the 20-Task Benchmark

Task names, operation families, scope, four capability dimensions and distinctive skills. Explanations follow the table.
IDFamilyTaskScopeHorizonPrecisionQRDistinctive skills
Family
HP: Handling & Placement, TM: Transfer & Mixing, IO: Instrument Operation, MD: Measurement & Dosing. Tasks that combine two families show both, such as IO + TM.
Scope
Op.: basic operation, Proc.: experimental procedure.
Horizon
Number of atomic-skill steps. Short: 1–5 steps, Medium: 6–10 steps, Long: more than 10 steps.
Precision
Required positional accuracy. Low: tolerance above 2 cm, Medium: above 5 mm and up to 2 cm, High: 5 mm or less.
Q
Quantitative-target requirement. Yes: success requires reaching a specified quantitative target rather than merely obtaining a numerical readout, No: no quantitative target.
R
Reaction-observation requirement. Yes: success requires observing a reaction-induced visual state (a titration endpoint, precipitate formation or color change), No: no reaction observation.
Distinctive skills
Representative atomic skills of each task. Every skill a task uses is listed in its metrics under Task showcase.

Details

Training data and evaluation settings are built from three variation dimensions. Demonstrated tasks test within-task generalization; zero-shot tasks test cross-task generalization.

Data splits and generalization settings
Task splitDataConstructionTest settingEpisodes per taskGeneralization
Demonstrated
Tasks 01–14
Train500 demonstrations per task. Every episode combines a task-configuration, a background and an instruction variant.—500—
TestSame tasks, objectives and operation structure as in training.C-only, B-only, I-only and Mixed (C+B+I), 10 episodes each40Within-task: robustness to controlled physical, visual and language variation
Zero-shot
Tasks 15–20
TestNo task-specific demonstrations. Operation families and atomic skills also appear in the demonstrated tasks.Mixed (C+B+I)20Cross-task: new operation compositions and/or experimental conditions
Task configuration (C)
50 variants per taskTask-relevant physical or experimental parameters: object positions, liquid amount or color where applicable, and the duration of time-dependent steps.
Background (B)
10 variants per taskTask-irrelevant visual context: tabletop appearance, room background and illumination.
Instruction (I)
20 variants per taskThe level of descriptive detail in the language instruction, with the same task semantics. Task cards show the canonical instruction, which states the scored requirements, including intermediate transfers, opposite-slot centrifuge loading, and the final upright reset.

Training episodes vary C, B and I jointly. In the test settings, C-only, B-only and I-only each shift one dimension in isolation, and Mixed (C+B+I) varies all three jointly; task identity, objective and operation structure stay the same as in training. Zero-shot tasks are evaluated only under Mixed (C+B+I).

Results

Task success stays below the stage score for every model. On the 14 demonstrated tasks, InternW0-Δ reaches 34.6 ± 1.1% success and a stage score of 60.5 ± 1.0, ahead of π0.5 (25.2 ± 1.4% and 51.2 ± 0.4), OpenWAM (22.7 ± 0.5% and 55.2 ± 0.5), and Qwen+π0.5 (20.8% and 42.4). Across all 20 tasks, InternW0-Δ reaches 24.5 ± 0.9% and 47.0 ± 0.8, compared with 17.6 ± 1.0% and 39.9 ± 0.3 for π0.5, 16.0 ± 0.3% and 42.5 ± 0.4 for OpenWAM, and 14.6% and 33.3 for Qwen+π0.5.

Success rate and stage score by operation family, for demonstrated tasks, zero-shot tasks, and all 20 tasks.
Model HP (3) TM (4) IO (4) MD (3) Demo (14) Zero-shot (6) All (20)
SRScore SRScore SRScore SRScore SRScore SRScore SRScore
π0.547.267.04.223.532.975.520.839.925.251.20.013.717.639.9
OpenWAM51.165.57.542.725.476.110.833.822.755.20.312.716.042.5
InternW0-Δ55.365.821.050.842.183.521.937.334.660.50.815.524.547.0
Qwen+π0.539.358.01.915.126.959.519.440.620.842.40.012.014.633.3

Success rate (SR, %) and stage score (0–100). Demonstrated columns pool C-only, B-only, I-only, and Mixed, with 40 episodes per task per run. All (20) is the task-balanced mean. π0.5, OpenWAM, and InternW0-Δ are each fine-tuned for 100k steps on the 14 demonstrated tasks, with global batch size 256, and the headline ± values are the standard deviation over three runs of 680 episodes. Qwen+π0.5 is Qwen3.8-Flash planning over frozen π0.5, with task memory and in-context demonstrations turned off; its numbers are one run of 246 completed episodes and have no standard deviation. Bold marks the best value in a column.

Within-task generalization and zero-shot transfer

Success rate and stage score for each within-task setting and for the six zero-shot tasks. Each cell is success rate, then stage score.
Split / settingInternW0-ΔOpenWAMπ0.5Qwen+π0.5
C-only (demo)34.3 / 60.722.6 / 54.924.0 / 50.0— / —
B-only (demo)33.1 / 58.724.5 / 56.228.6 / 54.027.4 / 45.3
I-only (demo)35.2 / 61.023.3 / 56.423.8 / 49.8— / —
Mixed (demo)35.7 / 61.420.2 / 53.424.3 / 50.919.0 / 41.6
All demo settings34.6 / 60.522.7 / 55.225.2 / 51.220.8 / 42.4
Zero-shot (6 tasks)0.8 / 15.50.3 / 12.70.0 / 13.70.0 / 12.0

Cells are success rate (%) / stage score (0–100). Qwen+π0.5 has no C-only or I-only episodes in the completed run. On the six held-out tasks every model stays below 1% success. Partial stage credit comes mainly from opening steps that match a demonstrated operation.

Zero-shot task