Agents write robot manipulation programs
The agent observes camera images and robot state, then writes Python that issues bounded native control actions. There is no task-specific motion or grasp library.
RoboSWE-ADAPT
In-context adaptation for embodied coding agents.
No demonstrations, no fine-tuning, no pick-and-place API. Agents program a simulated robot from native controls and get five retained-context trials per RoboCasa365 task. We evaluate the resulting trial-by-trial performance trajectory.
Code and context persist across five trials
The physical simulator returns to the same admitted starting state. The agent retains its codebase, notes, and conversation, with model parameters and trial budgets held fixed.
Evaluation measures the full trial sequence
Task-success conditions determine each trial’s outcome. Per-trial outcomes, failure-to-success conversion, consistency, and successful-execution costs describe adaptation. Success does not end the series early.
RECORDED EXAMPLE BlendIngredients · Opus 5.5
- Runnable pilot candidates
- 11
- Curated entries · 85 native tasks
- 83
- Trials per task
- 5
- Control steps per trial
- 6,000
- Active time per trial
- 30 min
- Weight updates
- 0
Recorded robot execution
Recorded execution and agent traces
Inspect synchronized camera views, submitted code, and the agent’s recorded action rationale.
BlendIngredients / Opus 5.5
Download aligned evidence ↗Historical pilot · attempt 2 of 2 · 1,526 control steps · observed sustained success
Simulation-time playback removes thinking pauses. This archived example illustrates the interface; it is not a result from the new five-trial release.
Five-trial adaptive protocol
Evaluation protocol
Trial 1: task instruction and an empty codebase.
The agent receives the task instruction, camera observations, robot sensors, and the native control interface. It can inspect state, write or revise Python, and execute programs within the configured time and control-step limits.
Each series schedules five full trials, including those after an early success. Provider and infrastructure failures remain explicit incomplete slots. Tools, model parameters, and budgets are fixed; filesystem contents and conversation persist across fresh physical resets, without requiring the same container.
Program native robot controls
Python runs in an isolated code container. A restricted bridge exposes robot sensors, camera observations, and native control actions. The current development configuration limits each submitted program to 50 control steps and 90 seconds; each trial has 6,000 steps and 30 minutes of active time. Limits are recorded with the run.
Retain code, notes, and conversation
The codebase and notes persist across trials. SQLite preserves the interaction history. At the context limit, the active window slides and the agent can search archived turns; slide events are recorded.
Check task completion
The simulator evaluates the task’s success conditions over consecutive control steps. The harness then reruns the recorded code operations in a fresh simulator to check the outcome. An optional video audit compares those conditions with visible events across cameras.
Adaptive performance measurements
Metrics
Completion
Report verified success per trial. Trial-one completion is the baseline; completion by trial five counts tasks with at least one verified success across the series. Their difference is the descriptive adaptation gain. Report incomplete series and the denominator for each rate.
Consistency
Report the first successful trial and whether success persists through subsequent trials. Distinguish success from the outset, sustained failure-to-success conversion, and later regressions.
Efficiency
Compare control steps and simulation time across successful trials of the same task. Report the trajectory and consistent reductions; a short failed attempt receives no efficiency credit.
Recovery
Among tasks that fail trial one, report the fraction reaching verified success by trial five. Within-trial recoveries, such as retrieving a dropped object, remain inspectable in the trace; exploratory errors are not themselves a penalty.
These four dimensions define the evaluation; a composite score is not yet fixed. Definitions are versioned with the protocol. Aggregate reports must retain missing outcomes, state their denominators, and quantify uncertainty. Observed improvement is not a causal estimate of in-context learning.
The new five-trial release
Results
Inspect the published development series, their completed outcomes, and their remaining trials. Each task retains its five-trial denominator.
Loading published snapshot…Download results JSON ↗
The timestamp identifies this results snapshot. Running states come from saved job records. Incomplete trials and results awaiting checks remain unscored.
Reading trial receipts…
A scored outcome requires agreement between the original execution and the code-operation rerun. Provider failures, infrastructure failures, and pending checks are reported separately. These development series are not a ranked full-dataset leaderboard. Causal estimates of learning require matched fresh-context controls.
RoboCasa365 task selection
Task suite
The public prospective pilot contains 11 runnable task candidates. The broader catalogue contains 83 selections spanning 85 native tasks, including two proposed combinations. Scene admission and verifier review determine which tasks enter a run.
Proposed task combinations are labeled separately. Catalogue membership does not imply that every task or scene has passed admission.
Run the benchmark
Running evaluations
Build simulator images with Docker or Modal, admit the task scenes, and run the five-trial protocol. Named adapters support OpenAI, Anthropic, Google, and OpenRouter; each run records its model, context policy, settings, and budget. The pipeline saves code, traces, videos, and results.
Source and installation ↗THE PUBLIC CLI
# check scenes before model calls roboswe admit datasets/adaptive-pilot11 \ --config run.json --images images.json \ -e docker --out admitted.json # run the admitted five-trial series roboswe run admitted.json \ --config run.json --images images.json \ -e docker # use -e modal for Modal simulationInstallation, provider configuration & protocol ↗