DRAFT PREPRINT · METHODS

Background and methods draft. Evaluation results will be added after the trial series are complete. Not yet submitted to arXiv.

Introduction

A coding agent controlling a robot must connect several kinds of reasoning: interpreting visual observations, choosing a sequence of actions, expressing those actions as executable programs, and revising its approach when the world does not behave as expected. A dropped object or missed grasp is both a failure of the current action and useful evidence for the next one. Repeated attempts make this development process observable.

RoboSWE-ADAPT asks a focused question: how effectively can an agent develop a reusable action program for a household task over five trials? The physical scene resets between trials, while the agent retains its program library and conversation. Model parameters remain fixed. This setting measures adaptation of the agent system, including its code and available experience, rather than weight training.

The benchmark builds on RoboCasa365’s household environments and success predicates. Its contribution is an evaluation protocol and execution harness for coding agents: a controlled boundary between authored code and trusted simulation, a retained-development sequence, reproducible verification, and an inspectable record of decisions and physical actions. The intended release includes runnable task manifests, a command-line pipeline, and a browser-based evidence viewer. The current paper describes these methods; evaluation outcomes will be added only after the new protocol has produced complete runs.

Background and Related Work

Simulation and household manipulation.

RoboCasa provides simulated household environments and demonstrations for generalist robot learning [1]. RoboCasa365 extends this foundation to 365 tasks, including 65 atomic and 300 composite tasks [2]. RoboSWE-ADAPT uses selected task implementations from this ecosystem, retaining their provenance. The underlying robosuite framework provides modular robot models and controller interfaces on MuJoCo [3]. We evaluate program development above that interface, rather than introduce a new physics engine or claim a replacement for RoboCasa’s policy-learning evaluations.

Programs as embodied policies.

Code as Policies demonstrates language-model-generated programs that compose perception and control APIs [4]. Voyager develops an executable skill library through interaction and feedback in Minecraft [5]. These works motivate treating authored programs as inspectable, reusable products of interaction. RoboSWE-ADAPT concentrates on repeated execution of a fixed household task with a controlled physical reset and bounded budgets.

Learning from feedback without weight updates.

Reflexion uses feedback and retained linguistic reflection to improve subsequent trials [6]. Our protocol preserves interaction history and code, allowing participants to use available experience without requiring a particular reflection method. Since multiple retained resources change together, improvements in this setting alone do not identify a causal effect of conversational memory. A matched fresh-context control would be required to isolate that contribution.

Executable agent evaluation.

Terminal-Bench evaluates agents on tasks in executable terminal environments [7]. It is an inspiration for a cohesive task runner and inspectable trajectories. Here, program execution also changes a physical simulation, making camera evidence, control-step accounting, and task-state verification part of the evaluation record.

Task Catalogue and Admission

The starting catalogue contains 83 user-selected entries representing 85 native task names. Two entries propose combining GatherMarinadeIngredients with BlendMarinade, and ReheatMeal with ReturnHeatedFood. These compositions are candidates, not yet admitted custom environments. A composed task must preserve one continuous scene and receive its own identifier, instruction, budgets, and verifier review.

Selection used the official composite-task catalogue’s subtask counts, skill labels, demonstration durations, and videos [8]. The column labeled Horizon displays rounded median demonstration duration in seconds, drawn from episode_len_medians_train.json; it is neither the registry’s maximum control-step horizon nor a model wall-time limit [9]. The harder candidates generally have at least six annotated subtasks, more than three listed skills, and longer median demonstration durations. Approximately twenty shorter candidates have two to five subtasks and median demonstration durations below sixty seconds, typically thirty to fifty seconds. These criteria describe purposeful curation rather than strict eligibility thresholds or empirical difficulty labels. Examples include beverage sorting, dishwasher loading, appliance operation, food arrangement, and refrigerator organization.

Admission precedes release scoring. For each scene, the manifest must pin the upstream revision, task class, robot, layout, style, seed, controller configuration, observations, and execution limits. Checks establish that the scene loads, the initial state is valid and does not already satisfy success, the interface operates within the chosen budgets, and resets match the reference fingerprint. Predicate review checks whether the implemented condition matches the task description. Median demonstration duration, registry control-step horizon, configured control-step budget, and API wall-time budget are recorded as separate quantities.

Candidates with reproducible environment or operational defects may be excluded during development. The release manifest will retain exclusion reasons and the resulting admitted denominator. Exclusion must not depend on a particular model failing a legitimate task. Once a comparative evaluation starts, the admitted task set and settings are frozen; subsequent repairs produce a versioned protocol rather than retroactively changing scores. Earlier exploratory campaigns remain separate from this release.

Five-Trial Adaptive Protocol

An evaluation unit is a task-scene series of $T=5$ attempts. Attempt one starts with the standard interface instructions and an empty authored motion library. Subsequent attempts preserve the model’s available interaction history and authored code. Each attempt starts a fresh simulator, and its initial fingerprint must match the admitted reference before new model input is supplied. A fresh task starts a fresh development history; the protocol does not transfer task-specific libraries between benchmark tasks.

The same model pin, observation interface, controller, context policy, and budgets apply throughout a series. The experimental runner currently supports explicit budgets for active work, total control steps, and individual scripts. The remote pilot uses 1,800 active seconds, 6,000 control steps, and 50 steps per script, with a 90-second script limit. Earlier local development runs used a 60-second script limit and are identified separately. These are harness budgets, not assertions about upstream task horizons. Release settings are pinned per manifest before comparative runs.

A control step advances the configured controller tick; it is not necessarily one physics integrator step. Observing the scene does not advance physics. The simulator pauses between requested actions while the model reasons. Consequently, we report simulation/control progress separately from provider latency, code execution, transport, passive waits, replay compute, and end-to-end elapsed time. A successful action program is not by itself evidence of real-time physical deployment.

Failures remain part of the adaptive sequence. A completed unsuccessful attempt is not rerun to improve its outcome. Infrastructure failures and missing attempts are recorded explicitly, with any recovery linked to the original record. Successful and unsuccessful attempts both provide experience to the next scheduled trial.

Harness and Action Interface

The trusted coordinator packages instructions, camera images, sensor observations, and operation feedback, then asks the participant for a structured decision. Operations include authoring or reading files, executing a script, observing, waiting, and finishing. Authored Python runs in a networkless Docker sandbox and accesses the robot through a restricted bridge. The model receives documented native action dimensions, bounds, and coordinate conventions rather than prewritten task-solving primitives.

The simulator owns physical state and task evaluation. The authored program does not receive a writable simulator object, an administrative reset operation, provider credentials, or authority to set grading results. Sensor and image observations are selected by the harness; unrestricted internal task state is not passed through as a raw environment dump. Script stdout, errors, completed-step counts, and timing are recorded as distinct feedback. A successful code return or accepted control tick does not establish grasp, target attainment, or task completion.

The public runner accepts a dataset path and model/execution configuration through roboswe run, with Docker or Modal selected using -e. In the implemented architecture, Modal hosts the remote simulator while the coordinator and authored-code sandbox remain on the client host. This preserves the same code trust boundary across backends. Backend, image identity, hardware allocation, and controller frequency belong in the run manifest because they can affect timing and reproducibility.

Exact History and Context Windows

A coordinator-owned SQLite store archives exact provider message blocks with digests, timestamps, and attempt identifiers. Provider-exposed thinking and opaque replay fields are retained when available; opaque fields are not presented as readable reasoning. The viewer can align public reasoning, tool calls, execution output, and robot actions without manufacturing missing internal content.

The context policy reserves output capacity and request overhead before inference. For context capacity $C$, output reserve $O$, and safety allowance $S$, the input budget is $B=C-O-S$; the current allowance is $S=4096$ tokens. Request accounting includes instructions, schema, images, and conversational content. When necessary, the policy removes the oldest complete exchanges until at least 200,000 input accounting units have been removed, or more if required to fit. Native-count policies use token counts; the experimental OpenRouter policy uses the conservative proxy described below. Calls and results remain paired, and system instructions remain pinned. An oversized indivisible exchange is an error rather than a silently truncated record.

Sliding removes content from the active request, not the archive. A bounded literal-search operation becomes available after the first slide and returns excerpts from prior history. Slide events record their attempt, time, removed messages, and before/after accounting values and the counting method. No provider-generated summary replaces archived experience. Direct-provider adapters use native count endpoints with the actual request format. For Gemini through OpenRouter, the current policy maps the original text, images, response schema, and returned reasoning metadata into a Google count-only request. The input estimate starts at $\lceil 1.30 N \rceil + 8192$, where $N$ is the mapped native count. After a response, the multiplier may increase to $1.1 U/N$, where $U$ is the reported prompt usage; it never decreases within the series. Generation sends the original OpenRouter messages unchanged. This calibrated estimate is recorded separately from exact native counts because OpenRouter may transform the request internally. Earlier byte-based estimates are retained only in separately labeled development evidence. An overflow stops the run and preserves evidence rather than silently retrying. Because this policy can slide earlier than a native-count policy, provider comparisons report the policy and slide history alongside outcomes. A common archive and sliding rule does not guarantee equal retained context or cache discounts across providers.

Verification and Evidence

Mechanical evaluation.

The trusted simulator evaluates the upstream task predicate after control actions. Success requires a valid non-successful initial state and the configured consecutive success interval. The report preserves the observed predicate outcome and duration. Instructions and predicates are reviewed together because a reproducible predicate can still be an incomplete description of the intended behavior.

Independent replay.

The adaptive pipeline reruns recorded code operations from the complete starting workspace in a fresh workspace and simulator. Checkpoints preserve regular files, including generated binary data, with content hashes, permissions, and modification timestamps. Restoration validates the manifest before writing into an empty workspace. Unsupported file types and paths fail checkpoint validation explicitly. The same immutable actor image is used for execution and replay; trusted checkpoint maintenance runs separately from authored programs. Verification records execution agreement and reproduced success independently of the participant’s claim. Errors and mismatches remain distinct from a verified pass. Replay tests reproducibility of the execution; semantic predicate quality is addressed by task review and negative controls.

Optional visual cross-check.

A separate judge may inspect synchronized camera views against a predicate-specific description and visual evidence guide. It reports whether each observable condition is supported, contradicted, or unobservable. Mechanical outcomes are compared after the visual assessment rather than supplied as the answer to reproduce. A contradiction prompts review, while an unobservable condition remains unresolved. This audit is optional and supplementary, not the mechanical pass gate. Its model, prompt, input views, response, and cost are recorded independently.

Threat model and records.

The boundary protects grading state from ordinary authored-code modification and ignores self-reported reward values. Negative tests should cover forged administrative operations, fake reports, and violations of the task’s physical conditions. Published evidence includes code snapshots, model messages, controller traces, verification records, and videos. Remote artifact durability requires readback against recorded hashes before local eviction; a stored path alone is not a durability receipt.

Proposed Evaluation Measures

The following measurement plan separates reliable completion, adaptation, and efficiency. It is proposed for release review rather than presented as an already frozen composite leaderboard score.

For a complete valid series, let $y_t\in\{0,1\}$ denote verified success on attempt $t$. Mean success $A=T^{-1}\sum_{t=1}^{T}y_t$ rewards consistently correct behavior. A stable-acquisition score is $$Q=\frac{1}{T}\sum_{t=1}^{T}\prod_{j=t}^{T}y_j,$$ which equals the final uninterrupted success-streak length divided by five. Thus success on every attempt scores one, whereas two failures followed by three successes score $3/5$. Reporting $A$ alongside $Q$ preserves distinctions between early successes and late regressions. Missing or invalid attempts are reported separately, not imputed as successful or silently removed.

Control steps and simulated time are compared among successful attempts, including paired first-to-last ratios and complete trajectories. Fewer steps on a failed attempt do not constitute efficiency improvement. A sustained-improvement bonus, if adopted, will require a preregistered multi-attempt criterion rather than a single large early reduction; its thresholds are left open for review before release scoring.

Within-attempt recovery is a separate proposed diagnostic. Independently identified mishaps can be paired with regained task progress and recovery time or steps. Unrecovered events are censored at the attempt boundary. Raw exception counts are not a headline penalty: exploratory mistakes can support later success, and deliberately producing extra mishaps must not earn extra credit. Agent explanations help interpret a trace but do not alone prove physical recovery or accurate attribution.

Aggregation uses task-level paired comparisons so that five dependent attempts are not treated as five independent tasks. Reports should include admitted and completed denominators, task-level uncertainty, operational failures, and disaggregated timing and cost. Claims that retention caused improvement require matched fresh-context controls; the initial retained-trial benchmark measures observed adaptation of the complete system.

Release and Reproducibility Plan

The release package will connect each admitted task to its scene configuration, instruction, upstream predicate revision, observation contract, budgets, and replay settings. A single run command produces the trial series and verification records; the dashboard presents multi-angle videos with aligned code and available reasoning. Static exports support inspection without credentials or a running simulator.

The candidate catalogue and experimental execution pipeline are available in the development repository. Scene admission, provider validation, and the final scoring specification are being completed for RoboSWE-ADAPT. The paper will report the new protocol’s empirical evaluation after runs finish; historical pilot outcomes are not substituted for those measurements.