ProjectsRobot codeOpen-weight models
OpenCaP
How open-weight models fail at writing robot code, and the smallest fix for each failure
Twenty-one open-weight models write programs that control a robot. For every program that fails, we find the smallest edit that makes it work, keeping the model's own code, and record what that edit had to fix.
obs = get_observation()rgb = obs["robot0_robotview"]["images"]["rgb"]intrinsics = obs["robot0_robotview"]["intrinsics"]masks = segment_sam3_text_prompt(rgb, text_prompt="red cube")best_mask = max(masks, key=lambda d: d["score"])depth = obs["robot0_robotview"]["images"]["depth"]grasp_poses, grasp_scores = plan_grasp(depth, intrinsics, best_mask["mask"])best_idx = grasp_scores.argmax()best_T = grasp_poses[best_idx] # (4, 4)move_to_joints(solve_ik(best_T[:3, 3], best_T[:3, :3].T @ np.array([0, 0, 1])))close_gripper()move_to_joints(solve_ik(best_T[:3, 3] + np.array([0, 0, 0.1]), best_T[:3, :3].T @ np.array([0, 0, 1])))open_gripper()
Language models can write Code-as-Policies programs that drive a robot, and success rate is how they are usually compared. Success rate says how often a program fails but not why. OpenCaP takes every failing program from a grid of 21 open-weight models, four manipulation tasks in CaP-X and three levels of robot API, repairs each with the smallest edit that makes it work, and tags each cause the repair had to fix.
The level of the API decides what goes wrong. With high-level helpers, failures are mostly not about robotics: an uncalled function, a shadowed primitive, a missing final action, the wrong object. Remove the helpers, leaving only perception, inverse kinematics and the gripper, and coordinate-frame errors jump from 4% to 47% of failing programs, misread API calls from 26% to 60%, and the larger models lose their advantage: their failing programs carry about twice as many defects as before.
of failing programs carry a coordinate-frame error once the high-level helpers are removed: camera values used as world coordinates, wxyz and xyzw mixed up.
FRAME, S1 against S3/S4carry an interface error: a misread return value or argument, a wrong prompt, an unbound name.
INTERFACE, S1 against S3/S4defects per failing program for the 12B and larger models at S3/S4 compared with S1, 3.33 against 1.54. It holds for all 15 models of 3B and up.
Mann-Whitney p = 0.0002trials: 21 models, 4 tasks, 3 API levels, 30 fixed scenes each, with greedy decoding.
Qwen3, Qwen3-VL, Qwen3.5, Gemma 3How OpenCaP works
Each failing program is treated as evidence of a specific mistake, not as a zero. The repair is the measurement: the smaller the edit that makes a program work, the closer the model was, and the edit itself names what was wrong.
- Step 01
Run the grid
Every model writes one program per task and API level, run on 30 fixed scenes. S1 offers helpers such as
get_object_poseandgoto_pose. S3 keeps only perception (SAM3, Molmo, Contact-GraspNet),solve_ik,move_to_jointsand the gripper. S4 also removes the worked example. - Step 02
Repair minimally
Each failing program gets the smallest edit that makes it pass, keeping the model's code. A repair is accepted when no scene fails in all k draws, with k calibrated per task on the task's own reference program. Workarounds any correct program would need are shared and never counted as defects.
- Step 03
Tag the cause
Each repair gets one tag per distinct cause, in six families: syntax, structure, interface, semantic, trajectory and frame. 70 failing programs at S1 and 169 at S3/S4 are tagged, 116 of them against a verified repair.
Lowering the API changes what fails
At S1 the API returns world poses, so frame errors almost never happen. At S3 and S4 the model must turn camera-frame detections into world coordinates itself, and that is where it breaks. The rise holds on every task, and with unverified records removed.
| Family | S1 (n = 70) | S3/S4 (n = 154) | Verified only (n = 116) | Fisher p |
|---|---|---|---|---|
| Frame | 4% [1, 12] | 47% [40, 55] | 48% | ≈1e-11 |
| Interface | 26% [17, 37] | 60% [52, 67] | 52% | ≈3e-6 |
| Trajectory | 47% [36, 59] | 19% [14, 26] | 23% | ≈5e-5 |
| Structure | 29% [19, 40] | 35% [28, 43] | 33% | n.s. |
| Semantic | 94% [86, 98] | 95% [90, 97] | 93% | 1.0 |
The scale advantage disappears
With helpers, a larger model's failing programs are cleaner: defects per failing program fall from 4.2 at 2B and below to 1.5 at 7B and above. Without them, every size band sits between 3.2 and 4.6. Frame errors appear for 19 of the 21 models at S3/S4, against 3 of 21 at S1; the two exceptions are the smallest models, whose programs fail before reaching any coordinate math.
Read the failing programs
Every number above counts tags on real programs, and every one of them is here: the 70 failing programs at S1 and the 152 at S3 and S4 that carry a tag, each with the causes its repair had to fix. Choose a program to read it. Pointing at a cause marks the names its evidence mentions in the code.
Loading the programs
Checks and limits
The token cap does not explain the failures. Raising it from 2,048 to 8,192 tokens fixes no failure of a 12B or larger model: two models loop on one line at any budget, and another completes and still crashes. No program cheats. The observation also exposes object poses, and a scan of all 5,040 S3/S4 episodes found no program reading them.
The tags come from language models, not people. Claude Opus 5 tagged every program against its repair. An independent rater from a different model family, GPT-6 Astra, blind to the question, re-tagged a 60-program sample: per-family agreement is κ 0.82 for frame and 0.87 for interface, and both contrasts reproduce in its own labels. The size of the defect-count gap moves between raters, so the family contrasts are the stronger result. No human has rated the packet yet; it ships with the code, ready for one.
These are failing programs only: the counts describe what goes wrong, not how often programs fail.