ProjectsRobot codeOpen-weight models

OpenCaP

How open-weight models fail at writing robot code, and the smallest fix for each failure

Twenty-one open-weight models write programs that control a robot. For every program that fails, we find the smallest edit that makes it work, keeping the model's own code, and record what that edit had to fix.

cube_lifting.pyQwen2.5-Coder 7BS3
obs = get_observation()rgb = obs["robot0_robotview"]["images"]["rgb"]intrinsics = obs["robot0_robotview"]["intrinsics"]masks = segment_sam3_text_prompt(rgb, text_prompt="red cube")best_mask = max(masks, key=lambda d: d["score"])depth = obs["robot0_robotview"]["images"]["depth"]grasp_poses, grasp_scores = plan_grasp(depth, intrinsics, best_mask["mask"])best_idx = grasp_scores.argmax()best_T = grasp_poses[best_idx]  # (4, 4)move_to_joints(solve_ik(best_T[:3, 3], best_T[:3, :3].T @ np.array([0, 0, 1])))close_gripper()move_to_joints(solve_ik(best_T[:3, 3] + np.array([0, 0, 0.1]), best_T[:3, :3].T @ np.array([0, 0, 1])))open_gripper()

Language models can write Code-as-Policies programs that drive a robot, and success rate is how they are usually compared. Success rate says how often a program fails but not why. OpenCaP takes every failing program from a grid of 21 open-weight models, four manipulation tasks in CaP-X and three levels of robot API, repairs each with the smallest edit that makes it work, and tags each cause the repair had to fix.

The level of the API decides what goes wrong. With high-level helpers, failures are mostly not about robotics: an uncalled function, a shadowed primitive, a missing final action, the wrong object. Remove the helpers, leaving only perception, inverse kinematics and the gripper, and coordinate-frame errors jump from 4% to 47% of failing programs, misread API calls from 26% to 60%, and the larger models lose their advantage: their failing programs carry about twice as many defects as before.

4% → 47%

of failing programs carry a coordinate-frame error once the high-level helpers are removed: camera values used as world coordinates, wxyz and xyzw mixed up.

FRAME, S1 against S3/S4
26% → 60%

carry an interface error: a misread return value or argument, a wrong prompt, an unbound name.

INTERFACE, S1 against S3/S4
2×

defects per failing program for the 12B and larger models at S3/S4 compared with S1, 3.33 against 1.54. It holds for all 15 models of 3B and up.

Mann-Whitney p = 0.0002
7,560

trials: 21 models, 4 tasks, 3 API levels, 30 fixed scenes each, with greedy decoding.

Qwen3, Qwen3-VL, Qwen3.5, Gemma 3

How OpenCaP works

Each failing program is treated as evidence of a specific mistake, not as a zero. The repair is the measurement: the smaller the edit that makes a program work, the closer the model was, and the edit itself names what was wrong.

  1. Step 01

    Run the grid

    Every model writes one program per task and API level, run on 30 fixed scenes. S1 offers helpers such as get_object_pose and goto_pose. S3 keeps only perception (SAM3, Molmo, Contact-GraspNet), solve_ik, move_to_joints and the gripper. S4 also removes the worked example.

  2. Step 02

    Repair minimally

    Each failing program gets the smallest edit that makes it pass, keeping the model's code. A repair is accepted when no scene fails in all k draws, with k calibrated per task on the task's own reference program. Workarounds any correct program would need are shared and never counted as defects.

  3. Step 03

    Tag the cause

    Each repair gets one tag per distinct cause, in six families: syntax, structure, interface, semantic, trajectory and frame. 70 failing programs at S1 and 169 at S3/S4 are tagged, 116 of them against a verified repair.

Lowering the API changes what fails

At S1 the API returns world poses, so frame errors almost never happen. At S3 and S4 the model must turn camera-frame detections into world coordinates itself, and that is where it breaks. The rise holds on every task, and with unverified records removed.

Figure 1.Share of failing programs with at least one defect in each family, with Wilson 95% intervals. Frame and interface errors rise sharply once the helpers are gone. Trajectory falls because S1's approach-height argument, the source of most S1 trajectory tags, does not exist at S3/S4: that reflects the API, not the models. Semantic slips are near-universal at every level. Choose a family to read its programs in Figure 3.
Table 1.Share of failing programs with at least one tag in each family, with Wilson 95% intervals and Fisher's exact test between S1 and S3/S4. The verified-only column keeps the 116 programs tagged against a verified minimal repair.
FamilyS1 (n = 70)S3/S4 (n = 154)Verified only (n = 116)Fisher p
Frame4% [1, 12]47% [40, 55]48%≈1e-11
Interface26% [17, 37]60% [52, 67]52%≈3e-6
Trajectory47% [36, 59]19% [14, 26]23%≈5e-5
Structure29% [19, 40]35% [28, 43]33%n.s.
Semantic94% [86, 98]95% [90, 97]93%1.0

The scale advantage disappears

With helpers, a larger model's failing programs are cleaner: defects per failing program fall from 4.2 at 2B and below to 1.5 at 7B and above. Without them, every size band sits between 3.2 and 4.6. Frame errors appear for 19 of the 21 models at S3/S4, against 3 of 21 at S1; the two exceptions are the smallest models, whose programs fail before reaching any coordinate math.

Figure 2.Mean defect tags per failing program, by model size and API level. At S1 the count falls with scale; at S3 and S4 it does not. With an automated tagger instead of the filed tags, the S1 row reads 3.21 to 1.15, the same direction.

Read the failing programs

Every number above counts tags on real programs, and every one of them is here: the 70 failing programs at S1 and the 152 at S3 and S4 that carry a tag, each with the causes its repair had to fix. Choose a program to read it. Pointing at a cause marks the names its evidence mentions in the code.

Loading the programs

    Figure 3.The failing programs behind Figures 1 and 2, as each model wrote them, with the tags filed against each one's minimal repair. The marks show where a name the evidence mentions appears in the code; a missing action, for one, has nothing to mark. Programs longer than 90 lines are cut here and complete in the repository.

    Checks and limits

    The token cap does not explain the failures. Raising it from 2,048 to 8,192 tokens fixes no failure of a 12B or larger model: two models loop on one line at any budget, and another completes and still crashes. No program cheats. The observation also exposes object poses, and a scan of all 5,040 S3/S4 episodes found no program reading them.

    The tags come from language models, not people. Claude Opus 5 tagged every program against its repair. An independent rater from a different model family, GPT-6 Astra, blind to the question, re-tagged a 60-program sample: per-family agreement is κ 0.82 for frame and 0.87 for interface, and both contrasts reproduce in its own labels. The size of the defect-count gap moves between raters, so the family contrasts are the stronger result. No human has rated the packet yet; it ships with the code, ready for one.

    These are failing programs only: the counts describe what goes wrong, not how often programs fail.

    Citation

    Please cite this work as

    @misc{pakdamansavoji2026opencap,
      title        = {OpenCaP: How Open-Weight Models Fail at Writing Robot Code},
      author       = {Pakdamansavoji, Sajjad},
      year         = {2026},
      howpublished = {\url{https://github.com/forall-labs/OpenCaP}}
    }

    Reproducing

    Every number on this page is regenerated on a CPU from the tags and repairs in the GitHub repository, for example python scripts/cross_tier_report.py --no-truncated. The method, all tables and the limitations are in the write-up.

    Built on

    CaP-X, with models served by vLLM and perception from SAM3, Molmo and Contact-GraspNet.