ProjectsRobot policiesBenchmark

CALVIN-MAX

When does a mid-execution change hurt a robot policy?

A benchmark of 7,828 cases in which the world changes while a robot is working through a chain of instructions. Three runs per case, identical up to the change, separate the cost of a changed world from the effect of meeting it mid-motion.

The scene after an obstacle is inserted The original scene OriginalObstacle insertion
Steps after the change
Figure 1.One case per change, three runs that share every action up to the change. Base continues undisturbed; Pre meets the change at the start of the affected instruction; Mid meets it while the robot is moving. Frames are re-rendered from stored actions of held-out FLOWER cases, and the marks give each run's outcome on the affected instruction. The cases were chosen to illustrate the effect: for changes to what the robot sees, Mid completing while Pre fails; for physical changes, Mid failing. They are not typical outcomes.

Robots must often keep working after the world changes, as when a block moves or the camera shifts while the robot still acts on what it saw before. Recent work showed that one change during a task lowers success for every policy tested, but left open whether its timing matters and what it costs afterwards. CALVIN-MAX is a benchmark of 7,828 cases on CALVIN's five-instruction chains, each with one change to the objects, the camera, the robot or the instruction.

Each case compares three runs that share every action up to the change: one without it, one with it already in place when the affected instruction begins, and one in which it arrives while the robot is moving. On 1,526 held-out cases per policy, with hypotheses fixed in advance, each of ten released policies loses 4.4 to 9.9 points on the affected instruction. Timing matters only for some changes: a change to what the robot sees costs 5.4 points less mid-motion than from the start, whereas moving an object shows no such effect. The benefit comes from motion already under way, because it vanishes when the robot is rewound to the start of the instruction. The cost also outlasts the instruction: even a policy that completes it is 5.0 to 16.8 points less likely to complete the next one.

7,828

cases on 999 of CALVIN's evaluation chains, each fixed before any policy was run, in five tracks: core, phase, embodiment, intent and null.

Unseen scene D, trained on A, B, C
4.4–9.9

points lost on the affected instruction by every one of ten released policies once the world changes.

Held-out cases, Mid against Base
+5.4

points: a change to what the robot sees costs this much less mid-motion than from the start. Moving an object shows no effect of timing (−0.6).

Pooled over ten policies
5.0–16.8

points less likely to complete the next instruction, even after completing the affected one.

Every policy, Mid against Base

Three runs that differ only in when

A comparison with an undisturbed run mixes two opposing things. A policy hit mid-motion is committed to the old world: actions are queued, the arm is moving and its memory holds a scene that no longer exists. Yet it has also made progress, which may carry it to the goal regardless. A third run separates them.

  1. Base

    No change

    The chain runs undisturbed. One Base run per chain serves every case on that chain, and the onset of each change is computed from it with the simulator's privileged state, never from an outcome.

  2. Pre

    The changed world, from the start

    Replays Base to the start of the affected instruction and applies exactly the change Mid will receive, such as the same final pose of a moved block. Pre against Base is the cost of the changed world.

  3. Mid

    The change, while moving

    Replays Base to the onset, applies the change and hands control back with the policy's queued actions and memory intact. Pre and Mid end in the same world, so Mid against Pre is the effect of timing alone.

The changes

The eight core changes keep the names and purpose of LIBERO-MAX's, so the two benchmarks can be compared, and are grouped by how they reach the policy. CALVIN-MAX adds changes to the robot and to the instruction, two strengths per change, and onsets at six phases of the motion.

Figure 2.The scene before and after each change. Perceptual changes alter what the robot sees but not the task's geometry. Physical changes alter where the robot must go. Embodiment changes act on the robot itself. Not shown: a revised instruction, a null control that changes nothing, and a transient relocation that is undone a few steps later.

How it compares

BenchmarkChainOnlinePrefixPre / MidOnsetRobot and instructionBehaviourUnits
CALVIN✓✗✗✗✗✗✗1,000
LIBERO-Plus✗✗✗✗✗✗✗10,030
LIBERO-PRO✗✗✗✗✗✗✗10,000
LIBERO-MAX✗✓✓◐✗✗◐8,000
CALVIN-MAX✓✓✓✓✓✓✓7,828
Table 1.Chain: tasks of several instructions. Online: a change during execution. Prefix: identical actions up to the change. Pre / Mid: the same change before and during the motion. Onset: a varied moment of change. Robot and instruction: changes to the robot and to the instruction. Behaviour: measures of how the robot responds. ✓ tested, ◐ tested in part, ✗ not tested.

Every policy loses, but timing matters only for what the robot sees

Ten released policies span vision-language-action models, diffusion policies and models that predict from a history of frames. All lose once the world changes. For a perceptual change the timing effect is positive for every policy: arriving mid-motion costs less. For a physical change there is no detectable effect of timing. The difference between the two, the pre-registered primary test, is +6.0 points [+4.1, +7.9].

Three panels for ten policies and their pool. (a) Success on the affected instruction for Base, Pre and Mid: Mid and Pre below Base for every policy. (b) The timing effect Mid minus Pre, positive for perceptual changes and near zero for physical ones. (c) The gap between the two, positive for every policy.
Figure 3.(a) Success on the affected instruction over the eight core changes: Base (open circles), Pre (red squares) and Mid (blue diamonds). (b) The timing effect, Mid − Pre, for perceptual and physical changes. (c) The gap between the two, with 95% intervals. Held-out cases, per policy and for all ten together.

Explore one policy at a time

Timing, perceptual
Timing, physical
Difference
Figure 4.Success on the affected instruction for each core change, on the held-out cases of the chosen policy (142 to 180 cases per change). The readout gives the timing effect, Mid − Pre, pooled within each channel, with 95% intervals that resample whole chains, and the difference between the channels. Hover or focus a row for its exact numbers.

Which changes cost the most

Sensor noise (a median of −17.6 points across policies) and target relocation (−12.3) cost the most when present from the start; relocating the receptacle (−2.1) and shifting the camera (−3.8) the least. On timing the two channels separate cleanly: every physical change has a timing effect at or below zero, while the perceptual ones mostly gain when they arrive mid-motion.

Per change, the cost of the changed world (Pre minus Base) and the timing effect (Mid minus Pre) for each of ten policies. Physical changes have timing effects at or below zero; perceptual changes mostly gain mid-motion.
Figure 5.The cost of the changed world (Pre − Base, left) and the effect of timing (Mid − Pre, right) for each core change on the held-out cases. Each open circle is one policy; the diamond, shading and thin line give the median, middle half and full range.

The benefit comes from motion already under way

Why would a change hurt less when it arrives mid-motion? Two diagnostic runs test it. One rewinds the arm and the policy's memory to the start of the instruction while leaving the world changed. Mid beats the rewound run for every policy, so rewinding removes the benefit. What remains, the rewound run against Pre, is within ±3 points for three policies, and the steps lost in rewinding explain most of the remaining gap.

(a) Mid compared with Pre and with a rewound run for each policy: Mid beats the rewound run for every policy. (b) The rewound run minus Pre, inside a band of plus or minus 3 points for three policies.
Figure 6.(a) Mid against Pre and against a rewound run, for perceptual changes, with 95% intervals. (b) What rewinding leaves, the rewound run minus Pre, as run and with Pre given the same remaining steps, with 90% intervals; a check marks each policy inside the shaded ±3 band.

Perceptual changes cost only before contact

The same change at six moments of the motion, from the start of the instruction to placing the object, for three policies. Before contact, a corrupted view costs up to 20 points; after it, almost nothing.

Mid minus Base at six onsets for FLOWER, MoDE and X-VLA: losses of up to 20 points at the start of the instruction, near zero after contact.
Figure 7.Mid − Base at six onsets, 100 cases each, with 95% intervals that resample whole chains. Onsets differ in task mix, so the profile is descriptive.

The more a change hurts, the more timing helps

Five strengths of each change, from a quarter to four times the default. Within a policy, the benefit of a mid-motion arrival grows with the damage the change does, and a change that does not hurt gives no benefit.

For each policy and change, the mid-motion benefit plotted against the loss when present from the start, across five strengths: the benefit grows with the loss.
Figure 8.Mid − Pre against Pre − Base at five strengths (larger markers are stronger), 51 to 60 cases per level. Grey: a change that does not hurt the policy.

Under heavy sensor noise, some policies recover most of the loss

An exploratory stress test raises sensor noise to four times its default strength. A mid-motion arrival recovers half to two thirds of the loss for five policies, and about a tenth for four others.

Success under four-times sensor noise for Base, Pre and Mid for ten policies, with the share of the Pre loss that Mid recovers: 53 to 67 percent for five policies, 8 to 13 percent for four.
Figure 9.Success on the affected instruction under sensor noise at four times the default strength (development cases, exploratory). The column gives the share of the Pre loss that Mid recovers; GR-1 loses too little for a share.

The cost outlasts the instruction

Because CALVIN's tasks are chains of five instructions, we can measure what a change costs after it has been handled. The change stays in place, and even a policy that completes the affected instruction is less likely to complete the next one, by about the same amount whether the change arrived mid-motion or was there from the start.

For each of ten policies, success on the next instruction among runs that solved the affected instruction, for Base and Mid. Mid is lower for every policy, by 5.0 to 16.8 points.
Figure 10.Success on the next instruction, among cases where the affected instruction was solved in both runs, for Base (open circles) and Mid (blue diamonds), with the difference and odds ratios for the next instruction and the rest of the chain.

Earlier changes cost more

A change at the first instruction costs 13.2 points from the start, at the fifth 6.0. The perceptual timing benefit shrinks too, from +5.2 to +2.7, and a change at the first instruction shortens the run by about 0.8 instructions on average.

Four panels by position of the change in the chain: the configuration cost shrinks from about minus 13 to minus 6, the perceptual timing benefit from about 5 to 3, the physical timing effect stays near zero, and the chain-length loss shrinks toward zero.
Figure 11.Effects by the position k of the affected instruction. Thin lines: one policy each; thick: the mean over ten. (a) Pre − Base; (b, c) Mid − Pre, perceptual and physical; (d) chain length, Mid − Base, in instructions.

Most successes survive, but the share lost varies fivefold

Of the runs a policy completes without a change, 83 to 95% still succeed with one. The share that flips to failure ranges from 5% for FLOWER to 26% for 3D Diffuser Actor.

Per policy: retained ratio from 83 to 95 percent, conditional regression from 5.4 to 25.7 percent, and trigger and response coverage.
Figure 12.(a) Mid success over Base success. (b) The share of Base successes that fail in Mid. (c) The share of cases in which the change was realized (filled) and in which the policy was queried after it (hollow).

Policies

FamilyPolicies
Vision-language-actionFLOWER, X-VLA, π0.5, UniVLA, OpenVLA-OFT
DiffusionMoDE, 3D Diffuser Actor
Predictive, from a history of framesGR-1, Seer-Large
VLM with LSTMRoboFlamingo
Table 2.The ten released policies evaluated, on CALVIN ABC→D, each served behind a small adapter in the repository.

Citation

Please cite this work as

@misc{pakdamansavoji2026calvinmax,
  title        = {{CALVIN-MAX}: When Does a Mid-Execution Change Hurt a Robot Policy?},
  author       = {Pakdamansavoji, Sajjad},
  year         = {2026},
  howpublished = {\url{https://github.com/forall-labs/CALVIN-MAX}}
}

Reproducing

The case manifests, runners, policy adapters and the analysis outputs behind every number are in the GitHub repository. The hypotheses and analysis plan were fixed on 672 development cases before the held-out cases were run; see the pre-registration.

Built on

CALVIN, following the design of LIBERO-MAX, evaluated on the ten policies above.