Execution-Aware VLAs for Robust Manipulation

Sombit Dey, Jan-Nico Zaech, Luc Van Gool, Danda Pani Paudel

WRL @ NeurIPS 2026

The video submitted with the paper. It runs 2.5 minutes and has no sound.

TL;DR: Robots never execute exactly what a VLA commands, and how far off they are depends on the controller. We feed the policy its previous commands together with the robot's measured state and finetune it with RL under randomized controllers, so that it learns to correct for the difference.

Abstract

Pretrained vision-language-action (VLA) models typically assume particular execution dynamics, which limits their robustness when deployed with different robot controllers. In practice, physical robots exhibit tracking error, latency, actuator saturation, compliance effects, calibration inaccuracies and embodiment-specific dynamics, causing discrepancies between the commanded policy actions and the realized robot motion. We refer to this discrepancy as the execution gap. To address it, we propose Execution-Aware Vision-Language-Action (EA-VLA), an approach for adapting pretrained VLA models to varying execution dynamics. EA-VLA augments the robot policy with execution-state feedback, enabling the policy to reason about discrepancies between commanded and executed behavior during action generation. By conditioning on robot proprioception and execution-state information, the approach learns to compensate for the execution gap and tracking error. We evaluate EA-VLA in simulation and on real robots. In simulation, EA-VLA maintains performance across much larger perturbation magnitudes than baseline VLAs tolerate. On real robots it is robust to the underlying controller on multiple embodiments, with an 11% higher success rate on WidowX and up to 50% higher throughput on Franka. These results show that execution-state feedback can substantially improve the robustness of pretrained VLAs to variations in how their actions are executed.

The execution gap

A VLA outputs a chunk of target poses, and a low-level controller tries to reach them. On a real robot it never quite does: tracking error, latency, actuator limits, compliance and calibration all make the realized motion differ from the command. On a WidowX, the commanded and executed end-effector paths differ by up to 2 cm within an episode. On a Franka with a compliant controller, they can differ by more than 10 cm. That is small next to the workspace, but large next to the precision needed to grasp or place an object, and an imitation policy that ignores it lets the error compound.

Pretrained VLAs are trained on data from many robots and controllers, so at deployment they often meet execution behavior they have not seen. Retuning the controller for every robot and task is one fix, but there is no single best setting. A stiff controller tracks well and is a poor fit for contact. A compliant one is safer and tracks worse. We ask instead whether the policy can adapt to whichever controller it runs on.

Commanded versus executed end-effector position over time on Franka and WidowX, with the gap shaded.
Commanded actions (red, dashed) against the realized end-effector motion (black), with the gap shaded. (a) Franka under a compliant controller, where the deviation reaches 11 cm. (b) WidowX during a grasp approach, with an average offset of 13 mm.
Top: a naive VLA assumes commanded actions equal executed actions, and the executed path drifts away from the commanded one. Bottom: EA-VLA receives execution feedback from the robot and issues compensated actions that keep the executed path on target.
Top: a standard VLA plans as if the robot executes every command exactly. The robot does not, and the difference between the commanded and the executed path, the execution gap, grows over the episode. Bottom: EA-VLA is told what it commanded and where the robot actually went, and adjusts its next actions to compensate.

Method: show the policy what actually happened

EA-VLA keeps the pretrained VLA and adds one input. At every step, a small two-layer MLP φ encodes the previously commanded action together with the current and previous robot state into an execution-state token:

et = φ(acmdt−1, st, st−1)

The policy receives the last H = 2 of these tokens next to its usual camera images, instruction and proprioception. The tokens go only to the action expert, the flow-matching head that generates the actions. The encoder gets the raw commands and states rather than a precomputed residual, which we found to train more stably.

We then finetune the action expert with PPO in simulation, using a value head on the VLM features and a reward of 0.5 for grasping the target and 0.5 for completing the task. In every episode we randomize the stiffness, damping and friction of the PD controller by a factor k′ = k·exp(σε), with ε ~ U(−1, 1). The policy is never told these parameters. It has to infer them from how its own commands were carried out.

EA-VLA architecture: VLM, action expert with execution feedback from the robot, and the RL training loop with a critic head and policy gradient.
The VLM encodes the images, the instruction and the robot state. The action expert also receives execution feedback from the robot and denoises a chunk of actions. During RL training, a critic head on the VLM estimates value from the rollout rewards, and the action expert is updated by policy gradient.

Changing the controller

We train and test a WidowX in ManiSkill, set up to match BridgeData V2, and make the controller progressively more different from the nominal one by raising σ. RL finetuning without execution feedback scores best with the nominal controller, 83.4%, but it has fitted that one controller: at σ = 0.1 it loses 32.3 points. EA-VLA gives up some nominal performance (77.5%) and holds 80.1% at σ = 0.1 and 60.4% at σ = 0.5, where both baselines are below 15%. At σ = 1, the level it was trained at, it reaches 28.6%, more than three times the RL baseline.

trained at 100%75%50%25%0% 00.10.512 Controller randomization σ 77.5 80.1 60.4 28.6 6.1 83.4 EA-VLA (ours) + RL, no execution feedback SFT baseline
Success rate on simulated WidowX pick-and-place as the controller's stiffness, damping and friction are randomized further from their nominal values (σ = 0). SFT baseline: the supervised-finetuned VLA. + RL: the same RL finetuning as EA-VLA, without the execution-state input. The shaded column is the randomization level EA-VLA was trained at; σ = 2 is beyond it.
Table and history-length ablation

Success rate by randomization level

Modelσ = 00.10.512
SFT baseline48.7%49.7%14.2%4.3%2.9%
+ RL83.4%51.1%11.2%8.1%4.4%
EA-VLA77.5%80.1%60.4%28.6%6.1%

Length of the execution-state history H

Historyσ = 00.10.512
H = 083.4%51.1%11.2%8.1%4.4%
H = 170.3%70.5%51.7%22.2%6.9%
H = 2 (default)77.5%80.1%60.4%28.6%6.1%
H = 476.3%76.3%60.3%26.4%5.6%

H = 0 is the + RL model. A single step of history already lifts success at σ = 0.5 from 11.2% to 51.7%. Going from H = 2 to H = 4 adds nothing, so we use H = 2 everywhere.

On real robots

Franka. We finetune EA-VLA and the same VLA without execution feedback on 50 demonstrations of a bussing task: put seven objects into a bowl. The demonstrations were collected with a compliant controller, and we test with that controller and with a stiff one. We count how many objects each policy places within 200 steps, averaged over 15 trials, which also reflects failed grasps and retries. EA-VLA places 38% more objects with the compliant controller and 49% more with the stiff one.

WidowX. A model trained in simulation, with milder randomization (σ = 0.1) and an added constant drift, runs zero-shot on a real WidowX in the BridgeData V2 setup, over 25 pick-and-place trials per setting. When the previous command is fed back as the state, a common trick for delta-action VLAs, the two policies are about even. When the robot's measured state is fed back instead, which is the setting EA-VLA is trained for, the baseline drops 16 points and EA-VLA drops 8, ending 11 points ahead.

Baseline (no execution feedback)EA-VLA (ours)
Franka robot above a table with toy vegetables and a bowl.
Franka: bussing
Objects placed in the bowl per episode, of 7
in-domain 4.56.24.16.1 +38%+49% Compliant controllerStiff controller
WidowX gripper above a toy sink with a plate, a dish rack and toy vegetables.
WidowX: zero-shot from simulation
Success rate, 25 pick-and-place trials per setting
in-domain 63%66%47%58% +11 pts Previous commandas state Measured robotstate
In-domain is the condition the baseline is used to: the compliant controller the Franka demonstrations were collected with, and the previous-command state input on WidowX.

Execution feedback, not just more history

Could any history of robot states do the same job? We train a policy conditioned on past proprioception but not on past commands, and compare training rewards in simulation. EA-VLA climbs higher (right). The gap to the RL baseline with no state input is larger still: that baseline's reward peaks around step 300 and then falls (left). What helps is seeing commanded and executed motion side by side, not more temporal context on its own.

vs. RL baseline without state feedback
Training reward: EA-VLA rises to about 0.027 while the baseline peaks near 0.019 and falls to about 0.013.
vs. proprioceptive history only
Training reward: EA-VLA rises to about 0.027 while the proprioception-only policy reaches about 0.022.
Environment reward during RL training in simulation, smoothed. Light traces are the raw values.

Limitations

With the nominal controller, EA-VLA is 5.9 points below RL finetuning without execution feedback (77.5% vs. 83.4%). Robustness costs some peak in-domain performance.

The real-robot studies are small: one Franka task with 15 trials per controller, and 25 WidowX trials per setting.

EA-VLA needs commands and robot states in its training data. Datasets that were not collected by teleoperation, such as human or UMI-style demonstrations, contain neither. We vary the controller, not the robot's morphology, so whether the approach carries across very different embodiments is open. The feedback is limited to proprioception and past commands. Torque, force and tactile signals are left for future work.