Sombit Dey

I am a third-year PhD student in Robotics, Systems, and Control at INSAIT in Sofia, Bulgaria, advised by Prof. Luc Van Gool and Dr. Danda Pani Paudel. My research lies at the intersection of generalist foundational models and robotic manipulation, focusing on bridging vision-language-action (VLA) understanding with fast-reactive robot control.

Prior to my PhD, I completed my M.Sc. in Robotics, Systems and Control at ETH Zurich, where I worked on 3D visual grounding at the Computer Vision Lab (CVL) and on robotics at the Robotic Systems Lab (RSL). I received my B.Tech. in Electronics and Electrical Communication Engineering with a minor in Computer Science and Engineering from IIT Kharagpur, where I was a member of the Autonomous Ground Vehicle (AGV) research group with Prof. Debashish Chakravarty.

Email  /  GitHub  /  Google Scholar  /  LinkedIn

Asynchronous Robot Policies

We run the large vision-language model and the small action network of a robot policy as two separate processes. The robot reacts to every new camera frame without waiting for the large model, and the large model only needs to update now and then.

Execution-Aware VLAs for Robust Manipulation

Robots never execute exactly what a VLA commands, and how far off they are depends on the controller. EA-VLA feeds the policy its previous commands together with the robot’s measured state and finetunes it with RL under randomized controllers, so it learns to correct for the difference. It stays robust across controllers, with an 11% higher success rate on WidowX and up to 50% higher throughput on Franka.

AR-VLA: Autoregressive Action Expert for Vision-Language-Action Models

We replace the chunk-based action head of a VLA with a standalone autoregressive action expert that keeps a long-lived memory of its own actions while attending to vision-language features that refresh asynchronously. AR-VLA produces smoother trajectories, remembers task state that is no longer visible, and matches or beats chunk-based VLAs in SIMPLER and on a real WidowX robot.

SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding

We build SPEAR-VLM, a 3D-aware vision-language model that infers object coordinates from a single image, and use it to train SPEAR-1, a robotic foundation model with grounded 3D perception and language-conditioned control. SPEAR-1 trains on ~45M frames from 24 Open X-Embodiment datasets and matches or surpasses π0-FAST and π0.5 while using 20× fewer robot demonstrations. Model weights are openly released.

ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models

We study the visual generalization capabilities of three existing robotic foundation models, and propose a corresponding evaluation framework.

Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding

We introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description

Learning whom to trust in navigation: dynamically switching between classical and neural planning

We introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description