A robot that wakes up somewhere in a known building has to answer a basic question before it can do anything useful: where am I? In grocery aisles, office corridors, and rows of identical cubicles, the answer is hard, because the walls look the same everywhere. Classical lidar localization (AMCL) works well when it is told roughly where it started, but it cannot tell one aisle from another when it has to find itself from scratch.
People solve this by looking at what is around them, not at wall geometry. VLM-GLoc gives a robot the same ability. A vision-language model (VLM) describes what the camera sees in open-vocabulary language, and those descriptions become the observation model inside a standard particle filter. The VLM never outputs coordinates; probabilistic filtering, odometry, and range sensing still do the localization.
Rich language, not class labels. To an object detector, a cabinet is a “cabinet,” and an office has many. To VLM-GLoc it is a “yellow metal flammables storage cabinet,” which there is only one of. Descriptions keep color, material, readable text, packaging, and object state, exactly the details that distinguish repeated objects. The same interface works for office furniture seen by a quadruped and for long-tail grocery products seen by a smartphone.
Camera images are described by a VLM and embedded with a lightweight, fine-tuned sentence encoder. Matching against a prior semantic map seeds particles near the most likely poses (inverse semantic proposal); a hierarchical filter then leans on semantics first and tightens with range data once the particles converge.
Results
We evaluated on 20 real trajectories in each of two very different settings: a 3,700 sq. ft. lab and office space with a Unitree Go1 quadruped, tested a month after mapping while bikes, robots, and furniture had moved, and a 3,500 sq. ft. grocery store with a handheld smartphone. Every method starts with no knowledge of its pose.
Office (quadruped): global localization success
VLM-GLoc
87%
Fixed-class semantic MCL
17%
VLG-Loc
5%
AMCL (lidar)
3%
OMCL
0%
ROMAN
0%
Grocery store (smartphone): global localization success
VLM-GLoc
75%
ShelfAware
57%
AMCL (depth)
23%
Fixed-class semantic MCL
19%
ROMAN
10%
VLG-Loc
5%
OMCL
1%
Converged within 0.7 m and 45°. Mean over five seeds × 20 trajectories. In the grocery store, VLM-GLoc also keeps tracking stably on 66% of runs, versus 25% for ShelfAware, our earlier retail-specific system.
What makes it work
Each ablation changes one thing and re-runs all 20 trajectories with five seeds.
Office: global localization success
Full method
87%
No text augmentation
57%
Frozen text encoder
40%
Generic class labels
26%
Grocery: global localization success
Full method encoder trained on office only
75%
Frozen text encoder
71%
Keep blurry products
62%
Generic class labels
56%
Rich descriptions are the key ingredient. Replacing “yellow metal flammables cabinet” with “cabinet,” with everything else fixed, drops office success from 87% to 26%.
Augmentation in language, not pixels. Randomly dropping objects from text descriptions during fine-tuning teaches the encoder that a real camera rarely sees everything the map expects: +30 points over fine-tuning without it.
The VLM filters its own bad views. Letting the VLM flag blurry, unreadable products and discarding them adds 13 points in the handheld grocery setting.
It transfers. The encoder fine-tuned only on the office works directly in the grocery store, with no grocery training.
Where the robot thinks it is
Top-10 pose proposals (orange) from a single camera view, with the true pose in green and semantic similarity as the heatmap. Proposals land in the right region despite motion blur (b), mirrored viewpoints (c, f, g, h), repeated doors (e), and objects moved in the month since mapping (f, g).
Global localization in the grocery store from a smartphone, six runs. Particles (blue) start spread across the whole store and converge on the true pose (red) as product descriptions come in; the estimate is in green.
Paper
2026
ArXiv
VLM-GLoc: Vision-Language Model Enhanced Monte Carlo Localization for Robust Semantic Global Localization in Cluttered Quasi-Static Environments
Global localization in geometrically aliased, quasi-static environments such as grocery stores, offices, schools, and hospitals poses a significant challenge for mobile robots. Grocery stores with parallel aisles and a long tailed distribution of products, as well as offices and labs with repetitive furniture such as chairs, desks, monitors, and doors, exemplify common indoor environments that present geometric and even semantic ambiguity. Traditional approaches rely either on distinct geometric features or on domain-specific vision pipelines that struggle with long-tail semantic distributions and transient visual clutter. We present VLM-GLoc, a method for hierarchical semantic Monte Carlo Localization (MCL) that leverages open-vocabulary Vision-Language Models (VLMs) as a unified semantic observation front-end. We hypothesize a three-fold benefit from VLMs: (1) extracting highly discriminative rich text features, (2) implicit quality filtering of blurry or dynamic objects, and (3) permanence reasoning for targeted data augmentation. We introduce an inverse semantic proposal mechanism that seeds particles via text-to-map retrieval. Evaluated across two real-world environments with different characteristics and two different platforms: a 3,500 sq. ft. grocery store with a cellphone and a 3,700 sq. ft. lab space with a quadruped, VLM-GLoc achieves 70% and 74% global localization success respectively, substantially outperforming traditional geometry-only and domain-specific baselines.
@article{agrawal2026vlmgloc,title={VLM-GLoc: Vision-Language Model Enhanced Monte Carlo Localization for Robust Semantic Global Localization in Cluttered Quasi-Static Environments},author={Agrawal, Shivendra and Hayes, Bradley},journal={arXiv preprint arXiv:2605.30506},year={2026},topic={vlmgloc},url={https://arxiv.org/abs/2605.30506},}
The current version is under review. An earlier version of this work is available on arXiv.