LATIDIA · Robótica

Un agente para guiarlos a todos: empoderar a los MLLM para la navegación de visión y lenguaje a través de la representación explícita del mundo

arXiv: 2602.15400v2Announce Type: replace Abstract: Un agente de navegación debe comprender las instrucciones semánticas de alto nivel y razonar sobre la estructura espacial precisa. Modelos Multimodales de Lenguaje Grande (MLLMs), con su Str

La noticia

arXiv:2602.15400v2 Announce Type: replace Abstract: A navigation agent must understand high-level semantic instructions and reason about precise spatial structure. Multimodal Large Language Models (MLLMs), with their strong generalization, are a promising foundation for such agents. We propose GTA, a decoupled design that separates low-level spatial state estimation from high-level semantic planning. We introduce an explicit metric world representation that combines accumulated geometry with navigation history and exposes this information to the MLLM through a multimodal interface for decision-making. We further introduce counterfactual branch reasoning that checks anticipated outcomes against subsequent observations and revises navigation decisions when they disagree. The world representation grounds waypoint selection in reconstructed metric geometry. GTA achieves