LATIDIA · Investigación

MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

arXiv: 2610.20965v1Tipo de anuncio: nuevo Resumen: Los grandes modelos multimodales han logrado un progreso notable al escalar la capacidad del modelo y la capacitación en datos multimodales emparejados masivos. Sin embargo, este paradigma introduce un fondo

La noticia

arXiv:2610.20965v1 Announce Type: new Abstract: Large multimodal models have achieved remarkable progress by scaling model capacity and training on massive paired multimodal data. However, this paradigm introduces a fundamental bottleneck: aligned multimodal datasets are expensive to curate, difficult to scale across diverse modality combinations, and increasingly constrain further progress. We introduce Modality Self-Play (MSP), a training framework for multimodal foundation models based on proxy-mediated modality composition. Instead of relying on paired cross-modal data, MSP uses anchor-supporting proxy textual descriptions of unseen modalities as semantic bridges during training. Given an observed modality (e.g., image or audio), the model generates proxy descriptions for complementary unseen modalities, which are then encoded and aligned