JUICE：在预训练表示空间中统一多模态建模
原标题：JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces
AI 摘要
研究提出 JUICE，一种在冻结的预训练图像与文本编码器连续表示空间中，用单一流匹配 Transformer 联合建模多模态分布的方法，每个模态有独立噪声水平，通过选择哪些模态保持干净来决定任务类型。在 2.1B 参数、无语言模型初始化条件下，JUICE 在 GenEval 上达到 0.88、VQAv2 上达到 73.1，联合采样在迷宫任务上成功率从独立采样的 54.6% 提升到 93.7%。
正文节选
JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces Abstract Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both