阿里提出 Omni-Encoder：共享 SigLIP2 统一四模态表示
原标题：When Does Sharing Work? Scaling a Shared SigLIP2 Across Image, Video, Audio, and Audio-Visual Representations
AI 摘要
阿里巴巴团队提出 Omni-Encoder，在单个 SigLIP-2 Vision Transformer 中以完全共享参数处理图像、视频、音频和音视频输入，并通过模态原生输入阶段、基于绝对时间戳的多模态旋转位置编码以及交替模态训练实现端到端联合优化。该方法在 16 个基准上取得有竞争力的结果，且联合训练相比各模态单独训练均有提升，表明跨模态正向迁移的潜力。
正文节选
When Does Sharing Work? Scaling a Shared SigLIP2 Across Image, Video, Audio, and Audio-Visual Representations Abstract Contrastive pretraining has advanced image-text understanding, yet image, video, and audio encoders are still trained independently, with cross-modal alignment realized mainly through post-hoc binding to a frozen anchor space. We present Omni-Encoder, a general multimodal encoder that processes image, video, audio, and audio-visual inputs within a single SigLIP-2 Vision Transfor