ClustRS：无需训练的鲁棒视觉token剪枝算法
原标题：Clustering and Token Denoising for Faster and More Robust VLMs
AI 摘要
法国雷恩大学、CNRS、INRIA、IRISA和Vsora公司联合提出ClustRS，一种无需训练的视觉语言模型（VLM）token剪枝算法，包含注意力加权聚类和残差收缩去噪两个组件。实验表明，在LLaVA 1.5 7b上，ClustRS在极端噪声和token条件下（减少97%至16个token）比现有方法性能提升高达20%，并在LLaVA-OneVision上以不到三分之一的token匹配基线性能。该方法计算开销小，兼容KV-cache和flash-attention，适用于边缘设备部署。
正文节选
Clustering and Token Denoising for Faster and More Robust VLMs 2Univ. Rennes, CNRS, INRIA, IRISA, France 3Vsora, France Abstract Recent Visual-Language Models (VLMs) have enhanced the capabilities of pre-trained LLMs by adding vision tokens alongside text, with approaches like LLaVA showing impressive results. However, the computational burden of processing up to 576 or 729 visual tokens makes edge deployment challenging. While various token pruning techniques require retraining, some are traini