微小扰动即可改写感知：VLM 定向语义替换攻击
原标题：It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $\epsilon \leq 4/255$
AI 摘要
该论文提出一种针对视觉语言模型（VLM）的定向语义替换攻击方法 TokRA，通过在受害者模型合并后的 token 空间中对齐源图像与目标图像的视觉 token，在 ε≤4/255 的扰动预算下同时注入目标语义并消除源语义。在严格成功标准下，图像攻击在 ε=4/255 时完全替换率达 38%，视频攻击达 35.9%。研究还发现语义融合现象，即 LLM 会将矛盾的视觉信号合理化为连贯叙述，表明 VLM 在此扰动范围内的鲁棒性并不成立。
正文节选
It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at Abstract Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at . Therefore, VLMs seems robust to perturbations in this ran