用可训练前缀令牌减少检索嵌入中的表示干扰
原标题：Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
AI 摘要
该论文提出一种在嵌入模型训练时于输入前添加可训练[DUMP]前缀令牌的方法，在不改变模型架构、池化策略和对比检索损失的前提下，减少注意力汇等残差令牌级信息对检索嵌入的干扰。在六个嵌入模型上的实验显示，标准检索基准平均相对提升2.46%，长上下文检索场景提升2.41%。分析表明前缀令牌改变了低信息令牌的注意力模式，是一种轻量即插即用的检索表示改进方案。
正文节选
Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens Abstract. Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strat