Token边界的代价：压缩证书与预测
原标题：The Price of Token Boundaries: Compression Certificates and Prediction
AI 摘要
该论文研究预分词（pre-tokenisation）中正则表达式边界规则对压缩成本的影响，提出通过为候选 token 出现分配非负价格、结合最短路径与词汇预算选择来给出 token 数量的下界，并用独立整数检查器验证。在英文维基百科上，边界规则使最优 token 数增加，BPE 高于受限下界但低于无约束下界。配对多语言实验显示，无约束拟合词典压缩更好，但在所有语言中平均测试 bits per byte 更高，说明最小 token 数并非语言建模的完整目标。作者还引入 boundary licences 来研究中间边界策略。
正文节选
The Price of Token Boundaries: Compression Certificates and Prediction Abstract Pre-tokenisation restricts which text fragments can become prediction units, but its compression cost is obscured when tokenisers are compared only under the same boundaries. We measure this cost by bounding the minimum token count from both sides, with and without a regular-expression boundary rule. Non-negative prices on token occurrences yield a lower bound through shortest paths and vocabulary-budget selection; m