llama.cpp b11455 实现 PLaMo-3 分词器预分段

AI 摘要

llama.cpp 发布 b11455 版本，为 PLaMo-3 分词器实现了预分段（pre-segmentation）功能。该实现通过在 Unigram 动态规划前插入硬边界，处理 <|plamo:...|> 形式的文本、至少 4 个相同字符的连续串以及连续两个空格，并新增了 "plamo3" 词表类型。若不这样做，llama.cpp 对代码缩进和重复标点的分词结果会与参考实现不一致。

正文节选

<details open> vocab : implement PLaMo-3 tokenizer pre-segmentation (#30045) * vocab : implement PLaMo-3 tokenizer pre-segmentation The PLaMo-3 tokenizer inserts hard boundaries before running the Unigram DP, around <|plamo:...|>-looking text, and around runs of at least 4 identical characters or 2 spaces. Without them llama.cpp tokenizes code indentation and repeated punctuation differently from the reference. Reproduce the two re.sub() passes in llm_tokenizer_plamo2 by encoding each segmen

发布时间：2026-10-07 04:41

抓取时间：2026-10-07 05:29

来源机构：ggml-org