返回全部动态
ttok 1.0 发布：默认分词器切换至 GPT-5/GPT-6
原标题：ttok 1.0
AI 摘要
Simon Willison 发布 ttok 1.0，将默认分词器从 GPT-4 切换为 GPT-5/GPT-6 系列。此前 OpenAI 尚未正式确认 GPT-6 与 GPT-5 使用相同分词器，但 William Liu 的实验显示七个 GPT 模型在 31 个测试样本上分词结果完全一致，均报告 44,794 个 token。
以上摘要由 AI 生成，可能存在误差。事实请以原文为准。
正文节选
9th October 2026 I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead! I figured switching the default was a reasonable excuse to finally ship a 1.0. OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he
发布时间：2026-10-09 08:34
抓取时间：2026-10-09 12:33
来源机构：Simon Willison