Mixture of Wide Experts：稀疏路由实现可解释原生坐标
原标题：Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates
AI 摘要
Plaksha大学FAIRC的研究者提出Mixture of Wide Experts（MoWE），一种稀疏前馈层：路由器为每个token选择少量专家，每个专家读取完整表示但只写入固定且互不重叠的输出坐标。在约10B token、19层、计算量匹配的实验中，MoWE评估交叉熵为2.417，略优于稠密模型的2.452，但标准MoE最低为2.22；在坐标可读性指标上MoWE得0.523，高于稠密模型的0.375和标准MoE的0.268，其原生坐标可读性接近稠密SAE（0.489），MoWE自身SAE进一步提升至0.556。案例研究显示这些坐标可用于追踪预测依据并引导模型拒绝有害请求。
正文节选
Mixture of Wide Experts: Sparse Routing for Interpretable Native Coordinates | | | | | | | Tavish Mankash | | tavish@mankash.com | | Vardhaman Kalloli | | vardhaman.k.ug23@plaksha.edu.in | | Keshava Prasad | | keshava.prasad.ug25@plaksha.edu.in | | Deepan Muthirayan | | deepan.muthirayan@plaksha.edu.in | FAIRC \(\cdot\) Plaksha University Abstract Language models often spread one concept across many internal dimensions, making their internal features difficult to interpret. Sparse