VakyArth：评估大语言模型在印度语言中的语用能力

AI 摘要

VakyArth 是首个针对印度语言的语用基准，覆盖印地语、旁遮普语、泰米尔语和马拉雅拉姆语，评估模型在指示语、言语行为、含义、社会语用和连贯性五个方面的表现。研究发现多语言大模型在这些语言上普遍存在语用理解失败，且翻译性能不能可靠反映语用理解。该基准由母语者编写，包含多项选择、自然语言推理和翻译三种任务格式。

正文节选

VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages Abstract Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic languages unexplored despite their linguistic and cultural diversity. We introduce VakyArth, the first pragmatic benchmark for Indic languages, desi

发布时间：2026-09-03 12:00

抓取时间：2026-09-03 18:02

来源机构：arXiv