Common Crawl News 与 GDELT 新闻数据集对比研究
原标题：Comparison of Common Crawl News & GDELT
AI 摘要
该论文比较了两个公开新闻数据集 GDELT 和 Common Crawl News（CC-News），分析其内容与覆盖范围。GDELT 汇集全球广播、印刷和网络新闻，覆盖100多种语言，可追溯至1979年，使用 CAMEO 编码并依赖 Google Cloud 处理，但只提供事件元数据而非原始文本。CC-News 则通过基于 Nutch 的 CCBot 网络爬取全球新闻站点，数据以 WARC 格式存储在 Amazon S3 上，Common Crawl 数据曾贡献 GPT-3 训练中82%的原始 token。研究发现两者在新闻来源的采集地点上存在显著差异。
正文节选
Comparison of Common Crawl News & GDELT Abstract The corpus of worldwide news is important for natural language processing, knowledge graphs, large language models, and other technical efforts. Additionally, this corpus is important for understanding the people, places, organizations, and events that interact in real-time every day. This paper compares two news datasets used for these tasks today, namely the Global Database of Events, Language, and Tone (GDELT) and Common Crawl News. Our resear