ProdCrawl：可复现的聚焦爬虫测试集合
原标题：ProdCrawl: A Replayable Test Collection for Focused Crawling
AI 摘要
研究者发布 ProdCrawl，一个可复现的聚焦爬虫测试集合，包含 621 个公司及组织网站的静态镜像、约 110 万页面，并由三个大语言模型多数投票标注每页是否为产品页。任何爬虫都可在相同站点上运行并按相同标签评分，从而解决实时网页漂移与缺乏真实标注导致对比不可靠的问题。实验显示文本分类器和关键词匹配未显著优于广度优先爬取，而基于语义的神经嵌入优于所有简单策略。数据集、代码库和模型已公开。
正文节选
ProdCrawl: A Replayable Test Collection for Focused Crawling Abstract A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated unde