Building a Filtering Tst Collection for TREC 2002

Published

July 1, 2003

Author(s)

Ian Soboroff, S E. Robertson

Abstract

Test collections for the filtering track in TREC have typically used either past sets of relevance judgments, or categorized collections such as Reuters Corpus Volume 1 or OHSUMED, because filtering systems need relevance judgments during the experiment for training and adaptation. For TREC 2002, we constructed an entirely new set of search topics for the Reuters Corpus for measuring filtering systems. Our method for building the topics involved multiple iterations of feedback from assessors, and fusion of results from multiple search systems using different search algorithms. We also developed a second set of ¿inexpensive¿ topics based on categories in the document collection. We found that the initial judgments made for the experiment were sufficient; subsequent pooled judging changed system rankings very little. We also found that systems performed very differently on the category topics than on the assessor-built topics.

Proceedings Title

Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval

Conference Location

, MX

Pub Type

Conferences

Citation

Soboroff, I. and Robertson, S. (2003), Building a Filtering Tst Collection for TREC 2002, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development inInformation Retrieval, , MX (Accessed October 31, 2024)