Voorhees, E.
(2015),
On the Behavior of PRES Using Incomplete Judgment Sets, Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR), Northampton, MA, [online], https://doi.org/10.1145/2808194.2809484
(Accessed February 5, 2025)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
On the Behavior of PRES Using Incomplete Judgment Sets
Published
Author(s)
Abstract
PRES, the Patent Retrieval Evaluation Score, is a family of retrieval system evaluation measures that combines recall and user effort to better reflect the quality of a retrieval run with respect to recall-oriented search tasks. Previous analysis of the measure was done using the test collection for the CLEF-IP 2009 track, a collection that contains a limited range of number of relevant documents, making it difficult to assess the behavior of PRES for varying recall contexts. This paper examines the effect of incomplete judgments on PRES scores using the well-studied TREC-8 ad hoc test collection, a collection with a much more varied number-of-relevants profile. Experiments with small judgment sets created through a typical collection-building process show the PRES measures are resilient to incomplete judgment sets.Proceedings Title
Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR)
Conference Dates
September 27-30, 2015
Conference Location
Northampton, MA
Pub Type
Conferences
Download Paper
Keywords
incomplete judgments, information retrieval evaluation, test collections, TREC
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.
Created September 30, 2015, Updated November 10, 2018