U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Search Publications by: Ellen M. Voorhees (Assoc)

Search Title, Abstract, Conference, Citation, Keyword or Author
Displaying 1 - 25 of 170

Human Preferences as dueling Bandits

July 11, 2022
Author(s)
Xinyi Yan, Chengxi Luo, Charles Clarke, Nick Craswell, Ellen M. Voorhees, Pablo Castells
The dramatic improvements in core information retrieval tasks engendered by neural rankers create a need for novel evaluation methods. If every ranker returns highly relevant items in the top ranks, it becomes difficult to recognize meaningful differences

Too many Relevants: Whither Cranfield Test Collections?

July 11, 2022
Author(s)
Ellen M. Voorhees, Nick Craswell, Jimmy Lin
This paper presents the lessons regarding the construction and use of large Cranfield-style test collections learned from the TREC 2021 Deep Learning track. The corpus used in the 2021 edition of the track was much bigger than the corpus used in previous

Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

January 26, 2022
Author(s)
Ellen M. Voorhees, Ian Soboroff, Jimmy Lin
Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that poo

Searching for Answers in a Pandemic: An Overview of TREC-COVID

September 1, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Lucy L. Wang, William Hersh
We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19. The goals of TREC-COVID include the construction of a pandemic search test collection and the

On the Quality of the TREC_COVID IR Test Collections

July 11, 2021
Author(s)
Ellen M. Voorhees, Kirk Roberts
Shared text collections continue to be vital infrastructure for IR research. The COVID-19 pandemic offered an opportunity to create a test collection that captured the rapidly changing information space during a pandemic, and the TREC-COVID effort was

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

July 11, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos
The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data are likely deep neural networks. This paper

System Explanations: A Cautionary Tale

May 8, 2021
Author(s)
Ellen M. Voorhees
There are increasing calls for systems that are able to explain themselves to their end users to increase transparency and help engender trust. But, what should such explanations contain, and how should that information be presented? A pilot study of

TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection

February 19, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Tasmeer Alam, William Hersh, Kirk Roberts, Dina Demner-Fushman, Kyle Lo, Lucy L. Wang, Steven Bedrick
TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic. One of the key characteristics of pandemic search is the accelerated

Overview of the TREC 2019 Deep Learning Track

July 27, 2020
Author(s)
Ellen M. Voorhees, Nick Craswell, Bhaskar Mitra, Daniel Campos, Emine Yilmaz
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC

TREC-COVID: Rationale and Structure of an Information Retrieval Shared Task for COVID-19

July 8, 2020
Author(s)
Ellen M. Voorhees, Ian Soboroff, Tasmeer Alam, Kirk Roberts, William Hersh, Dina Demner-Fushman, Steven Bedrick, Kyle Lo, Lucy L. Wang
TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining nine important basic IR research

The Evolution of Cranfield

August 14, 2019
Author(s)
Ellen M. Voorhees
This chapter examines how the test collection paradigm, the dominant evaluation methodology in information retrieval, has been adapted to meet the changing requirements for information retrieval research in the era of community evaluation conferences such

On Building Fair and Reusable Test Collections using Bandit Techniques

October 17, 2018
Author(s)
Ellen M. Voorhees
While test collections are a vital piece of the research infrastructure for information retrieval, constructing fair, reusable test collections for large data sets is challenging because of the number of human relevance assessments required. Various

Using Replicates in Information Retrieval Evaluation

August 2, 2017
Author(s)
Ellen M. Voorhees, Daniel V. Samarov, Ian M. Soboroff
This paper explores a method for more accurately estimating the main effect of the system in a typical test-collection-based evaluation of information retrieval systems, and thus increasing the sensitivity of system comparisons. Randomly partitioning the