Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Promoting Repeatability Through Open Runs

Published

Author(s)

Ellen M. Voorhees, Shahzad K. Rajput, Ian M. Soboroff

Abstract

TREC 2015 introduced the concept of ‘Open Runs’ in response to the increasing focus on repeatability of information retrieval experiments. An Open Run is a TREC submission backed by a software repository such that the software in the repository reproduces the system that created that exact run. The id of the repository was captured during the process of submitting the run and published as part of the metadata describing the run in the TREC proceedings. Submitting a run as an Open Run was optional: either a repository id was provided at submission time or it was not, and further processing of the run was identical in either case. Unfortunately, this initial offering was not successful. While a healthy 79 runs were submitted as Open Runs, we could not in fact reproduce any of them. This paper explores possible reasons for the difficulties and makes suggestions for how to address the deficiencies so as to strengthen the Open Run program for TREC 2016.
Proceedings Title
Proceedings of The 7th International Workshop on Evaluating Information Access (EVIA 2016)
Conference Dates
June 7, 2016
Conference Location
Tokyo
Conference Title
The 7th International Workshop on Evaluating Information Access (EVIA 2016)

Keywords

information retrieval, repeatability, search engines, TREC

Citation

Voorhees, E. , Rajput, S. and Soboroff, I. (2016), Promoting Repeatability Through Open Runs, Proceedings of The 7th International Workshop on Evaluating Information Access (EVIA 2016), Tokyo, -1 (Accessed October 31, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created June 7, 2016, Updated April 17, 2018