U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Search Publications by: Ian Soboroff (Fed)

Search Title, Abstract, Conference, Citation, Keyword or Author
Displaying 1 - 25 of 131

Don't Use LLMs to Make Relevance Judgments

February 25, 2025
Author(s)
Ian Soboroff
Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be

TREC 2015 Dynamic Domain Track Overview

October 1, 2024
Author(s)
Ian Soboroff, Hui Yang, John Frank
Search tasks for professional searchers, such as law enforcement agencies, police officers, and patent examiners, are often more complex than open domain Web search tasks. When professional searchers look for relevant information, it is often the case that

Report on The Search Futures Workshop at ECIR 2024

August 7, 2024
Author(s)
Leif Azzopardi, Charles Clarke, Paul Kantor, Bhaskar Mitra, Johanne Trippas, Zhaochun Ren, Ian Soboroff
The First Search Futures Workshop, in conjunction with the Fourty-sixth European Conference on Information Retrieval (ECIR) 2024, looked into the future of search to ask questions such as: ˆ How can we harness the power of generative AI to enhance, improve

On the Evaluation of Machine-Generated Reports

July 14, 2024
Author(s)
James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, Noah Hibbler
Large Language Models (LLMs) have enabled new ways to satisfy information needs. Although great strides have been made in applying them to settings like document ranking and short-form text generation, they still struggle to compose complete, accurate, and

Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance

June 17, 2024
Author(s)
Somnath Banerjee, Avik Halder, Rajarshi Mandal, Sayan Layek, Ian Soboroff, Rima Hazra, Animesh Mukherjee
The integration of pretrained language models (PLMs) like BERT and GPT has revolutionized NLP, particularly for English, but it has also created linguistic imbalances. This paper strategically identifies the need for linguistic equity by examining several

Overview of TREC 2023

May 1, 2024
Author(s)
Ian Soboroff
TREC 2023 is the thirty-second edition of the Text REtrieval Conference (TREC). The main goal of TREC is to create the evaluation infrastructure required for large-scale testing of information retrieval (IR) technology. This includes research on best

2024 NIST Generative AI (GenAI): Data Creation Specification for Text-to-Text (T2T) Generators

April 1, 2024
Author(s)
Yooyoung Lee, George Awad, Asad Butt, Lukas Diduch, Kay Peterson, Seungmin Seo, Ian Soboroff, Hariharan Iyer
Generator (G) teams will be tested on their system ability to generate content that is indistinguishable from human-generated content. For the pilot study, the evaluation will help determine strengths and weaknesses in their approaches including insights

2024 NIST Generative AI (GenAI): Evaluation Plan for Text-to-Text (T2T) Discriminators

April 1, 2024
Author(s)
Yooyoung Lee, George Awad, Asad Butt, Lukas Diduch, Kay Peterson, Seungmin Seo, Ian Soboroff, Hariharan Iyer
Generator (G) teams will be tested on their system's ability to generate content that is indistinguishable from human-generated content. For the pilot study, the evaluation will help determine strengths and weaknesses in their approaches including insights

Overview of the NTCIR-17 FairWeb-1 Task

February 13, 2024
Author(s)
Sijie Tao, Nuo Chen, Tetsuya Sakai, Zhumin Chu, Hiromi Arai, Ian Soboroff, Nicola Ferro, Maria Maistro
This paper provides an overview of the NTCIR-17 FairWeb-1 Task. FairWeb-1 is an English web search task which seeks more than an ad-hoc web search task. Our task considers not only document relevance but also group fairness. We designed three types of

The BETTER Cross-Language Information Retrieval Datasets

July 27, 2023
Author(s)
Ian Soboroff
The IARPA BETTER (Better Extraction from Text Through Enhanced Retrieval) program held three evaluations of information retrieval (IR) and information extraction (IE). For both tasks, the only training data available was in English, but systems had to

What Makes a Good Podcast Summary?

July 11, 2022
Author(s)
Rezvaneh Rezapour, Sravana Reddy, Rosie Jones, Ian Soboroff
Abstractive summarization of podcasts is motivated by the growing popularity of podcasts and the needs of their listeners. Podcasting is a markedly different domain from news and other media that are commonly studied in the context of automatic

Overview of TREC 2021

May 6, 2022
Author(s)
Ian Soboroff
TREC 2021 is the thirtieth edition of the Text REtrieval Conference (TREC). The main goal of TREC is to create the evaluation infrastructure required for large-scale testing of retrieval technology. This includes research on best methods for evaluation as

Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

January 26, 2022
Author(s)
Ellen M. Voorhees, Ian Soboroff, Jimmy Lin
Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that poo

PSCR 2021: Social Media Incident Streams

October 1, 2021
Author(s)
Ian Soboroff
Monitoring social media for public safety is incredibly challenging. The TREC Social Media Incident Streams project collects social media during emergency events, annotates and labels it for public safety use, and provides a metrics-focused environment

PSCR 2021: Pecha Kucha Portfolio Overviews

September 28, 2021
Author(s)
John Beltz, Scott Ledgerwood, Roger Blalock, Joe Grasso, John S. Garofolo, Jesse Frey, Cara O'Malley, Fernando Cintron, Bill Fisher, Gema Howell, Yee-Yin Choong, Jack Lewis, Paul Merritt, Edmond J. Golden III, Ian Soboroff, Craig Connelly, Gary Howarth, Brianna Vendetti, Katelynn Kapalo, Margaret Pinson
PSCR Research Portfolio Leaders join their staff to provide an overview of the projects housed within their PSCR portfolio. Each portfolio overview is delivered in a traditional Pecha Kucha style presentation, dividing topics into 20 slides that when

Searching for Answers in a Pandemic: An Overview of TREC-COVID

September 1, 2021
Author(s)
Ellen M. Voorhees, Ian Soboroff, Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Lucy L. Wang, William Hersh
We present an overview of the TREC-COVID Challenge, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19. The goals of TREC-COVID include the construction of a pandemic search test collection and the