NIST 2022 Language Recognition Evaluation Plan

Published

August 31, 2022

Author(s)

Yooyoung Lee, Craig Greenberg, Lisa Mason, Elliot Singer

Abstract

The 2022 NIST language recognition evaluation (LRE22) is the 9th cycle in an on-going language recognition evaluation series that began in 1996. The objectives of the evaluation series are (1) to advance technologies in language recognition with innovative ideas, (2) to facilitate the development of language recognition technology incorporating these ideas, and (3) to measure the performance of the current state-of- the-art technology. Targeting researchers working on the general problem of text-independent, speaker independent language recognition, the evaluation is designed to focus on core technology issues and to be simple and accessible to those wishing to participate. LRE22 will be organized in a similar manner to LRE17, focusing on differentiating closely related languages. As in LRE17, fixed and open training conditions will be offered to allow cross-system comparisons and to understand the effect of additional and unconstrained amounts of training data on system performance. LRE22 consists of language detection data in conversational telephone speech (CTS) and broadcast narrow band speech (BNBS). A new focus in LRE22 will be language recognition of African languages, including low-resource languages. The LRE22 evaluation data will consist of multiple segments extracted from each recording, where each sample is select to a specified amount of speech, randomly sampled between 3 and 30 seconds. NIST will distribute to participants a small development set representative of the test set. Participants will be asked to provide score vectors with the log-likelihood scores, rather than the log-likelihood ratios. The primary metric will support equal weighting of data sources. Participation in LRE22 is open to all who find the evaluation of interest and are able to comply with the evaluation rules set forth in this plan. Participating teams must be represented at the LRE 2022 virtual workshop. The details can be found on the LRE22 website https://lre.nist.gov.

Citation

Language Recognition Evaluation

Pub Weblink

https://lre.nist.gov

Pub Type

Websites

Download Paper

Local Download

Keywords

language recognition, evaluation, performance measure

Citation

Lee, Y. , Greenberg, C. , Mason, L. and Singer, E. (2022), NIST 2022 Language Recognition Evaluation Plan, Language Recognition Evaluation, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=935161, https://lre.nist.gov (Accessed December 21, 2024)