Measurement Uncertainties of Three Score Distributions and Two Thresholds with Data Dependency

Published

September 18, 2014

Author(s)

Jin Chu Wu, Alvin F. Martin, Craig S. Greenberg, Raghu N. Kacker

Abstract

The National Institute of Standards and Technology conducts an ongoing series of Speaker Recognition Evaluations (SRE). Recently a new paradigm was adopted to evaluate the performance of speaker recognition systems, in which three distributions of scores, i.e., target, known non-target, and unknown non-target scores, as well as two thresholds, were employed. The new detection cost function was defined to be an average of the two weighted sums of the probabilities of type I and type II errors corresponding to the two thresholds. In addition, data dependency due to multiple use of the same subjects is also involved. The data were reorganized into a two-layer structure in view of the probability theory, and then the uncertainties of the detection cost functions were computed using the nonparametric three-sample two-layer bootstrap method. Comparing these results with those calculated by using all the raw data and the nonparametric three-sample bootstrap method with the i.i.d. assumption, the measurement accuracies, i.e., the detection cost functions, have changed little; but the measurement uncertainties, i.e., the standard errors of the detection cost function, have improved as a result of taking account of the data dependency. Forty speaker recognition systems were taken as examples.

Citation

NIST Interagency/Internal Report (NISTIR) - 8025

Report Number

8025

NIST Pub Series

NIST Interagency/Internal Report (NISTIR)

Pub Type

NIST Pubs

Download Paper

DOI Link

Keywords

metrology, measure, accuracy, uncertainty, bootstrap, data dependency, speaker recognition

Citation

, J. , Martin, A. , Greenberg, C. and Kacker, R. (2014), Measurement Uncertainties of Three Score Distributions and Two Thresholds with Data Dependency, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.8025 (Accessed March 13, 2025)