Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Combinatorial Testing Metrics for Machine Learning

Published

Author(s)

Erin Lanus, Laura Freeman, D. Richard Kuhn, Raghu N. Kacker

Abstract

This short paper defines a combinatorial coverage metric for comparing machine learning (ML) data sets and proposes the differences between data sets as a function of combinatorial coverage. The paper illustrates its utility for evaluating and predicting performance of ML models. Identifying and measuring differences between data sets can be of significant value for ML problems, where the accuracy of the model is heavily dependent on the degree to which training data are sufficiently representative of data that will be encountered in application. The utility of the method is illustrated for transfer learning, the problem of predicting performance of a model trained on one data set when applied to another.
Conference Dates
April 12-16, 2021
Conference Location
Porto , PT
Conference Title
IEEE International Conference on Software Testing, Verification and Validation Workshop (ICSTW)

Keywords

combinatorial testing, machine learning, operatingenvelopes, transfer learning, test-set selection

Citation

Lanus, E. , Freeman, L. , Kuhn, D. and Kacker, R. (2021), Combinatorial Testing Metrics for Machine Learning, IEEE International Conference on Software Testing, Verification and Validation Workshop (ICSTW), Porto , PT, [online], https://doi.org/10.1109/ICSTW52544.2021.00025, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=931825 (Accessed October 31, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created April 12, 2021, Updated November 29, 2022