Programs for Evaluating Software Testing Tools and Techniques (PESTTT)

CONTACT:
James R. Lyle, Software and Systems Division
301-975-3270
james.lyle [at] nist.gov (james[dot]lyle[at]nist[dot]gov)

IMPETUS/HOW PROJECT BEGAN: We initiated the project to satisfy a clear need for an unbiased reference suite to evaluate and compare software testing techniques. Often the publication of a new testing technique or strategy includes a theoretical analysis and an ad hoc empirical evaluation. Since each researcher usually uses a unique set of programs for an empirical evaluation, there is no basis for comparison between different techniques. A common suite of faulty programs would remedy this problem.

OBJECTIVE(S): The project objective is to develop and make available to researchers and tool vendors a set of reference materials for the empirical evaluation and comparison of software testing techniques. This set of reference materials includes a diverse and comprehensive suite of formally specified program modules consisting of an oracle and many faulty versions, each seeded with one error selected from some commonly available error taxonomy. The PESTTT materials can be used to evaluate and compare both white box and black box testing techniques.

TIMELINE/MILESTONES: A NISTIR technical report and an initial suite of C programs will be made available over the WWW sometime in 2000.

NIST INVOLVEMENT: SSD (Neva Carlson, Mary Laamanen, Jim Lyle)

OUTSIDE INVOLVEMENT: CRADA with Perennial Corporation.

WHAT WORK HAS ITL DONE: A prototype implementing a subset of features and capabilities has been developed, including the design of the application structure, some initial program modules, and a navigation and access tool for the modules.

WHAT WORK HAS BEEN DONE BY COLLABORATORS: Perennial Corporation provided NIST with ACVS, a validation suite for FIPS-160 ANSI/ISO C. Some of the modules are being evaluated for suitability for inclusion in PESTTT.

WHAT DEVELOPMENTS HAVE OCCURRED AND HOW HAS THE PROJECT BEEN MODIFIED: A module navigation and access tool is under development in Java. The initial staff (Lyle) has been expanded twice (to add Laamanen part time and Carlson in summer 1999). It was observed that PESTTT is useful not only to software testing researchers, but also to software testing tool vendors. This observation broadens the focus and potential user base of PESTTT. A draft conference paper has been submitted but not yet accepted for publication.

PUBLICATIONS AND ARTIFACTS:

PEST: Programs for Evaluating Software Testing Tools and Techniques (pdf)

A unix tar file with two sample modules can be downloaded (a unix system is need to un-tar the file with the command:   
tar xvf pest-1.0.tar) -- [file no longer available]

WHAT INDUSTRIES HAVE BENEFITED FROM NIST'S WORK AND HOW: PESTTT has not yet been released. However, two direct beneficiaries are the testing tool vendors and any users of testing tools. The vendors benefit by having a suite of programs to use for evaluation of their products. The users of testing tools benefit by having an independent means of comparison of testing tools.

ACKNOWLEDGEMENTS OF ITL'S EFFORTS: PESTTT has not yet been released.

FUTURE RELATED ACTIVITIES: The initial version of PESTTT is aimed at testing complete programs; additional work is planned for testing object components such as Java classes.