The Second Static Analysis Tool Exposition (SATE) 2009

Published

July 2, 2010

Author(s)

Vadim Okun, Paul E. Black, Aurelien M. Delaitre

Abstract

The NIST SAMATE project conducted the second Static Analysis Tool Exposition (SATE) in 2009 to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement and speed adoption of tools. Briefly, participating tool makers ran their tool on a set of programs. Researchers led by NIST performed a partial analysis of tool reports. The results and experiences were reported at the SATE 2009 Workshop in Arlington, VA, in November, 2009. The tool reports and analysis were made publicly available in 2010. This paper describes the SATE procedure and provides our observations based on the data collected. The procedure was improved based on the SATE 2008 experience. The changes included selecting subsets of tool warnings for analysis randomly and also based on the human analysis, more detailed analysis categories and criteria, expanding the output format with a richer description of weakness paths, and a more careful analysis of tool warnings. The SATE data suggests that while tools often look for different types of weaknesses and the number of warnings varies widely by tool, there is a higher degree of overlap among tools for well known weakness categories, such as buffer errors. Also, while human analysis is best for some types of weaknesses, tools find a significant portion of weaknesses considered important by human experts. This paper identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software can be used for empirical research. Second, the analysis of tool reports indicates actual weaknesses that exist in the software and that are reported by the tools. Finally, the analysis may also be used as a basis for a further study of the weaknesses and of static analysis.

Citation

Special Publication (NIST SP) - 500-287

Report Number

500-287

NIST Pub Series

Special Publication (NIST SP)

Pub Type

NIST Pubs

Download Paper

Local Download

Keywords

Software security, static analysis tools, security weaknesses, vulnerability

Citation

Okun, V. , Black, P. and Delaitre, A. (2010), The Second Static Analysis Tool Exposition (SATE) 2009, Special Publication (NIST SP), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=905879 (Accessed December 17, 2024)