A Combinatorial Approach to Detecting Buffer Overflow Vulnerabilities

Published

June 14, 2011

Author(s)

Raghu N. Kacker, Yu Lei, David R. Kuhn, Wenhua Wang

Abstract

Buffer overflow vulnerabilities are program defects that can cause a buffer overflow to occur at runtime. Many security attacks exploit buffer overflow vulnerabilities to compromise critical data structures. In this paper, we present a black-box testing approach to detecting buffer overflow vulnerabilities. In most cases the attacker can influence the behavior of a target system only by controlling its external parameters. Therefore, launching a successful attack often amounts to a clever way of tweaking the values of external parameters. Our approach identifies two conditions that must be met in order to trigger a buffer overflow, and is centered on how to tweak external parameter values in a systematic manner to satisfy these two conditions. A novel aspect of our approach is that it adapts a general software testing technique called combinatorial testing to the domain of security testing. In particular, our approach exploits the fact that combinatorial testing often achieves a high level of code coverage. We report a prototype tool, called Fugai, that implements our approach. The results of applying Fugai to five open-source programs show that our approach is very effective in detecting buffer overflow vulnerabilities in these programs.

Citation

ACM Transactions on Software Engineering and Methodology

Pub Type

Journals

Download Paper

Local Download

Keywords

software security, security testing, buffer overflow vulnerability

Citation

Kacker, R. , Lei, Y. , Kuhn, D. and , W. (2011), A Combinatorial Approach to Detecting Buffer Overflow Vulnerabilities, ACM Transactions on Software Engineering and Methodology, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=905210 (Accessed October 13, 2025)