Micro-Signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection

Published

February 5, 2017

Author(s)

Richard Harang, Peter Mell

Abstract

The field of intrusion detection is divided into signature detection and anomaly detection. The former involves identifying patterns associated with known attacks and the latter involves attempting to learn a 'normal' pattern of activity and then producing security alerts when behaviors outside of those norms is detected. The n-grams methodology has arguably been the most successful technique for anomaly detection (including for network packet inspection). In this work, we identify a new type of intrusion detection that neither uses typical signatures nor is anomaly based (though it is closely related to both). We generate n-grams from both malicious content and Snort signatures and use sets of these 'micro-signatures' to identify attacks. This micro-signature capability arises implicitly when the training sets for n-gram anomaly detection systems are scrubbed of malicious content and thus is not new. It was added explicitly by the seminal Anagram network anomaly approach, but was portrayed as a minor enhancement and its effect was not evaluated. In reproducing the Anagram results we find that for our data, the micro-signatures provide the vast majority of the detection capability. What appears on the surface to be an anomaly detection approach achieves most of its effectiveness from a (sometimes merely implicit) signature subsystem. We furthermore find that these micro- signatures enable highly effective standalone detection systems as well as hybrid micro- signature/anomaly systems that generalize to multiple attack classes. Our results thus shed new light into the functioning of n-gram anomaly detection systems, reveal the need to evaluate the micro-signature contribution within n-gram anomaly research, and open a new avenue of research into how to best use micro-signatures in future detection systems.

Proceedings Title

Foundations and Practice of Security: 9th International Symposium, FPS 2016

Conference Dates

October 24-25, 2016

Conference Location

Quebec City, CA

Conference Title

9th International Symposium on Foundations and Practice of Security

Pub Type

Conferences

Download Paper

https://doi.org/10.1007/978-3-319-51966-1_3

Local Download

Keywords

intrusion detection, anomaly detection, n-grams, micro-signatures, security

Citation

Harang, R. and Mell, P. (2017), Micro-Signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection, Foundations and Practice of Security: 9th International Symposium, FPS 2016, Quebec City, CA, [online], https://doi.org/10.1007/978-3-319-51966-1_3, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=920217 (Accessed December 26, 2024)