Design and Implementation of an Automated Network Traffic Analysis System using Elastic Stack

Published

April 1, 2024

Author(s)

Zineb Maasaoui, Anis Bekri, Mheni Merzouki, Abdella Battou, Amar Abane, Ahmed LBATH

Abstract

This paper builds upon our previous work on Network Security Traffic Analysis Platforms (NSTAP) [1], presenting an advanced framework for the real-time monitoring of network traffic and endpoint security in large-scale enterprises. We employ a fully integrated technology stack that includes Elastic Stack, ZEEK, Osquery, Kafka, and GeoLocation data to create a comprehensive security analytics solution. A significant contribution of this research is the integration of supervised machine learning models into our platform, trained specifically on the UNSW-NB15 dataset. We explored three supervised machine learning algorithms - Random Forest (RF), Decision Trees (DT), and Support Vector Machines (SVM). For SVM, we also tested a dimensionality reduction algorithm to maximize model accuracy and optimized both computation time and performance. The evaluation is based on Accuracy, False Positive Rate (FPR) and revealed that the Random Forest Classifier, in conjunction with Pearson correlation-based feature selection methods, achieved the highest accuracy of 99.32\% and an error rate of 0.67\%.These findings not only substantiate the robustness of our unified platform but also set the stage for future research in developing scalable, efficient, and automated security solutions tailored for large enterprises.

Proceedings Title

20th ACS/IEEE International Conference on Computer Systems and Applications

Conference Dates

December 4-7, 2023

Conference Location

Giza, EG

Pub Type

Conferences

Download Paper

DOI Link

Keywords

Elastic stack, Network security, Network logging, Artificial Intelligence, Data Analysis

Citation

Maasaoui, Z. , Bekri, A. , Merzouki, M. , Battou, A. , Abane, A. and Lbath, A. (2024), Design and Implementation of an Automated Network Traffic Analysis System using Elastic Stack, 20th ACS/IEEE International Conference on Computer Systems and Applications, Giza, EG, [online], https://doi.org/10.1109/AICCSA59173.2023.10479347 (Accessed October 19, 2024)