Real Time Detection of Security Threats through Massive Network Traffic Streams Analysis and Adaptive Learning in Large Scale Networks under a Dynamic Environment

Abstract

Day by day, the number of devices in large-scale networks, such as the Internet and the Internet of Things (IoT), is exponentially growing. Concurrently, the number of cybersecurity attacks is also increasing and causing a risk to their overall security, accessibility, and privacy. SonicWall, a cybersecurity firm, revealed in its 2024 Mid-Year Report that IoT attacks worldwide rose by 107%, with affected devices averaging 52.8 hours under attack. Hence, timely identification and classification of these security attacks is significant to mitigate them. This requires the ability to process and analyze network traffic streams originating from devices in both the IoT and the Internet in real-time. Many researchers have proposed machine learning-based attack detection systems for real-time detection and classification of attacks in network traffic streams from the Internet and IoT. However, the performance of these systems deteriorates when concept drift occurs, when the volume and velocity of network traffic streams increase, or when they are deployed on resource-constrained IoT devices. Here, concept drift refers to the changes in the statistical properties of attack patterns over time. The main objective of this research is to solve these problems by building a scalable stream processing pipeline that collects network traffic streams from geographically distributed IoT devices and network nodes, thereby enabling real-time attack and malicious node classification in the presence of concept drift. To achieve this objective, this research work presents three significant contributions. The first work proposes a scalable, real-time P2P bot host detection system that classifies P2P hosts as either P2P bots or benign by extracting minimal statistical network traffic features from massive network traffic streams. The proposed system consists of two components: P2P host detection and P2P bot host detection. The P2P host detection component first identifies P2P hosts by utilizing Destination Diversity Ratio (DDR)....