Anomaly detection of denial-of-service network traffic attacks using autoencoders and isolation forest - Ghulam Hussain, Muhammad Thaqif; Shafeeq Lone, Aman; Maspo, Nur-Adib; Attarbashi, Zainab (IIUM Press, 2026)
This paper presents an unsupervised network-based anomaly detection framework that integrates deep autoencoders with the Isolation Forest algorithm. The framework analyzes extracted traffic features, including packet length and IP address patterns, to detect deviations from normal behaviour without requiring labelled data. Autoencoders reconstruct benign traffic to highlight subtle deviations, while Isolation Forest efficiently assigns anomaly scores to identify statistical outliers in large-scale, unlabelled datasets. Experimental evaluation shows that the Isolation Forest model achieves a low mean squared error (MSE) of 0.0065 with an accuracy of 9.79%, indicating stable anomaly score separation, whereas the standalone autoencoder records a substantially higher reconstruction error (MSE = 3.92 × 10¹?) and an accuracy of 6.09%, reflecting the difficulty of modelling complex and highly variable network traffic patterns. By combining both approaches, the proposed framework improves overall detection performance, achieving a higher accuracy of 13.55%, and demonstrates enhanced capability in detecting both volumetric and stealthy attacks, such as application-layer denial-of-service (DoS) traffic. Visualization of traffic behaviour further supports the analysis, revealing clearer separation between normal and anomalous flows when both models are integrated. These findings highlight the complementary strengths of statistical outlier detection and deep learning-based reconstruction, providing a practical foundation for adaptive and real-time anomaly monitoring in dynamic network environments