Reinforcement Learning Approach for Cybersecurity Threat Detection
Article Sidebar
Main Article Content
Abstract
Traditional intrusion detection systems (IDS) struggle to detect evolving cyber threats due to their reliance on static signatures and fixed decision boundaries. Existing machine learning-based approaches partially address this limitation but often fail to generalize to zero-day attacks and lack adaptability in dynamic network environments. To address these challenges, this paper proposes a reinforcement learning-based intrusion detection system (RL-IDS) that model’s detection as a sequential decision-making problem using flow-level telemetry. The framework is implemented on the CIC-IDS2017 dataset with an isolated zero-day partition within a custom OpenAI Gym environment, incorporating asymmetric reward design, curriculum learning, entropy annealing, and early stopping to train Q-learning, Deep Q-Network (DQN), and Proximal Policy Optimization (PPO) agents. Experimental results show that the PPO-based RL-IDS achieves an F1-score of 0.857 with less than 4% false positives on known attacks, outperforming both DQN and a 400-tree Random Forest baseline. More importantly, it detects 27.7% of previously unseen zero-day attacks (Heartbleed and Infiltration), where the Random Forest fails completely. The system also processes over 290,000 flows per second, demonstrating real-time feasibility. These results demonstrate that reinforcement learning enables a practical balance between accuracy, adaptability, and efficiency, making it a promising solution for next-generation intrusion detection systems.
Manuscript received: 31 Dec 2025 | Revised: 25 Apr 2026 | Accepted: 22 May 2026 | Published: 31 Jul 2026
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
M. Ahsan, N. Rifat, M. Chowdhury and R. Gomes, "Detecting Cyber Attacks: A Reinforcement Learning Based Intrusion Detection System," 2022 IEEE International Conference on Electro Information Technology (eIT), pp. 461-466, 2022.
DOI: https://doi.org/10.1109/eit53891.2022.9813892
H. Alavizadeh, H. Alavizadeh and J. Jang-Jaccard, "Deep Q-Learning Based Reinforcement Learning Approach for Network Intrusion Detection," Computers, vol. 11, no. 3, pp. 41, 2022.
DOI: https://doi.org/10.3390/computers11030041
H. Benaddi, K. Ibrahimi, A. Benslimane, M. Jouhari and J. Qadir, "Robust Enhancement of Intrusion Detection Systems Using Deep Reinforcement Learning and Stochastic Game," IEEE Transactions on Vehicular Technology, vol. 71, no. 10, pp. 11089-11102, 2022.
DOI: https://doi.org/10.1109/tvt.2022.3186834
G. Caminero, M. Lopez-Martin and B. Carro, "Adversarial environment reinforcement learning algorithm for intrusion detection," Computer Networks, vol. 159, pp. 96-109, 2019.
DOI: https://doi.org/10.1016/j.comnet.2019.05.013
S. Hussain, J. He, N. Zhu, F.R. Mughal, M.I. Hussain, A.D. Algarni, S. Ahmad, M.M. Zarie and A.A. Ateya, "An Adaptive Intrusion Detection System for WSN using Reinforcement Learning and Deep Classification," Arabian Journal for Science and Engineering, vol. 50, no. 15, pp. 12463-12477, 2025.
DOI: https://doi.org/10.1007/s13369-024-09769-x
M. Lopez-Martin, B. Carro and A. Sanchez-Esguevillas, "Application of deep reinforcement learning to intrusion detection for supervised problems," Expert Systems with Applications, vol. 141, pp. 112963, 2020.
DOI: https://doi.org/10.1016/j.eswa.2019.112963
F. Louati, F.B. Ktata and I. Amous, "Big-IDS: a decentralized multi agent reinforcement learning approach for distributed intrusion detection in big data networks," Cluster Computing, vol. 27, no. 5, pp. 6823-6841, 2024.
DOI: https://doi.org/10.1007/s10586-024-04306-9
M.A. Merzouk, J. Delas, C. Neal, F. Cuppens, N. Boulahia-Cuppens and R. Yaich, "Evading Deep Reinforcement Learning-based Network Intrusion Detection with Adversarial Attacks," Proceedings of the 17th International Conference on Availability, Reliability and Security, pp. 1-6, 2022.
DOI: https://doi.org/10.1145/3538969.3539006
S. Mohamed and R. Ejbali, "Deep SARSA-based reinforcement learning approach for anomaly network intrusion detection system," International Journal of Information Security, vol. 22, no. 1, pp. 235-247, 2023.
DOI: https://doi.org/10.1007/s10207-022-00634-2
S. Otoum, B. Kantarci and H. Mouftah, "Empowering Reinforcement Learning on Big Sensed Data for Intrusion Detection," ICC 2019 - 2019 IEEE International Conference on Communications (ICC), pp. 1-7, 2019.
DOI: https://doi.org/10.1109/icc.2019.8761575
A.M. Pasikhani, J.A. Clark and P. Gope, "Adversarial RL-Based IDS for Evolving Data Environment in 6LoWPAN," IEEE Transactions on Information Forensics and Security, vol. 17, pp. 3831-3846, 2022.
DOI: https://doi.org/10.1109/tifs.2022.3214099
K. Sethi, Y.V. Madhav, R. Kumar and P. Bera, "Attention based multi-agent intrusion detection systems using reinforcement learning," Journal of Information Security and Applications, vol. 61, pp. 102923, 2021.
DOI: https://doi.org/10.1016/j.jisa.2021.102923
B. Sharma, L. Sharma, C. Lal and S. Roy, "Explainable artificial intelligence for intrusion detection in IoT networks: A deep learning based approach," Expert Systems with Applications, vol. 238, pp. 121751, 2024.
DOI: https://doi.org/10.1016/j.eswa.2023.121751
E. Suwannalai and C. Polprasert, "Network Intrusion Detection Systems Using Adversarial Reinforcement Learning with Deep Q-network," 2020 18th International Conference on ICT and Knowledge Engineering (ICT&KE), pp. 1-7, 2020.
DOI: https://doi.org/10.1109/ictke50349.2020.9289884
H. Tan, L. Wang, D. Zhu and J. Deng, "Intrusion Detection Based on Adaptive Sample Distribution Dual-Experience Replay Reinforcement Learning," Mathematics, vol. 12, no. 7, pp. 948, 2024.
DOI: https://doi.org/10.3390/math12070948
S. Vadigi, K. Sethi, D. Mohanty, S.P. Das and P. Bera, "Federated reinforcement learning based intrusion detection system using dynamic attention mechanism," Journal of Information Security and Applications, vol. 78, pp. 103608, 2023.
DOI: https://doi.org/10.1016/j.jisa.2023.103608
W. Villegas-Ch, J. Govea, R. Gutierrez, A.M. Navarro and A. Mera-Navarrete, "Effectiveness of an Adaptive Deep Learning-Based Intrusion Detection System," IEEE Access, vol. 12, pp. 184010-184027, 2024.
DOI: https://doi.org/10.1109/access.2024.3512363
Y. Xia, S. Dong, T. Peng and T. Wang, "Wireless Network Abnormal Traffic Detection Method Based on Deep Transfer Reinforcement Learning," 2021 17th International Conference on Mobility, Sensing and Networking (MSN), pp. 528-535, 2021.
DOI: https://doi.org/10.1109/msn53354.2021.00083
B. Yang, M.H. Arshad and Q. Zhao, "Packet-Level and Flow-Level Network Intrusion Detection Based on Reinforcement Learning and Adversarial Training," Algorithms, vol. 15, no. 12, pp. 453, 2022.
DOI: https://doi.org/10.3390/a15120453
S. Yu, R. Zhai, Y. Shen, G. Wu, H. Zhang, S. Yu and S. Shen, "Deep Q-Network-Based Open-Set Intrusion Detection Solution for Industrial Internet of Things," IEEE Internet of Things Journal, vol. 11, no. 7, pp. 12536-12550, 2024.
DOI: https://doi.org/10.1109/jiot.2023.3333903
C.M. Chang, M.N. Al-Andoli, and C. Zheng, "Hybrid Phishing Detection Model: Integrating BERT with TF-IDF for Enhanced Email Security," International Journal on Robotics, Automation and Sciences, vol. 7, no. 3, pp. 43-48, 2025.
DOI: https://doi.org/10.33093/ijoras.2025.7.3.6
S. Mushtaq and M.M. Su'ud, "Detection of Malicious URLs A Deep Learning and Machine Learning Perspective," International Journal on Robotics Automation and Sciences, vol. 8, no. 1, pp. 1, 2026.