Calibrated Probabilistic Predictions for Phishing Email Detection Using Sentence-BERT with Machine Learning Models
Article Sidebar
Main Article Content
Abstract
Phishing email detection poses a critical challenge in cybersecurity, and addressing it requires integrating calibrated probabilistic predictions with advanced semantic embeddings from Sentence-BERT (SBERT). Phishing remains a persistent threat, exploiting deceptive emails to obtain sensitive information. Although many detection models focus on maximizing accuracy, they often disregard probability calibration, which is essential for trustworthy decision-making. To improve reliability, the all-mpnet-base-v2 SBERT model was employed to capture deep semantic representations of emails, strengthening the distinction between phishing and legitimate messages. Nine machine learning and deep learning classifiers, including tree-based ensembles (Extra Trees, Random Forest, LightGBM), linear (Logistic Regression), probabilistic (Gaussian Naïve Bayes), instance-based (k-Nearest Neighbors), neural networks (Multi-Layer Perceptron, Gated Recurrent Unit), and kernel-based (Support Vector Machine) were tested on a balanced dataset enhanced with SMOTE. Evaluation metrics included accuracy, precision, recall, F1-score, and the Brier score, which measures calibration quality. Results show that although most classifiers achieved strong accuracy, their calibration performance varied. LightGBM, MLP, GRU, and SVM produced superior calibration with lower Brier scores, delivering more reliable probability estimates for phishing risk assessment. Conversely, Gaussian Naïve Bayes showed weaker calibration, reflecting the limitations of conditional independence assumptions with complex embeddings. Overall, the findings highlight the importance of combining robust semantic feature extraction with calibrated probabilistic predictions to improve both effectiveness and trustworthiness in phishing detection systems. This integrated approach strengthens cybersecurity operations by enabling more nuanced, confidence-aware decision-making.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published in JIWE are licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) License. Readers are allowed to
- Share — copy and redistribute the material in any medium or format under the following conditions:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use;
- NonCommercial — You may not use the material for commercial purposes;
- NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
References
[1] M. A. Daniel, S.-C. Chong, L.-Y. Chong, and K.-K. Wee, "Optimising phishing detection: A comparative analysis of machine learning methods with feature selection," Journal of Informatics and Web Engineering, vol. 4, no. 1, pp. 200-212, Feb. 2025, doi: 10.33093/jiwe.2025.4.1.15.
[2] T. S. Tajamul and K. Aman, "Unveiling the efficacy of AI-based algorithms in phishing attack detection," Journal of Informatics and Web Engineering, vol. 3, no. 2, pp. 116-133, Jun. 2024, doi: 10.33093/jiwe.2024.3.2.9.
[3] H. Faotu and O. N. Asheshemi, "Human vulnerabilities in cybersecurity: Analyzing social engineering attacks and AI-driven machine learning countermeasures," Journal of Science and Technology, vol. 30, no. 1, pp. 72-84, Dec. 2025, doi: 10.20428/jst.v30i1.2597.
[4] F. Serpa and D. V. Voinea, "Synthetic media: Deepfakes, AI-generated content, and authenticity in the digital society," Encyclopedia, vol. 6, no. 8, p. 175, Aug. 2026, doi: 10.3390/encyclopedia6080175.
[5] N. Fatin, N. M. Nazlan, S. Fadzlun, and M. Salleh, "SpoofSentry: Email spoofing detection tool using domain-based authentication," Applied Information Technology and Computer Science, vol. 7, no. 1, pp. 1165-1185, Jul. 2026, doi: 10.30880/aitcs.2026.07.01.063.
[6] A. Wong, A. Abuadbba, M. Almashor, and S. Kanhere, "PhishClone: Measuring the efficacy of cloning evasion attacks," Sep. 2022, [Online]. Available: http://arxiv.org/abs/2209.01582.
[7] A. Basit, M. Zafar, and X. Liu, "A comprehensive survey of AI-enabled phishing attacks detection techniques," Telecommunication Systems, vol. 76, no. 1, pp. 139-154, Jan. 2021.
[8] R. O. Lane, "A comprehensive review of classifier probability calibration metrics," 2025.
[9] M. K. Moizuddin, M. Kabeer, and M. Misbahuddin, "Cyber-phishing analysis offering cyber security for social networks," in Proc. 2024 IEEE Int. Conf. Blockchain and Distributed Systems Security (ICBDS), Pune, India: IEEE, Oct. 2024.
[10] H. Ghalechyan, E. Israyelyan, A. Arakelyan, G. Hovhannisyan, and A. Davtyan, "Phishing URL detection with neural networks: An empirical study," Scientific Reports, vol. 14, no. 1, Dec. 2024, doi: 10.1038/s41598-024-74725-6.
[11] S. Shahriar, A. Mukherjee, and O. Gnawali, "Improving phishing detection via psychological trait scoring," 2022.
[12] T. Jiang, S. Huang, Z. Luan, D. Wang, and F. Zhuang, "Scaling sentence embeddings with large language models," Jul. 2023, [Online]. Available: http://arxiv.org/abs/2307.16645.
[13] S. Atawneh and H. Aljehani, "Phishing email detection model using deep learning," Electronics, vol. 12, no. 20, Oct. 2023, doi: 10.3390/electronics12204261.
[14] I. AbdulNabi and Q. Yaseen, "Spam email detection using deep learning techniques," in Proc. Procedia Computer Science, Elsevier B.V., 2021, pp. 853-858, doi: 10.1016/j.procs.2021.03.107.
[15] H. Milner and M. Baron, "Establishing an optimal online phishing detection method: Evaluating topological NLP transformers on text message data," Journal of Data Science and Intelligent Systems, vol. 2, no. 1, pp. 37-45, Jul. 2023, doi: 10.47852/bonviewjdsis32021131.
[16] F. S. Alsubaei, A. A. Almazroi, and N. Ayub, "Enhancing phishing detection: A novel hybrid deep learning framework for cybercrime forensics," IEEE Access, vol. 12, pp. 8373-8389, 2024, doi: 10.1109/ACCESS.2024.3351946.
[17] A. Ullah, R. A. Shah, S. A. Nawaz, N. Ahmad, and M. H. Malik, "Enhancing phishing detection, leveraging deep learning techniques," JCBI, Feb. 2024.
[18] A. O. Balogun et al., "Improving the phishing website detection using empirical analysis of Function Tree and its variants," Heliyon, vol. 7, no. 7, Jul. 2021, doi: 10.1016/j.heliyon.2021.e07437.
[19] G. S. Nayak, B. Muniyal, and M. C. Belavagi, "Enhancing phishing detection: A machine learning approach with feature selection and deep learning models," IEEE Access, vol. 13, pp. 33308-33320, 2025, doi: 10.1109/ACCESS.2025.3543738.
[20] Q. Qi, Z. Wang, Y. Xu, Y. Fang, and C. Wang, "Enhancing phishing email detection through ensemble learning and undersampling," Applied Sciences, vol. 13, no. 15, Aug. 2023, doi: 10.3390/app13158756.
[21] B. Lim, R. Huerta, A. Sotelo, A. Quintela, and P. Kumar, "EXPLICATE: Enhancing phishing detection through explainable AI and LLM-powered interpretability," Mar. 2025, [Online]. Available: http://arxiv.org/abs/2503.20796.
[22] P. An, R. Shafi, T. Mughogho, and O. A. Onyango, "Multilingual email phishing attacks detection using OSINT and machine learning," Jan. 2025, [Online]. Available: http://arxiv.org/abs/2501.08723.
[23] M. A. Adebowale, K. T. Lwin, and M. A. Hossain, "Intelligent phishing detection scheme using deep learning algorithms," Journal of Enterprise Information Management, vol. 36, no. 3, pp. 747-766, Apr. 2023, doi: 10.1108/JEIM-01-2020-0036.
[24] F. Khalid, M. Yaseen, G. Rahman, N. Mazhar, M. A. Nauman, and A. Mustapha, "Hybrid feature learning with TF-IDF and SBERT for ambiguous requirement classification," International Journal of Advanced Computer Science and Applications, vol. 17, no. 4, p. 570, Apr. 2026.
[25] N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks," [Online]. Available: https://github.com/UKPLab/.
[26] S. Chakraborty, "Phishing email detection," Kaggle, accessed Sep. 8, 2025. [Online]. Available: https://www.kaggle.com/datasets/subhajournal/phishingemails.