Empirical Evaluation of Feature Selection Strategies for Interpretable and Robust Healthcare Machine Learning
Article Sidebar
Main Article Content
Abstract
Feature Section (FS) is a fundamental step in developing accurate, robust, and interpretable Machine Learning (ML) models, especially when working with high-dimensional structured datasets commonly found in healthcare, finance, and other data-intensive domains. Selecting the most relevant subset of features not only improves predictive performance but also enhances computational efficiency and model transparency. This paper presents a comprehensive and systematic study of widely adopted feature selection strategies, including filter, wrapper, embedded, model-agnostic methods, and dimensionality reduction techniques. Each category is examined through experimental evaluations across multiple classifiers and benchmark datasets to understand how different feature selection mechanisms affect model performance, interpretability, and stability. The results show that while filter methods, such as mutual information and chi-square, are computationally efficient and suitable for large-scale screening, wrapper and embedded methods achieve higher accuracy at the cost of increased computational demand. The model-agnostic approach, represented by Permutation Feature Importance (PFI), was employed to assess the relative contribution of each feature to model performance, thereby providing an interpretable and unbiased measure of feature relevance. Dimensionality reduction techniques, such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), further assist in managing multicollinearity, improving class separability, and enhancing model generalization by transforming features into more discriminative lower-dimensional spaces. Moreover, the study captures the trade-offs between accuracy, speed, and explainability, offering practical recommendations for researchers and practitioners to select suitable methods based on their specific context and computational resources. The findings underscore the importance of adaptive, context-aware feature selection pipelines, particularly in healthcare analytics, where interpretable and trustworthy models can improve diagnostic confidence and support effective decision-making.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published in JIWE are licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) License. Readers are allowed to
- Share — copy and redistribute the material in any medium or format under the following conditions:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use;
- NonCommercial — You may not use the material for commercial purposes;
- NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
References
[1] U. Stanczyk and G. Baron, "Does all or nothing always work best? In search of advantageous representation of attributes," Applied Sciences, vol. 16, no. 6, p. 2679, Mar. 2026, doi: 10.3390/app16062679.
[2] W. Cunha, A. Moreo, A. Esuli, F. Sebastiani, L. Rocha, and M. A. Goncalves, "A Noise-Oriented and Redundancy-Aware instance selection framework," ACM Transactions on Information Systems, vol. 43, no. 2, pp. 1-33, Nov. 2024, doi: 10.1145/3705000.
[3] J. M. Kernbach and V. E. Staartjes, "Foundations of Machine Learning-Based Clinical Prediction Modeling: Part II-Generalization and Overfitting," Machine Learning in Clinical Neuroscience: Foundations and Applications, pp. 15-21, Dec. 2021, doi: 10.1007/978-3-030-85292-4_3.
[4] F. Xiang, C. Xing, Y. Ai, and Z. Wang, "An adaptive-weighted multi-level feature evidence and redundancy penalty method," Engineering Applications of Artificial Intelligence, vol. 180, p. 115245, May 2026, doi: 10.1016/j.engappai.2026.115245.
[5] Li, X., Xiong, H., Li, X., "Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond," Knowledge and Information Systems, vol. 64, no. 12, pp. 3197-3234, Sep. 2022, doi: 10.1007/s10115-022-01756-8.
[6] P. Dhal and C. Azad, "A comprehensive survey on feature selection in the various fields of machine learning," Applied Intelligence, vol. 52, no. 4, pp. 4543-4581, Jul. 2021, doi: 10.1007/s10489-021-02550-9.
[7] T. Miller, I. Durlik, A. Lobodzinska, L. Dorobczynski, and R. Jasionowski, "AI in Context: Harnessing Domain Knowledge for Smarter Machine Learning," Applied Sciences, vol. 14, no. 24, p. 11612, Dec. 2024, doi: 10.3390/app142411612.
[8] C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong, "Interpretable machine learning: Fundamental principles and 10 grand challenges," Statistics Surveys, vol. 16, pp. 1-85, Jan. 2022, doi: 10.1214/21-ss133.
[9] C. Aliferis and G. Simon, "Overfitting, underfitting and general model Overconfidence and Under-Performance pitfalls and best practices in machine learning and AI," in Computers in Health Care, 2024, pp. 477-524, doi: 10.1007/978-3-031-39355-6_10.
[10] U. M. Khaire and R. Dhanalakshmi, "Stability of feature selection algorithm: A review," Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 4, pp. 1060-1073, Jun. 2019, doi: 10.1016/j.jksuci.2019.06.012.
[11] H. Wang, Q. Liang, J. T. Hancock, and T. M. Khoshgoftaar, "Feature selection strategies: a comparative analysis of SHAP-value and importance-based methods," Journal of Big Data, vol. 11, no. 1, Mar. 2024, doi: 10.1186/s40537-024-00905-w.
[12] D. Theng and K. K. Bhoyar, "Feature selection techniques for machine learning: a survey of more than two decades of research," Knowledge and Information Systems, vol. 66, no. 3, pp. 1575-1637, Dec. 2023, doi: 10.1007/s10115-023-02010-5.
[13] M. A. Bouke, A. Abdullah, J. Frnda, K. Cengiz, and B. Salah, "BukaGini: A Stability-Aware GiNI Index feature selection algorithm for robust model performance," IEEE Access, vol. 11, pp. 59386-59396, Jan. 2023, doi: 10.1109/access.2023.3284975.
[14] J. Wang, P. Xu, X. Ji, M. Li, and W. Lu, "Feature selection in machine learning for perovskite materials design and discovery," Materials, vol. 16, no. 8, p. 3134, Apr. 2023, doi: 10.3390/ma16083134.
[15] X. Zhang and H. Xiong, "Interpretable feature interaction via statistical self-supervised learning on tabular data," arXiv.org, Mar. 23, 2025. https://arxiv.org/abs/2503.18048
[16] A. Kodipalli, V. S. Devi, S. Guruvare, and T. Ismail, "Explainable AI-based feature importance analysis for ovarian cancer classification with ensemble methods," Frontiers in Public Health, vol. 13, Mar. 2025, doi: 10.3389/fpubh.2025.1479095.
[17] L. Lyu, N. Roy, H. Oosterhuis, and A. Anand, "Is interpretable machine learning effective at feature selection for Neural Learning-to-Rank?" in Lecture Notes in Computer Science, 2024, pp. 384-402, doi: 10.1007/978-3-031-56066-8_29.
[18] R. Moraffah, P. Sheth, S. Vishnubhatla, and H. Liu, "Causal feature selection for responsible machine learning," arXiv.org, Feb. 05, 2024. https://arxiv.org/abs/2402.02696
[19] M. A. Jahin, M. F. Mridha, N. Dey, and Md. J. Hossen, "Human-in-the-Loop feature selection using interpretable Kolmogorov-Arnold Network-Based double Deep Q-Network," IEEE Open Journal of the Computer Society, vol. 7, pp. 389-403, Jan. 2026, doi: 10.1109/ojcs.2026.3652986.
[20] S. S. Nagaraju, "Automation and Feature Selection Enhancement with Reinforcement Learning RL," arXiv.org, Mar. 15, 2025. https://arxiv.org/abs/2503.11991
[21] T. Zhao, J. Cabral, and G. Zhu, "A novel explainable artificial intelligence framework using knockoffs techniques with applications to sports analytics," Annals of Operations Research, Apr. 2025, doi: 10.1007/s10479-025-06575-y.
[22] T. Shaik, X. Tao, H. Xie, L. Li, N. Higgins, and J. D. Velasquez, "Towards transparent Deep Learning in Medicine: feature contribution and Attention Mechanism-Based Explainability," Human-Centric Intelligent Systems, Jun. 2025, doi: 10.1007/s44230-025-00104-7.
[23] M. C. Barbieri, B. I. Grisci, and M. Dorn, "Analysis and comparison of feature selection methods towards performance and stability," Expert Systems with Applications, vol. 249, p. 123667, Mar. 2024, doi: 10.1016/j.eswa.2024.123667.
[24] A. Lamsaf, R. Carrilho, J. C. Neves, and H. Proenca, "Causality, Machine Learning, and Feature Selection: a survey," Sensors, vol. 25, no. 8, p. 2373, Apr. 2025, doi: 10.3390/s25082373.
[25] X. Cheng, "A comprehensive study of feature selection techniques in machine learning models," Artificial Intelligence and Digital Technology, vol. 1, no. 1, pp. 65-78, Nov. 2024, doi: 10.70088/xpf2b276.
[26] A. N. Prasad, "Data quality and preprocessing," in Apress eBooks, 2024, pp. 109-223, doi: 10.1007/979-8-8688-1023-7_3.
[27] A. Demircioglu, "The effect of preprocessing filters on predictive performance in radiomics," European Radiology Experimental, vol. 6, no. 1, Sep. 2022, doi: 10.1186/s41747-022-00294-w.
[28] J. Maldonado, M. C. Riff, and B. Neveu, "A review of recent approaches on wrapper feature selection for intrusion detection," Expert Systems with Applications, vol. 198, p. 116822, Mar. 2022, doi: 10.1016/j.eswa.2022.116822.
[29] R. Jiao, B. H. Nguyen, B. Xue, and M. Zhang, "A survey on Evolutionary multiobjective feature selection in classification: Approaches, applications, and challenges," IEEE Transactions on Evolutionary Computation, vol. 28, no. 4, pp. 1156-1176, Jul. 2023, doi: 10.1109/tevc.2023.3292527.
[30] Y. Wang, "A comparative analysis of model agnostic techniques for explainable artificial intelligence," Research Reports on Computer Science, Aug. 2024, doi: 10.37256/rrcs.3220244750.