A Conceptual Framework on Development of Sign Language Chatbot for E-Commerce
Article Sidebar
Main Article Content
Abstract
This study proposes and validates a conceptual framework using a sign language (SL) chatbot in the e-commerce domain to improve accessibility. Currently, the use of SL is a less-explored area in online platforms. It is difficult for any communication-challenged person to interact online for buying products and services, using a chatbot, particularly for people using SL. The objective of this study is to introduce a novel hybrid architecture using SL Recognition with a conversational-based chatbot agent via a custom Application Programming Interface (API) in e-commerce platforms. Our work proposes a combined hybrid chatbot framework model using Convolutional Neural Network (CNN) and Natural Language Processing (NLP). Python libraries such as Keras, OpenCV, and MediaPipe frameworks were used to read the signs in the system. To test this study at this initial stage, a preliminary feasibility experiment was conducted. Purposive sampling has been used to select 8 participants familiar with American Sign Language (ASL), who were tested under various conditions, including different lighting and clothing. The SL recognition module’s initial performance data has been analyzed using precision, recall, and F1 scores to assess ASL recognition and achieved an accuracy of 98%. This whole work showcases a blueprint to develop inclusive e-commerce platforms to encourage accessibility.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published in JIWE are licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) License. Readers are allowed to
- Share — copy and redistribute the material in any medium or format under the following conditions:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use;
- NonCommercial — You may not use the material for commercial purposes;
- NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.
References
D. Bragg et al., “Sign language recognition, generation, and translation: An interdisciplinary perspective,” in Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, New York, NY, USA, pp. 16–31, 2019, doi:10.1145/3308561.3353774.
M. Al-Qurishi, T. Khalid, and R. Souissi, “Deep learning for sign language recognition: Current techniques, benchmarks, and open issues,” IEEE Access, vol. 9, pp. 126917–126951, 2021, doi:10.1109/ACCESS.2021.3110912.
D. W. Harahap, and L. Fitria, “Aplikasi chatbot berbasis web menggunakan metode dialogflow,” J-Icom - Jurnal Informatika Dan Teknologi Komputer, vol. 1, no. 1, pp. 6–13, 2020, doi:10.33059/j-icom.v1i1.2796.
S. Richardson, F. Jovanka, P. K. Zabrina, M. P. Adiati, and D. Rosman, “The consequences of digital concierge chatbots acceptance on hotel guest experience and satisfaction at IHG hotel and resort,” in 2023 International Conference on Digital Applications, Transformation & Economy (ICDATE), Miri, Malaysia, pp. 1–5, 2023, doi:10.1109/ICDATE58146.2023.10248751.
N. Ketkar, and J. Moolayil, “Convolutional neural networks,” in Deep Learning with Python: Learn best practices of deep learning models with PyTorch, Apress: Berkeley, CA, USA, pp. 197–242, 2021, doi:10.1007/978-1-4842-5364-9_6.
T. Zemcik, “A brief history of chatbots,” DEStech Transactions on Computer Science and Engineering, 2019, doi:10.12783/dtcse/aicae2019/31439.
K. Klein, and L. F. Martinez, “The impact of anthropomorphism on customer satisfaction in chatbot commerce: An experimental study in the food sector,” Electronic Commerce Research, vol. 23, no. 4, pp. 2789–2825, 2023, doi:10.1007/s10660-022-09562-8.
M. Li, and R. Wang, “Chatbots in e-Commerce: The effect of chatbot language style on customers’ continuance usage intention and attitude toward brand,” Journal of Retailing and Consumer Services, vol. 71, Art. no. 103209, 2023, doi:10.1016/j.jretconser.2022.103209.
P. Sindhu, and K. Bharti, “Influence of chatbots on purchase intention in social commerce,” Behaviour & Information Technology, pp. 1–22, 2023, doi:10.1080/0144929x.2022.2163188.
J. M. Solis-Quispe, K. M. Quico-Cauti, and W. Ugarte, “Chatbot to simplify customer interaction in e-Commerce channels of retail companies,” in HCI International 2021, pp. 561–570, 2021, doi:10.1007/978-3-030-68285-9_52.
M. M. Khan, “Development of an e-commerce sales chatbot,” in 2020 IEEE 17th International Conference on Smart Communities: Improving Quality of Life Using ICT, IoT and AI (HONET), Charlotte, NC, USA, pp. 173–176, 2020, doi:10.1109/HONET50430.2020.9322667.
H. Mostaq et al., “Research and development of an e-commerce with sales chatbot,” in 2022 IEEE World AI IoT Congress (AIIoT), pp. 557–564, 2022, doi:10.1109/aiiot54504.2022.9817272.
V. Oguntosin, and A. Olomo, “Development of an e-commerce chatbot for a university shopping mall,” Applied Computational Intelligence and Soft Computing, vol. 2021, no. 1, 2021, doi:10.1155/2021/6630326.
A. Nursetyo, D. R. I. M. Setiadi, and E. R. Subhiyakto, “Smart chatbot system for e-commerce assistance based on AIML,” in 2018 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), Yogyakarta, Indonesia, pp. 641–645, 2018, doi:10.1109/ISRITI.2018.8864349.
J.-S. Chen, T.-T. Le, and D. Florence, “Usability and responsiveness of artificial intelligence chatbot on online customer experience in e-retailing,” International Journal of Retail & Distribution Management, vol. 49, no. 11, pp. 1512–1531, 2021, doi:10.1108/IJRDM-08-2020-0312.
R. Parkar et al., “AI and web-based interactive college enquiry chatbot,” in 2021 13th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), Pitesti, Romania, pp. 1–5, 2021, doi:10.1109/ECAI52376.2021.9515065.
S. Meshram et al., “College enquiry chatbot using Rasa framework,” in 2021 Asian Conference on Innovation Technology (ASIANCON), Pune, India, pp. 1–8, 2021, doi:10.1109/ASIANCON51346.2021.9544650.
M. G. C. P. Srivastava, A. Chakraborty, S. Ghosh, and H. Raj, “Development of information technology telecom chatbot: An artificial intelligence and machine learning approach,” in 2021 2nd International Conference on Intelligent Engineering and Management (ICIEM), London, UK, pp. 216–221, 2021, doi:10.1109/ICIEM51511.2021.9445354.
World Health Organization, “Deafness and hearing loss,” Mar. 2023. [Online]. Available: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss.
A. Pardasani, A. K. Sharma, S. Banerjee, V. Garg, and D. S. Roy, “Enhancing the ability to communicate by synthesizing American sign language using image recognition in a chatbot for differently abled,” in 2018 7th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India, pp. 529–532, 2018, doi:10.1109/ICRITO.2018.8748590.
C. Apuzzo, and G. Burresi, “Designing accessible chatbots for deaf people,” in 2022 11th Mediterranean Conference on Embedded Computing (MECO), pp. 1–4, 2022, doi:10.1109/MECO55406.2022.9797093.
M. Mohandes, M. Deriche, and J. Liu, “Image-based and sensor-based approaches to Arabic sign language recognition,” IEEE Transaction on Human-Machine System, vol. 44, no. 4, pp. 551–557, 2014, doi:10.1109/THMS.2014.2318280.
R. Sreemathy, M. Turuk, J. Jagdale, A. Agarwal, and V. Kumar, “Indian sign language interpretation using convolutional neural networks,” in 2023 10th International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India, pp. 789–794, 2023, doi:10.1109/SPIN57001.2023.10116204.
A. Abul et al., “CNN based feature extraction and classification for sign language,” Multimedia Tools and Applications, vol. 80, no. 2, pp. 3051–3069, 2021, doi:10.1007/S11042-020-09829-Y.
N. Singla, M. Taneja, N. Goyal, and R. Jindal, “Feature fusion and multi-stream CNNs for scale-adaptive multimodal sign language recognition,” in 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, pp. 1266–1273, 2023, doi:10.1109/ICACCS57279.2023.10112915.
K. Bantupalli, and Y. Xie, “American sign language recognition using deep learning and computer vision,” in 2018 IEEE International Conference on Big Data, Seattle, WA, USA, pp. 4896–4899, 2018, doi:10.1109/BigData.2018.8622141.
D. Tasmere, and B. Ahmed, “Hand gesture recognition for Bangla sign language using deep convolution neural network,” in 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh, pp. 1–5, 2020, doi:10.1109/STI50764.2020.9350438.
T. F. Dima, and M. E. Ahmed, “Using YOLOv5 algorithm to detect and recognize American sign language,” in 2021 International Conference on Information Technology (ICIT), Amman, Jordan, pp. 603–607, 2021, doi:10.1109/ICIT52682.2021.9491642.
A. Sharma, A. Mittal, S. Singh, and V. Awatramani, “Hand gesture recognition using image processing and feature extraction techniques,” Procedia Computer Science, vol. 173, pp. 181–190, 2020, doi:10.1016/j.procs.2020.06.022.
W. Feng, Z. Zhang, T. He, and C. Lee, “AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove,” Nature Communications., vol. 12, no. 1, p. 5378, 2021, doi:10.1038/S41467-021-25637-W.
M. M. Damaneh, F. Mohanna, and P. Jafari, “Static hand gesture recognition in sign language based on convolutional neural network with feature extraction method using ORB descriptor and Gabor filter,” Expert Systems with Applications, vol. 211, p. 118559, 2023, doi:10.1016/j.eswa.2022.118559.
A. Wadhawan and P. Kumar, “Deep learning-based sign language recognition system for static signs,” Neural Computing and Applications, vol. 32, no. 18, pp. 7957–7968, 2020, doi:10.1007/s00521-019-04691-y.
M. M. Rahman et al., “A new benchmark on American sign language recognition using convolutional neural network,” in 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh, pp. 1–6, 2019, doi:10.1109/STI47673.2019.9067974.
M. Al-Hammadi et al., “Hand gesture recognition for sign language using 3DCNN,” IEEE Access, vol. 8, pp. 79491–79509, 2020, doi:10.1109/ACCESS.2020.2990434.
Q. M. Areeb, M. Maryam, M. Nadeem, R. Alroobaea, and F. Anwer, “Helping hearing-impaired in emergency situations: A deep learning-based approach,” IEEE Access, vol. 10, pp. 8502–8517, 2022, doi:10.1109/ACCESS.2022.3142918.
P. A. Rodriguez-Correa, A. Valencia-Arias, O. N. Patino-Toro, Y. Oblitas Diaz, and R. Teodori De la Puente, “Benefits and development of assistive technologies for deaf people's communication: A systematic review,” Frontiers in Education, vol. 8, 2023, doi:10.3389/feduc.2023.1112440.
F. H. F. Botelho, “Accessibility to digital technology: Virtual barriers, real opportunities,” Assistive Technology, vol. 33, no. sup1, pp. 27–34, 2021, doi:10.1080/10400435.2021.1945705.
S. Svensson et al., “The emerging ‘right to repair’ legislation in the EU and the US,” in Proceedings of the Going green - Care Innovation 2018, Vienna, Austria, 2018. https://api.semanticscholar.org/CorpusID:211235579.
G. Astbrink, and W. Tibben, “The role of public procurement in improving accessibility to ICT,” Telecommunications Journal of Australia, vol. 63, no. 2, 2013, doi:10.7790/tja.v63i2.412.
A. Foley, and B. A. Ferri, “Technology for people, not disabilities: Ensuring access and inclusion,” Journal of Research in Special Educational Needs, vol. 12, no. 4, pp. 192–200, 2012, doi:10.1111/j.1471-3802.2011.01230.x.
D. T. S. Warnars, L. P. Putra, H. L. H. S. Warnars, and W. H. Utomo, “Intelligent e-commerce for special needs,” in 2019 7th International Conference on Cyber and IT Service Management (CITSM), Jakarta, Indonesia, pp. 1–5, 2019, doi:10.1109/CITSM47753.2019.8965341.
G. M. Carmo, D. B. Paiva, and M. I. Cagnin, “Incorporating accessibility in web interface development for deaf users,” in Proceedings of the XX Brazilian Symposium on Software Quality (SBQS), New York, NY, USA, pp. 1–10, 2021, doi:10.1145/3493244.3493246.
N. Singla, “American sign language letter recognition from images using CNN,” in 2023 2nd International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT), Trichirappalli, India, pp. 1–9, 2023, doi:10.1109/ICEEICT56924.2023.10156922.
S. Janagiraman, P. Mithun, J. Sandhiya, and S. Priyanka, “Enabling inclusive communication: A sign language chatbot for diverse audiences,” in 2025 International Conference on Automation and Computation (AUTOCOM), IEEE, pp. 33–39, 2025, doi:10.1109/AUTOCOM64127.2025.10957203.
F. Fang, Y. A, and S. Wang, “SignLLM: Sign language production large language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 2500–2509, 2025, doi:10.48550/arXiv.2405.10718.
R. Zabin, M. R. T. Hossain, K. F. Haque, and K. M. Rumman, “SignNet-Nano: Efficient sign language recognition for real-time edge deployment,” in 2025 2nd International Conference on Computing, Applications and Systems (COMPAS), Kushtia, Bangladesh, Oct. 2025, doi:10.13140/RG.2.2.35509.54249.
S. Mukherjee, A. Chatterjee, K. Chandrakar, and S. Saxena, “Artificial Intelligence powered chatbots for the speech disabled and hearing impaired using Feedforward Neural Network,” in 2024 1st International Conference on Advances in Computing, Communication and Networking (ICAC2N), Greater Noida, India, pp. 1071–1075, 2024, doi:10.1109/ICAC2N63387.2024.10894803.
M. Hassan, K. Assaleh, and T. Shanableh, “User-dependent sign language recognition using motion detection,” in 2016 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, pp. 852–856, 2016, doi:10.1109/CSCI.2016.0165.
C. Serban, and F. Bota, “A framework for software fault prediction using neural networks,” in Modelling and Development of Intelligent Systems (MDIS 2019), D. Simian and L. Stoica, Eds. Cham: Springer, 2020, pp. 137–148, doi:10.1007/978-3-030-39237-6_12.
M. Tomanek, R. Cermak, and Z. Smutny, “A framework for web development projects based on project management and agile development principles,” 10th European Conference on Management Leadership and Governance (ECMLG), 2014, doi:10.13140/2.1.1262.4165.
W. Utomo, Y. Suhanda, H. Ar-Rasyid, and A. Dharmalau, “Indonesian language sign detection using Mediapipe with Long Short-Term Memory (LSTM) algorithm,” Journal of Informatics and Web Engineering, vol. 4, no. 3, pp. 245–258, 2025, doi:10.33093/jiwe.2025.4.3.15.
Z. J. Wang et al., “CNN explainer: Learning convolutional neural networks with interactive visualization,” in IEEE Transactions on Visualization and Computer Graphics, vol. 27, no. 2, pp. 1396–1406, 2021, doi:10.1109/TVCG.2020.3030418.
J.-R. Lee, K.-W. Ng, and Y.-J. Yoong, “Face and facial expressions recognition system for blind people using ResNet50 architecture and CNN,” Journal of Informatics and Web Engineering, vol. 2, no. 2, pp. 284–298, 2023, doi:10.33093/jiwe.2023.2.2.20
M. Lamothe, Y.-G. Gueheneuc, and W. Shang, “A systematic review of API evolution literature,” ACM Computing Surveys, vol. 54, no. 8, p. 171, 2021, doi:10.1145/3470133.