IMPLEMENTATION OF RETRIEVAL-AUGMENTED GENERATION BASED CHATBOT FOR EXPORT COMPANY CUSTOMER SERVICE AUTOMATION
DOI:
https://doi.org/10.34288/jri.v8i4.590Keywords:
Retrieval Augmented Generation, Chatbot, Customer Service, Export Company, DeepEval, Botika Agentic PlatformAbstract
This study discusses the issue of customer service in export companies, which is still limited in terms of service availability and the ability to handle recurring inquiries from international customers. Limitations in human resources result in responses to customer inquiries not being performed optimally, especially outside of operating hours. This study aims to design, implement, and evaluate a Retrieval-Augmented Generation (RAG)-based chatbot to support the automation of customer service in export companies using the Botika Agentic Platform. The study contributes by implementing and quantitatively evaluating a RAG-based chatbot on a Platform as a Service (PaaS) environment using the DeepEval framework with the LLM-as-a-Judge approach. The methods used in this study include four stages, namely data collection through interviews and analysis of company documents, system design, implementation through the Botika platform, and testing using the DeepEval framework with the LLM-as-a-Judge approach. The testing dataset consists of 93 questions covering four knowledge base domains, namely company profile, export and shipping procedures, partnership information, and product information. Evaluation results indicate that the chatbot achieved an average Answer Relevancy score of 0.98, Faithfulness of 1.00, and Contextual Relevancy of 0.88. The system performs well, with high Answer Relevancy and Faithfulness scores, while Contextual Relevancy is relatively lower, particularly in the product information domain. This is due to the dense structure of product documents, which contain many product variations within a single document, thereby affecting the accuracy of the retrieval process. Overall, the proposed RAG-based chatbot demonstrates its potential to support customer service automation in export companies by providing accurate and contextually grounded responses, while highlighting the importance of knowledge base organization for improving retrieval performance.
Downloads
References
Abel Uzoka, Emmanuel Cadet, & Pascal Ugochukwu Ojukwu. (2024). Leveraging AI-Powered chatbots to enhance customer service efficiency and future opportunities in automated support. Computer Science & IT Research Journal, 5(10), 2485–2510. https://doi.org/10.51594/csitrj.v5i10.1676
Denis Rothman. (2024). RAG-Driven Generative AI: Build custom retrieval augmented generation pipelines with LlamaIndex, Deep Lake, and Pinecone. Packt Publishing Ltd.
Es, S., James, J., Espinosa-Anke, L., Schockaert, S., & Gradients, E. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation. In https://aclanthology.org/2024.eacl-demo.16.pdf. https://doi.org/doi: 10.18653/v1/2024.eacl-demo.16
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., … Wang, H. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. Retrieved from http://arxiv.org/abs/2312.10997
Hariyadi, I. N. R., & Soraya, D. U. (2025). JEPIN (Jurnal Edukasi dan Penelitian Informatika). Https://Jurnal.Untan.Ac.Id/Index.Php/Jepin/Article/View/101635, 11(3), 442–447. https://doi.org/10.26418/jp.v11i3.101635
Jeffrey Ip. (2025). RAG Evaluation Metrics: Assessing Answer Relevancy, Faithfulness, Contextual Relevancy, And More - Confident AI. Retrieved May 25, 2026, from https://www.confident-ai.com/blog/rag-evaluation-metrics-answer-relevancy-faithfulness-and-more website: https://www.confident-ai.com/blog/rag-evaluation-metrics-answer-relevancy-faithfulness-and-more
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., … Fung, P. (2023, December 31). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, Vol. 55. Association for Computing Machinery. https://doi.org/10.1145/3571730
Khasanova Zafar kizi, M., & Suh, Y. (2025). Design and Performance Evaluation of LLM-Based RAG Pipelines for Chatbot Services in International Student Admissions. Electronics (Switzerland), 14(15). https://doi.org/10.3390/electronics14153095
Klesel, M., & Wittmann, H. F. (2025). Retrieval-Augmented Generation (RAG). Business and Information Systems Engineering, 67(4), 551–561. https://doi.org/10.1007/s12599-025-00945-3
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Retrieved from https://github.com/huggingface/transformers/blob/master/
Lin, X., Wang, X., Shao, B., & Taylor, J. (2024). How Chatbots Augment Human Intelligence in Customer Services: A Mixed-Methods Study. Journal of Management Information Systems, 41(4), 1016–1041. https://doi.org/10.1080/07421222.2024.2415773
Marcineková, K., Sujová, A. J., & Ďurica, R. (2025). Implementing AI Chatbots in Customer Service Optimization—A Case Study in Micro-Enterprise. Information (Switzerland), 16(12). https://doi.org/10.3390/info16121078
Muhammad Abdul Gofur, & Akbar, M. (2024). Prototipe chatbot pada aplikasi e-commerce berbasis web (studi kasus: toko seni gypsum). Jurnal RESTIKOM : Riset Teknik Informatika Dan Komputer, 6, 240–250. https://doi.org/10.52005/restikom.v6i2.338
Nathalie, J., Paramita, S., Komunikasi, H., Di, B., Ekspor, P., Kasus, S., … Logam, D. (2025). PT Sinar Dunia Logam). https://doi.org/10.24912/pr.v9i2.33186
Okonkwo, C. W., & Ade-Ibijola, A. (2021, January 1). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, Vol. 2. Elsevier B.V. https://doi.org/10.1016/j.caeai.2021.100033
Ozturk, O. (2024). The Impact of AI on International Trade: Opportunities and Challenges. Economies, 12(11). https://doi.org/10.3390/economies12110298
Pria Purnama, A., & Akbar, M. (2026). Pengembangan Chatbot Pintar Untuk Layanan Konsultasi Hukum Menggunakan OpenAI GPT-4. SMARTICS Journal. https://doi.org/10.21067/smartics.v12i1.13817
Pujiono, I., Agtyaputra, I. M., & Ruldeviyani, Y. (2024). IMPLEMENTING RETRIEVAL-AUGMENTED GENERATION AND VECTOR DATABASES FOR CHATBOTS IN PUBLIC SERVICES AGENCIES CONTEXT. JITK (Jurnal Ilmu Pengetahuan Dan Teknologi Komputer), 10(1), 216–223. https://doi.org/10.33480/jitk.v10i1.5572
Shahriar, S., Lund, B. D., Mannuru, N. R., Arshad, M. A., Hayawi, K., Bevara, R. V. K., … Batool, L. (2024). Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency. Applied Sciences (Switzerland), 14(17). https://doi.org/10.3390/app14177782
Yang, R., Fu, M., Tantithamthavorn, C., Arora, C., Vandenhurk, L., & Chua, J. (2025). RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice. Journal of Systems and Software, 226. https://doi.org/10.1016/j.jss.2025.112436
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., … Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Retrieved from http://arxiv.org/abs/2306.05685
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Jenri Dewany Widya Sunandar, Mutaqin Akbar
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The Jurnal Riset Informatika has legal rules for accessing digital electronic articles uunder a Creative Commons Attribution-NonCommercial 4.0 International License . Articles published in Jurnal Riset Informatika, provide Open Access, for the purpose of scientific development, research, and libraries.
