In this study, comparative analysis between state-of-the-art generative AI chatbots is done by employing survey-based analytics and performance measurements. This research focuses on the comparative analysis of generative AI chatbots through measuring response accuracy, contextual awareness, interaction capabilities, usability, response time, and user satisfaction levels. An extensive survey study was conducted involving 30 participants who have previous experience using AI chatbot services for educational, business, and general communication purposes. Data collection was performed using quantitative measures including survey questionnaires, and then data analysis was conducted through descriptive statistics and comparison of performance indicators. From the survey results, current AI chatbots offer more conversational intelligence and context-awareness than rule-based conversational systems. Among the examined chatbots, some chatbots managed to obtain a greater degree of user satisfaction because of their capabilities of producing coherent, contextually appropriate, and accurate responses. On the other hand, issues such as those of misinformation, inconsistency in complex reasoning, and contextual constraints were also discovered. It shows how significant it is to adopt an evaluation strategy that is based on users when examining conversational AI technology. This paper provides useful insights about the advantages and disadvantages of existing chatbot technologies.
Smutny, P., & Bojko, M. (2024). Comparative analysis of chatbots using large language models for web development tasks. Applied Sciences, 14(21), 10048.
Escobar-Grisales, D., Vásquez-Correa, J. C., & Orozco-Arroyave, J. R. (2024). Evaluation of effectiveness in conversations between humans and chatbots. Multimedia Tools and Applications.
Sallam, M. (2023). ChatGPT utility in healthcare: A comparative evaluation of AI chatbot responses. Journal of Medical Internet Research, 25.
Wangsa, K., Karim, S., Gide, E., & Elkhodr, M. (2024). A systematic review and comprehensive analysis of pioneering AI chatbot models. Future Internet, 16(7), 219.
Alabduljabbar, R. (2024). User-centric AI: Evaluating the usability of generative AI applications through user reviews on app stores. ResearchGate Preprint.
Xu, Y., & Liu, Y. (2023). Can chatbots satisfy me? A mixed-method comparative study of satisfaction with task-oriented chatbots. Computers in Human Behavior, 145.
Lin, Z., et al. (2024). Interpretable user satisfaction estimation for conversational systems with large language models. Microsoft Research Technical Report.
Zhang, H., & Chen, L. (2025). Beyond benchmarks: How users evaluate AI chat assistants. arXiv preprint arXiv:2603.25220.
Microsoft Research & Salesforce (2025). Analysis of conversational AI reliability and long-context performance degradation. AI Systems Report.
OpenAI. (2025). Understanding hallucinations in large language models. OpenAI Research Blog.
Brown, T. B., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS).
Vaswani, A., et al. (2017). Attention is all you need. NeurIPS.
OpenAI. (2023). GPT-4 technical report. OpenAI Technical Report.
Google DeepMind. (2023). Gemini technical overview. Google AI Research Documentation.
Anthropic. (2023). Claude: Constitutional AI and harmlessness training. Anthropic Technical Report.
Microsoft. (2023). Copilot: AI-powered productivity assistant overview. Microsoft Official Documentation.
Thant, K. S., & Tin, H. H. K. (2023). The impact of manual and automatic testing on software testing efficiency and effectiveness. Indian journal of science and research, 3(3), 88-93.
Generative AI, Chatbots, User-Centered Evaluation, Survey Analysis, Comparative Performance Assessment.
