As chatbots become the primary means of customer communication on the web, ensuring their correctness and consistency across updates poses a significant challenge. Manual testing of chatbot conversations is time-consuming, difficult to reproduce, and impractical on a large scale. This thesis addresses this problem by designing and implementing a Python library for the automated regression and large-scale testing of web-based chatbots. The library serves as the core of the work: it provides a reusable, general-purpose API for defining structured test cases organized into categories and subcategories, executing them in parallel against a live chatbot. The results are evaluated using both deterministic exact-match comparisons and semantic evaluations powered by large language models (LLMs). Packaging the core logic as a standalone library makes the solution broadly applicable beyond any single application - it can be integrated into CI/CD pipelines, scripted workflows, or other tools.
A desktop application will be developed on top of the library as a proof of concept to demonstrate its capabilities through a graphical interface for test case management, execution, and result visualization. The library covers the full testing lifecycle - from test case management and configurable execution through result evaluation to statistical reporting and export. To address the inherent uncertainty of automated evaluation, a verification mechanism is included: after each test run, a random sample of results is presented to the user for manual review, from which an overall correctness confidence score is derived.
An important part of the thesis is the verification and validation of the evaluator functionality: the library and its desktop application will be assessed through a non-trivial number of conversations of various types and topics, demonstrating its suitability for production chatbot testing on large amounts of test cases. The work will involve analyzing existing approaches to chatbot and conversational AI testing, designing the architecture of the library and the application, implementing the core functionality, and evaluating its effectiveness on a real-world chatbot deployments. The practical outcome will be a reusable Python library for automated chatbot evaluation and a standalone, locally-running desktop application that demonstrates the functionality of the library, making automated, repeatable, and large-scale chatbot testing accessible.
Source code availability: The source code of the library and the demo UI application showcasing its usage will be uploaded to IS as two separate projects.
License: Both projects will be released under a custom (proprietary) license that permits free use exclusively for educational purposes, personal projects, and academic research. Any commercial use, use within employment, internal business use, incorporation into products or services, offering as SaaS, redistribution, creation of derivative works for profit, use for training AI/ML models, or any other use beyond what is explicitly permitted requires prior written consent from the author. The software is provided "as is," without any warranties.