The sec-certs project (sec-certs.org) aims to be the one-stop shop to explore the security certification ecosystems of Common Criteria and FIPS 140. We aggregate and annotate certification data, enabling unified search, vulnerability investigation, trend analyses, side-by-side comparisons, notifications and other features.

This thesis aims to evaluate multiple LLM configurations for the use as chat agent interfacing certificate documentation available through sec-certs. For a successful thesis, the student will perform the following work:

• Familiarize themselves with the data available in sec-certs, common use cases of the tool and current LLM deployment.
• Devise an appropriate methodology for testing LLM chat interface in our context.
• Create a representative dataset of at least 50 questions covering multiple use-cases and test types to evaluate differences among LLMs used in sec-certs. The tests must be run on at least 10 different Common Criteria certificates.
• Implement a simple testing script/framework automating the evaluation of selected LLM configurations on the devised dataset.
• Run the evaluation and interpret the result, aiming to select the most appropriate configuration for upstream deployment.
• Submit a ready-to-merge pull request to integrate the new code directly into the upstream of sec-certs, ideally aiming for a successful merge before the defense.
• Optionally, conduct additional tests in cases where results of the main testing are inconclusive.

The thesis is a part of a long-term research project in cooperation of the Centre for Research on Cryptography and Security (CRoCS) at FI MUNI and Red Hat.