# 🧬 LLM Bio-Model Verifier

A Streamlit web application for benchmarking large language models (LLMs) on the task of **phenotype extraction from biological literature**, followed by formal verification against Boolean attractor models.

Built as part of a diploma thesis at Masaryk University (MUNI).

---

## Overview

The tool automates the following pipeline:

```
Input documents (PDF / HTML / images)
        │
        ▼
  LLM Extraction  ──── system prompt + optional model variable context
        │
        ▼
  JSON phenotypes  (list of named variable states)
        │
        ▼
  Formal Verification  ──── biodivine_aeon attractor analysis
        │
        ▼
  Run stored to history  ──── SQLite index + JSON file
```

---

## Features

| Area | Capability |
|---|---|
| **Extraction** | Supports text (TXT, HTML), PDF (native or text-extracted), and images |
| **Multimodal** | Images sent as binary to vision-capable models; others skip images gracefully |
| **PDF handling** | Per-model override: auto-detect / always extract text / always send native PDF |
| **Model context** | Optionally injects AEON/BNET variable list into the prompt to anchor naming |
| **Verification** | Uses `biodivine_aeon` to compute attractors and match extracted phenotypes |
| **Background jobs** | Batch runs execute in background threads (up to 3 concurrent); cancellable |
| **Repetitions** | Each configuration can be repeated N times to test output consistency |
| **History** | SQLite-indexed run history with charts, LLM comparison tables, milestone grouping |
| **Failure diagnostics** | Classify unverified phenotypes as hallucination / misinterpretation / model discrepancy |
| **i18n** | Full Slovak / English UI toggle |

---

## Project Structure

```
VerifierApp/
├── app.py                        # Streamlit entry point
├── config.py                     # Paths, directory init, prompt & capability helpers
├── localisation_module.py        # All UI strings (SK + EN) and icon map
├── requirements.txt
│
├── core/
│   ├── extractor.py              # LLM call + prompt assembly + JSON parsing
│   ├── verifier.py               # biodivine_aeon attractor matching
│   ├── models.py                 # Model list fetching + capability registry
│   ├── job_manager.py            # Background thread pool (max 3 workers)
│   ├── runs_db.py                # SQLite index + precomputed summary tables
│   └── prompt_helper.py          # Prompt file utilities
│
├── ui/
│   ├── testing.py                # New Test Run page (single + batch + background)
│   ├── history.py                # History & Statistics page
│   └── data_management.py        # Model & Input Data Management page
│
└── data/                         # Runtime data (gitignored in practice)
    ├── inputs/                   # Input document groups (one subfolder per group)
    ├── models/                   # AEON/BNET model groups (one subfolder per group)
    ├── prompts/                  # System prompt files (base_prompt.txt is active)
    ├── runs/                     # JSON run results + runs_cache.db
    └── model_capabilities.json   # Per-model PDF override settings
```

---

## Installation

### Prerequisites

- Python 3.10+
- An [e-INFRA CZ LLM API](https://llm.ai.e-infra.cz) token (and/or a Google Gemini API token)

### Setup

```powershell
# 1. Clone / navigate to the project
cd VerifierApp

# 2. Create and activate a virtual environment
python -m venv .venv
.venv\Scripts\Activate.ps1

# 3. Install dependencies
pip install streamlit pandas plotly openai pydantic-ai python-dotenv biodivine_aeon pillow pypdf beautifulsoup4

# 4. Create a .env file with your API credentials
```

### `.env` file

Create a `.env` file in the `VerifierApp/` directory:

```env
E_INFRA_API_TOKEN=your_einfra_token_here
GEMINI_API_TOKEN=your_gemini_token_here   # optional
```

---

## Running the App

From the **project root** (one level above `VerifierApp/`):

```powershell
streamlit run VerifierApp/app.py
```

The app will open at `http://localhost:8501`.

---

## Usage Guide

### 1. Upload Data (`Model & Data Management`)

- **Input Documents** — upload PDF, HTML, TXT, or image files. HTML is auto-cleaned, images are resized to 1024px JPEG.
- **Math Models** — upload `.aeon` and/or `.bnet` files (one folder per model, e.g. `id-061`).
- **LLM Capabilities** — configure per-model PDF handling (auto / extract text / send native PDF). Settings are saved automatically on change.

### 2. Run a Benchmark (`New Test Run`)

**Single Run:**
1. Select an input folder and a reference model.
2. Select an LLM model.
3. Optionally edit the system prompt.
4. Toggle *Add model variable context to prompt* (on by default — provides variable names from the BNET/AEON file to the LLM).
5. Set milestone label and repetition count.
6. Click **Run Extraction & Verification** — runs in the background immediately.

**Batch Run:**
1. Select multiple LLMs to compare.
2. Map input folders to model folders (add as many pairs as needed).
3. Set repetitions and milestone, then click **Run Mapped Benchmark**.
4. All combinations run in background (max 3 concurrent). Navigate freely while they execute.

Background job status is visible in the sidebar badge and in the **Background Jobs** panel on the Testing page.

### 3. View Results (`History & Statistics`)

- **Statistics & Charts** — LLM comparison table (success rate, model coverage, avg tokens), bar charts, milestone summary.
- **Failure Diagnostics** — browse all unverified phenotypes; compare LLM output against attractor states; classify each as *Logical Hallucination*, *Contextual Misinterpretation*, or *Model Discrepancy*. Classifications are saved to `diagnostics_cache.json` automatically.
- **Detailed Runs** — inspect raw JSON output and token usage for any individual run.

> **Coverage metric:** both the overview tile and the table use the **mean of per-run coverage rates**, so both numbers are always consistent.

---

## Supported Models

The app fetches available models automatically from the configured API endpoints.  
Capability detection (vision / native PDF support) is based on model ID substring matching — see `core/models.py` → `_CAPABILITY_PATTERNS`. Overrides can be set per-model in the UI.

**e-INFRA CZ endpoint:** `https://llm.ai.e-infra.cz/v1/`  
**Google Gemini endpoint:** `https://generativelanguage.googleapis.com/v1beta/openai/`

---

## Data Storage

| Path | Contents |
|---|---|
| `data/runs/*.json` | One JSON file per run (authoritative source of truth) |
| `data/runs/runs_cache.db` | SQLite index (auto-rebuilt; safe to delete) |
| `data/runs/diagnostics_cache.json` | Failure classification annotations |
| `data/prompts/base_prompt.txt` | Active system prompt |
| `data/model_capabilities.json` | PDF override settings per model ID |

Run JSON files are the single source of truth. The SQLite cache can be deleted and rebuilt at any time via the **Rebuild full index** button in the History page.

---

## Localisation

The app supports **Slovak** (default) and **English**. Toggle via the sidebar selector.  
All strings live in `localisation_module.py` → `TRANSLATIONS`. LLM-facing prompt strings are always sent in English regardless of UI language.

---

## Known Issues / Limitations

- Background job state is **in-memory only** — if the Streamlit server restarts, active jobs are lost (completed run files are preserved).
- Model capability detection is heuristic (substring matching). New model deployments may need a manual override in the UI.
- `input_dir.iterdir()` order is OS-dependent, so multi-file prompt order is non-deterministic.

