AI Reproducibility
aicr is a reproducibility harness for LLM deployment testing. It drives multiple model deployments in parallel, runs a set of prompts a configurable number of times across each, and records cryptographically signed, hashed evidence for every response.
Built to support the Clinical-AI Reproducibility Annex.
What it does
- Authenticates with Microsoft Entra ID (
DefaultAzureCredential) for Azure deployments, or an API key for OpenAI-compatible APIs. - Sends fully parameterised request bodies assembled from
config/deployments.json. - Captures the full raw JSON response for every run to its own file under
output/<timestamp>/runs/. - Computes a SHA-256 of the assistant message content as the identity key — produces identity groups automatically across runs.
- Scores list tasks (exact matches, coverage, hallucinations, ordering accuracy) and title-recall tasks.
- Writes
manifest.json,manifest.csv,summary_counts.csv,summary_pct.csv.
Get started
Download the desktop app or CLI binary and follow the Your first project guide. Experiments can be configured, run, and analysed either from the command line or in the desktop GUI.
Browse results
The Results viewer is hosted at ai-copyright-reproducibility.tauheed-elahee.com/viewer/.
Sections
| Section | What’s inside |
|---|---|
| Getting Started | Download binaries, build from source, run your first experiment |
| Project layout | Directory structure, configuration files, and deployment modes |
| Input | text.json, queries.json, prompts.json — what to ask and who to ask it to |
| Running | CLI reference, helper scripts, and troubleshooting |
| Desktop GUI | The aicr-gui desktop app — configure, run, and analyse experiments visually |
| Output | Output files, manifest columns, scoring algorithm, and interpreting results |
| Results viewer | Hosted Blazor viewer — load local or default manifests |
| Methodology | Hashing rationale, system_fingerprint drift, identity groups |
| Glossary | Definitions for every key term |