--- license: mit language: - ru tags: - medical - clinical-nlp - llm-evaluation - diagnosis - triage - benchmark pretty_name: 'Beyond Majority Voting: A New Framework for Evaluating AI Diagnostic Systems Against Expert Consensus' --- # Beyond Majority Voting — Evaluation Harness Evaluation code and aggregated results for the paper **“Beyond Majority Voting: A New Framework for Evaluating AI Diagnostic Systems Against Expert Consensus”** (Kopanichuk et al., preprint submitted to *Scientific Reports*, 2026). The harness implements the paper’s **relative** diagnostic-quality metrics — **RPAD** (Relative Precision of Algorithmic Diagnostics) and **RRAD** (Relative Recall of Algorithmic Diagnostics). Instead of scoring a model against a single "gold" label, it compares the model against a panel of physicians and normalizes that agreement by the agreement observed *among the experts themselves*. A value above `1.0` means the model agrees with the panel at least as well as the experts agree with one another. > ⚠️ **Data note (PHI).** This repository contains **only code and aggregated > results**. The raw patient↔physician dialogues contain personal health > information and are **not** included here. The **de-identified** dataset — > released under the written informed consent for open-access publication > described in the paper — is available at > **https://huggingface.co/datasets/kopan/med-eval-360**. ## Method - **Setup.** `n = 360` Russian-language telemedicine dialogues. A panel of `z = 7` resident-physician experts and each evaluated LLM independently produce a *bag* of up to `k = 3` diagnoses per case (and, separately, a routing / specialty recommendation). - **Pairwise precision & recall** (`src/pair_metrics.py`, eqs. 1–2): `P_AE@k` and `R_AE@k` between the algorithm `A` and an expert `E`, built from the multiplicity function **μ** (eq. 9) and the characteristic function **χ** (eq. 10) implemented in `src/characteristic_functions.py`. - **Diagnosis matching** (`src/matcher.py`, Table 1, match function `M`): clinically equivalent diagnoses worded differently must count as a match. Here matching is done via Russian morphological normalization (`pymorphy3`) + transliteration + a **precomputed supervised match table** (`pair-match.json`) and a normalization map (`preprocessor.json`). The table encodes the decisions of the supervised meta-model described in the paper (reported match-function quality: P = 0.91, R = 0.90, F1 = 0.91, accuracy = 0.98). - **Relative metrics** (`src/scores.py`): | code `ScoreNames` | paper metric | definition | |---|---|---| | `optimistic` | `P_opt@k`, `R_opt@k` (eqs. 3–4) | `max` over experts ÷ `min` over expert–expert pairs | | `averaged` | `P_avg@k`, `R_avg@k` (eqs. 5–6) | `mean` over experts ÷ `mean` over expert–expert pairs | | `realistic` | **RPAD@k**, **RRAD@k** (eqs. 7–8) | hardness-weighted blend of the two (`H ∈ [0,1]`; `H = 0` → optimistic, `H = 1` → averaged) | `main.py` runs with `K_MAX = 3` and `HARDNESS = 0.5`. ## Repository layout ``` service/ # evaluation harness main.py # entry point: metrics for every model and k = 1..3 src/ config.py # model / metric / score enums + data paths matcher.py # diagnosis matching (morphology + supervised pair table) pair_metrics.py # pairwise precision / recall / F1 (eqs. 1, 2) characteristic_functions.py # μ and χ (eqs. 9, 10) scores.py # optimistic / averaged / realistic (eqs. 3–8) metric.py # final metric assembly text_processor.py # top-k helper data/output/ # results (no PHI) metrics_1-360.json # final relative metrics per model failures.txt # log of unmatched diagnosis pairs (terms only) preproc_failures.txt # log of preprocessing mismatches (terms only) ``` ## Models evaluated `giga_max`, `giga_plus`, `giga_pro`, `qwen`, `deepseekr1`, `deepseekv3`, `llama_405b`, `mistral`, `gpt4o`, `deepseekr1distqwen32b`. In the paper, **DeepSeek-V3** is the most reliable candidate, with GigaChat-Max and GPT-4o not significantly different. ## Installation & running ```bash cd service uv sync # Fetch the de-identified inputs into ../data/input/ (file names must match config.py) hf download kopan/med-eval-360 --repo-type dataset --local-dir ../data/input python main.py # writes ../data/output/metrics_1-360.json ``` ## Input data format (`data/input/`, not shipped here) ```jsonc // targets_1-360.json — expert annotations (assessors "01", "02", ...) { "01": { "diag": { "": ["diagnosis", ...] }, "doc": { "": ["specialty", ...] } }, ... } // predicts_1-360_.json — model predictions { "diag": { "": ["d1", "d2", "d3"] }, "doc": { "": ["s1", "s2", "s3"] } } // pair-match.json — precomputed match decisions for diagnosis pairs { "|": [proba, is_match] } // preprocessor.json — raw → normalized diagnosis map { "": "" } // chats_1-360.json — patient↔physician dialogues (CONTAINS PHI; de-identified version in the dataset repo) ``` ## Results (`data/output/metrics_1-360.json`) For each model and `k = 1..3`: `scores` (optimistic / averaged / realistic × precision / recall / F1, separately for `diag` and `doc`) plus the per-pair `one_vs_one` values. ## Ethics & consent The study protocol was reviewed and approved by the Local Ethics Committee of the V.A. Almazov National Medical Research Centre, Ministry of Health of the Russian Federation (protocol No. 0310-24, 10 October 2024), and conducted in accordance with the Declaration of Helsinki. Written informed consent — including consent to publish de-identified study materials in an open-access publication — was obtained from all participants. ## Citation If you use this code, the metrics, or the dataset, please cite the paper: ```bibtex @article{kopanichuk2026beyond, title = {Beyond Majority Voting: A New Framework for Evaluating AI Diagnostic Systems Against Expert Consensus}, author = {Kopanichuk, Ilia and Anokhin, Petr and Shaposhnikov, Vladimir and Makharev, Vladimir and Tsapieva, Ekaterina and Bespalov, Iaroslav and Gombolevskiy, Victor and Kurapeev, Dmitry and Dylov, Dmitry V. and Oseledets, Ivan}, year = {2026}, note = {Preprint submitted to Scientific Reports}, } ``` And the de-identified dataset: ```bibtex @misc{medeval360, title = {Med-Eval-360: De-identified Russian Telemedicine Diagnostic Dialogues}, author = {Kopanichuk, Ilia and Anokhin, Petr and Shaposhnikov, Vladimir and Makharev, Vladimir and Tsapieva, Ekaterina and Bespalov, Iaroslav and Gombolevskiy, Victor and Kurapeev, Dmitry and Dylov, Dmitry V. and Oseledets, Ivan}, year = {2026}, howpublished = {\url{https://huggingface.co/datasets/kopan/med-eval-360}}, } ``` ## Contact Corresponding author: Ilia Kopanichuk — kopanichuk@airi.net (@kopanichuk)