Fetching metadata from the HF Docker repository...
Feb 17 03:34 Implement validation pipeline fixes (P1-P7) and experimental track system bshepp
Pipeline fixes:
- P1: Question type classifier (question_classifier.py) - 9 types via heuristic regex
- P3: Preserve question stem - _extract_vignette -> _split_question returning (vignette, stem)
- P4: Type-aware scoring - score_case() dispatcher with per-type scorers (diagnostic, treatment, mechanism, lab_finding, generic)
- P5: Improved fuzzy matching - recall-based with medical stopwords, 3-strategy approach
- P6: MCQ answer selection - select_mcq_answer() uses MedGemma to pick A/B/C/D given CDS report
- P7: Stratified reporting - per-type breakdown table + pipeline-appropriate subset metrics
Experimental track system:
- CLAUDE.md, TRACKS.md, EXPERIMENT_PLAN.md - project intelligence and track registry
- Track A: Baseline pipeline tagged (all app/ files)
- Track B: RAG variants (10 configs, chunking/embedding experiments)
- Track C: Iterative refinement (4 configs, serial refinement rounds)
- Track D: Arbitrated parallel (4 configs, specialist + arbiter pattern)
- Shared utilities… Feb 15 22:05 fix: resolve 13 Pylance type-checking warnings bshepp
- test_e2e.py: initialize steps/result before polling loop to avoid possibly-unbound
- test_rag_quality.py: add assert _collection is not None after initialization (3 sites)
- validation/base.py: filename param typed Optional[str] instead of str
- validation/run_validation.py: capture getenv result to avoid str|None assignment 19:13 MedGemma validation: 50-case MedQA run, TGI endpoint config, prompt improvements bshepp
- Fixed TGI token limits (MAX_INPUT_TOKENS=12288, MAX_TOTAL_TOKENS=16384)
- Reduced per-step max_tokens for faster generation
- Improved clinical reasoning prompt (disease-level dx, not symptoms)
- Fixed Unicode encoding issues for Windows console
- Fixed error masking in orchestrator (failed steps now surface errors)
- Fixed endpoint URL in .env
- Added analyze_results.py for question-type categorization
- Results: 94% pipeline success, 38% top3 accuracy, 14% dx-only accuracy
- Paused endpoint to save costs 02:25 Add incremental checkpoint saves, --resume flag, fix enum case-sensitivity, add HF_TOKEN to template bshepp
Feb 14 05:38 Add validation framework for MedQA, MTSamples, and PMC Case Reports bshepp
Three-dataset validation suite that evaluates the CDS pipeline against
external clinical benchmarks:
- MedQA: 1273 USMLE-style questions, measures diagnostic accuracy
(top-1, top-3, mentioned-anywhere)
- MTSamples: ~5000 medical transcriptions, measures parse robustness
and field extraction completeness across specialties
- PMC Case Reports: Published case reports from PubMed, measures
real-world diagnostic accuracy against gold-standard diagnoses
Architecture:
- validation/base.py: Core framework (runner, fuzzy matching, scoring)
- validation/harness_medqa.py: MedQA fetcher + harness
- validation/harness_mtsamples.py: MTSamples fetcher + harness
- validation/harness_pmc.py: PMC Case Reports fetcher + harness
- validation/run_validation.py: Unified CLI runner
Uses direct Orchestrator calls (no server needed). Tested end-to-end:
3 MedQA cases, 66.7% top-1 accuracy, 100% parse success rate." 
