Upload folder using huggingface_hub
Browse files- README.md +65 -0
- nli-verifier-260623.json +190 -0
- nli-verifier-260623.summary.json +11 -0
README.md
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: custom
|
| 4 |
+
pipeline_tag: text-classification
|
| 5 |
+
tags:
|
| 6 |
+
- clinical-trials
|
| 7 |
+
- nli
|
| 8 |
+
- verifier
|
| 9 |
+
- calibration
|
| 10 |
+
- healthcare
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# CureLink NLI Verifier 260623
|
| 14 |
+
|
| 15 |
+
This repository contains the lightweight JSON verifier artifact used to calibrate
|
| 16 |
+
`CureLink/curelink-biomed-nli-v2` outputs inside the CureLink backend.
|
| 17 |
+
|
| 18 |
+
It is not a standalone Transformer checkpoint. It is a small logistic-regression-style
|
| 19 |
+
artifact consumed by the CureLink Node.js runtime in
|
| 20 |
+
`backend/nliVerifier.js`.
|
| 21 |
+
|
| 22 |
+
## Files
|
| 23 |
+
|
| 24 |
+
- `nli-verifier-260623.json`: verifier weights and feature definitions
|
| 25 |
+
- `nli-verifier-260623.summary.json`: offline training summary
|
| 26 |
+
|
| 27 |
+
## What It Does
|
| 28 |
+
|
| 29 |
+
The CureLink runtime first gets a raw NLI label from the base model, then uses this
|
| 30 |
+
verifier to adjust likely disagreements before hard guardrails run.
|
| 31 |
+
|
| 32 |
+
Typical inputs include:
|
| 33 |
+
|
| 34 |
+
- raw entailment / neutral / contradiction scores
|
| 35 |
+
- criterion section
|
| 36 |
+
- negation and admin/consent flags
|
| 37 |
+
- explicit-evidence features from patient answers
|
| 38 |
+
- criterion-family indicators
|
| 39 |
+
|
| 40 |
+
Typical outputs are:
|
| 41 |
+
|
| 42 |
+
- `MET`
|
| 43 |
+
- `UNCERTAIN`
|
| 44 |
+
- `NOT_MET`
|
| 45 |
+
|
| 46 |
+
## Training Snapshot
|
| 47 |
+
|
| 48 |
+
- training rows: `5812`
|
| 49 |
+
- eval rows: `1453`
|
| 50 |
+
- eval accuracy: `0.9780`
|
| 51 |
+
- eval macro F1: `0.9737`
|
| 52 |
+
|
| 53 |
+
## Runtime Use
|
| 54 |
+
|
| 55 |
+
The live backend points `NLI_VERIFIER_MODEL_PATH` at the JSON file and enables it with:
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
NLI_VERIFIER_ENABLED=1
|
| 59 |
+
NLI_VERIFIER_MODEL_PATH=/app/cure-link-site/backend/models/nli-verifier-260623.json
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
## Notes
|
| 63 |
+
|
| 64 |
+
- This verifier is intended to improve production trial-matching decisions.
|
| 65 |
+
- It should be evaluated together with backend guardrails, not in isolation as a general-purpose public benchmark model.
|
nli-verifier-260623.json
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"labels": [
|
| 3 |
+
"MET",
|
| 4 |
+
"UNCERTAIN",
|
| 5 |
+
"NOT_MET"
|
| 6 |
+
],
|
| 7 |
+
"feature_names": [
|
| 8 |
+
"score_entailment",
|
| 9 |
+
"score_neutral",
|
| 10 |
+
"score_contradiction",
|
| 11 |
+
"margin_top12",
|
| 12 |
+
"raw_is_met",
|
| 13 |
+
"raw_is_uncertain",
|
| 14 |
+
"raw_is_not_met",
|
| 15 |
+
"section_is_inclusion",
|
| 16 |
+
"section_is_exclusion",
|
| 17 |
+
"criterion_is_negated",
|
| 18 |
+
"criterion_is_admin",
|
| 19 |
+
"anchor_ratio",
|
| 20 |
+
"family_count",
|
| 21 |
+
"recent_biologic_yes",
|
| 22 |
+
"recent_biologic_no",
|
| 23 |
+
"recent_biologic_unknown",
|
| 24 |
+
"ivig_yes",
|
| 25 |
+
"ivig_no",
|
| 26 |
+
"ivig_unknown",
|
| 27 |
+
"stable_treatment_yes",
|
| 28 |
+
"stable_treatment_no",
|
| 29 |
+
"stable_treatment_unknown",
|
| 30 |
+
"recent_infection_yes",
|
| 31 |
+
"recent_infection_no",
|
| 32 |
+
"recent_infection_unknown",
|
| 33 |
+
"active_infection_yes",
|
| 34 |
+
"active_infection_no",
|
| 35 |
+
"active_infection_unknown",
|
| 36 |
+
"diabetes_type1",
|
| 37 |
+
"diabetes_type2_or_none",
|
| 38 |
+
"diabetes_unknown",
|
| 39 |
+
"family_diabetes",
|
| 40 |
+
"family_biologic",
|
| 41 |
+
"family_ivig",
|
| 42 |
+
"family_stable_treatment",
|
| 43 |
+
"family_infection",
|
| 44 |
+
"family_pregnancy",
|
| 45 |
+
"family_age",
|
| 46 |
+
"family_diagnosis",
|
| 47 |
+
"family_medication",
|
| 48 |
+
"family_liver",
|
| 49 |
+
"family_asthma"
|
| 50 |
+
],
|
| 51 |
+
"coefficients": [
|
| 52 |
+
[
|
| 53 |
+
-0.05870580829494506,
|
| 54 |
+
-0.2471024387213374,
|
| 55 |
+
3.354598045071172,
|
| 56 |
+
-2.1539088819291736,
|
| 57 |
+
-0.93081592505874,
|
| 58 |
+
2.6052228490319025,
|
| 59 |
+
-1.8085300251068401,
|
| 60 |
+
0.8871869264873299,
|
| 61 |
+
-1.0213100276209977,
|
| 62 |
+
-1.3542579295902986,
|
| 63 |
+
2.4242773214897957,
|
| 64 |
+
1.675454636754197,
|
| 65 |
+
0.6617755039390514,
|
| 66 |
+
0.008329025741989283,
|
| 67 |
+
-0.3578711873719977,
|
| 68 |
+
0.0,
|
| 69 |
+
-0.11829372276680071,
|
| 70 |
+
0.5273478165819441,
|
| 71 |
+
0.0,
|
| 72 |
+
0.8116729326038225,
|
| 73 |
+
-0.5047656104004493,
|
| 74 |
+
0.0,
|
| 75 |
+
0.0,
|
| 76 |
+
0.40571148553961406,
|
| 77 |
+
0.0,
|
| 78 |
+
0.0,
|
| 79 |
+
0.4090540938151436,
|
| 80 |
+
0.0,
|
| 81 |
+
-0.2575006069433746,
|
| 82 |
+
1.0745285454508833,
|
| 83 |
+
0.0,
|
| 84 |
+
1.5430043566700047,
|
| 85 |
+
-0.7585962554451525,
|
| 86 |
+
0.0,
|
| 87 |
+
-2.3779315446699267,
|
| 88 |
+
-1.4197934568335808,
|
| 89 |
+
-0.8160130007099368,
|
| 90 |
+
1.9583712507889415,
|
| 91 |
+
2.1734250085417695,
|
| 92 |
+
1.7682160007720742,
|
| 93 |
+
-3.2224850856968237,
|
| 94 |
+
1.8135782305216843
|
| 95 |
+
],
|
| 96 |
+
[
|
| 97 |
+
1.1805404296777204,
|
| 98 |
+
-2.5126247600546767,
|
| 99 |
+
1.7010380543373635,
|
| 100 |
+
1.2815491098763698,
|
| 101 |
+
2.323826030087859,
|
| 102 |
+
-3.0015479842331727,
|
| 103 |
+
0.7165148181394332,
|
| 104 |
+
0.14774910184545395,
|
| 105 |
+
-0.1089562378513296,
|
| 106 |
+
0.8966410347980129,
|
| 107 |
+
1.5224762713639035,
|
| 108 |
+
-0.826688756463995,
|
| 109 |
+
-0.5294775391262355,
|
| 110 |
+
0.1137884915814507,
|
| 111 |
+
0.24543839113929267,
|
| 112 |
+
0.0,
|
| 113 |
+
0.1265377061728757,
|
| 114 |
+
0.2885738247965893,
|
| 115 |
+
0.0,
|
| 116 |
+
0.8424064880507692,
|
| 117 |
+
-1.3999052423778533,
|
| 118 |
+
0.0,
|
| 119 |
+
0.0,
|
| 120 |
+
-0.9239593085737039,
|
| 121 |
+
0.0,
|
| 122 |
+
0.0,
|
| 123 |
+
0.41511153096946485,
|
| 124 |
+
0.0,
|
| 125 |
+
0.38896493103513263,
|
| 126 |
+
0.616371866718546,
|
| 127 |
+
0.0,
|
| 128 |
+
-0.31497829363028673,
|
| 129 |
+
-0.05588464824872116,
|
| 130 |
+
0.0,
|
| 131 |
+
-2.8579638055130534,
|
| 132 |
+
-0.2759640988785288,
|
| 133 |
+
0.16752605907705148,
|
| 134 |
+
0.4942888712682904,
|
| 135 |
+
-1.5269538191680725,
|
| 136 |
+
1.5552778253063868,
|
| 137 |
+
0.21207628038307777,
|
| 138 |
+
2.07309809027763
|
| 139 |
+
],
|
| 140 |
+
[
|
| 141 |
+
-1.121834621382774,
|
| 142 |
+
2.759727198776019,
|
| 143 |
+
-5.0556360994085265,
|
| 144 |
+
0.8723597720528167,
|
| 145 |
+
-1.393010105029122,
|
| 146 |
+
0.39632513520127194,
|
| 147 |
+
1.092015206967405,
|
| 148 |
+
-1.0349360283327804,
|
| 149 |
+
1.1302662654723201,
|
| 150 |
+
0.4576168947922831,
|
| 151 |
+
-3.946753592853701,
|
| 152 |
+
-0.8487658802902067,
|
| 153 |
+
-0.13229796481280715,
|
| 154 |
+
-0.12211751732344012,
|
| 155 |
+
0.11243279623270414,
|
| 156 |
+
0.0,
|
| 157 |
+
-0.008243983406074946,
|
| 158 |
+
-0.8159216413785344,
|
| 159 |
+
0.0,
|
| 160 |
+
-1.6540794206545926,
|
| 161 |
+
1.9046708527783047,
|
| 162 |
+
0.0,
|
| 163 |
+
0.0,
|
| 164 |
+
0.5182478230340882,
|
| 165 |
+
0.0,
|
| 166 |
+
0.0,
|
| 167 |
+
-0.8241656247846093,
|
| 168 |
+
0.0,
|
| 169 |
+
-0.13146432409175796,
|
| 170 |
+
-1.6909004121694249,
|
| 171 |
+
0.0,
|
| 172 |
+
-1.2280260630397184,
|
| 173 |
+
0.8144809036938713,
|
| 174 |
+
0.0,
|
| 175 |
+
5.235895350182977,
|
| 176 |
+
1.6957575557121123,
|
| 177 |
+
0.6484869416328859,
|
| 178 |
+
-2.4526601220572286,
|
| 179 |
+
-0.6464711893736893,
|
| 180 |
+
-3.3234938260784586,
|
| 181 |
+
3.0104088053137383,
|
| 182 |
+
-3.8866763207993107
|
| 183 |
+
]
|
| 184 |
+
],
|
| 185 |
+
"intercepts": [
|
| 186 |
+
-0.2342463314822243,
|
| 187 |
+
-0.00033122549989492763,
|
| 188 |
+
0.23457755698217014
|
| 189 |
+
]
|
| 190 |
+
}
|
nli-verifier-260623.summary.json
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"train_rows": 5812,
|
| 3 |
+
"eval_rows": 1453,
|
| 4 |
+
"eval_accuracy": 0.978,
|
| 5 |
+
"eval_f1_macro": 0.9737,
|
| 6 |
+
"label_order": [
|
| 7 |
+
"MET",
|
| 8 |
+
"UNCERTAIN",
|
| 9 |
+
"NOT_MET"
|
| 10 |
+
]
|
| 11 |
+
}
|