tronrover commited on
Commit
e83980e
·
verified ·
1 Parent(s): a4e8c3b

Card: batch drift up to 0.09 as measured; legal-label note without private-set numbers

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -174,8 +174,8 @@ and false-positive rate at 0.5 are the untuned default.
174
  PolyGuardMix (which contains WildGuardMix, the source of PolyGuard's prompts) and deepset. Near-copies
175
  of training prompts were removed from the test sets, but these four sets are in-distribution for Sev. Tev1-4B and Kev-4B were trained on general decision data, not on safety or
176
  security data, and Jev's training data is not disclosed.
177
- - **Legal advice is hard to label.** The human reviewers of the legal set agree only at
178
- Krippendorff's alpha ~0.37, so small gaps there are noise.
179
 
180
  ## Choosing a threshold
181
 
@@ -276,7 +276,7 @@ the same rows.
276
  - **Thresholds.** See above: tax and legal need a higher threshold than 0.5.
277
  - **Long inputs.** The text is cut to its first 1,024 tokens. Training texts are single messages or one
278
  user message with its reply; long multi-turn transcripts are out of distribution.
279
- - **Batching.** In bf16, the same text can score up to about 0.04 differently at batch 1 and batch 16
280
  near the middle of the range, from padding numerics; scores near 0 or 1 do not move.
281
  - **Financial and medical advice** were trained on synthetic data only and have not been evaluated on
282
  human-labelled data.
 
174
  PolyGuardMix (which contains WildGuardMix, the source of PolyGuard's prompts) and deepset. Near-copies
175
  of training prompts were removed from the test sets, but these four sets are in-distribution for Sev. Tev1-4B and Kev-4B were trained on general decision data, not on safety or
176
  security data, and Jev's training data is not disclosed.
177
+ - **Legal advice is hard to label.** Whether a reply gives advice to this person or explains the law in
178
+ general is where human reviewers disagree most, so small gaps on legal are noise.
179
 
180
  ## Choosing a threshold
181
 
 
276
  - **Thresholds.** See above: tax and legal need a higher threshold than 0.5.
277
  - **Long inputs.** The text is cut to its first 1,024 tokens. Training texts are single messages or one
278
  user message with its reply; long multi-turn transcripts are out of distribution.
279
+ - **Batching.** In bf16, the same text can score up to about 0.09 differently at batch 1 and batch 16
280
  near the middle of the range, from padding numerics; scores near 0 or 1 do not move.
281
  - **Financial and medical advice** were trained on synthetic data only and have not been evaluated on
282
  human-labelled data.