precisionalgorithms commited on
Commit
3f64c14
·
verified ·
1 Parent(s): 2b94010

Raw eval output for both runs

Browse files
eval/README.md ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Eval results
2
+
3
+ Raw output from the harness so the numbers in the model card can be checked rather than
4
+ taken on trust.
5
+
6
+ - `v3_base.json` / `v3_tuned.json`: the released model against its base, 120 generated
7
+ cases, both bf16, identical serving settings. This is the pair the model card reports.
8
+ - `v4_base.json` / `v4_tuned.json`: the later run that tried to fix the refusal rate by
9
+ rebalancing the training data and made it worse. Kept because the negative result is the
10
+ useful part.
11
+
12
+ Reproduce with:
13
+
14
+ python -m pat.cli run --url <endpoint> --model <name> --generated 120 --gen-seed 1 --out out.json
15
+ python -m pat.cli compare v3_base.json v3_tuned.json
eval/v3_base.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3_tuned.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v4_base.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v4_tuned.json ADDED
The diff for this file is too large to render. See raw diff