README.md: link KnowLine-4B-Gen2, add DI 0.3 calibration
Browse files
README.md
CHANGED
|
@@ -21,6 +21,9 @@ tags:
|
|
| 21 |
|
| 22 |
[Inference guide](INFERENCE.md) · [中文说明](README.zh.md) · weights Apache-2.0
|
| 23 |
|
|
|
|
|
|
|
|
|
|
| 24 |
Highlights:
|
| 25 |
|
| 26 |
- **Decision Index 0.3, public suite: 60.47** (self-run). For comparison: Jev 1.13 scores 57.96 (board), and the best
|
|
@@ -101,6 +104,7 @@ The model is trained in a self-evolving loop, and new versions will follow.
|
|
| 101 |
|
| 102 |
| model | date | DI 0.3 public | DI 0.2.1 | golden held-out (en / zh-Hans / zh-Hant) | notes |
|
| 103 |
|---|---|---|---|---|---|
|
|
|
|
| 104 |
| KnowLine-4B-Gen1 (this model) | 2026-10-07 | 60.47 | 60.92 | 68.7 / 69.9 / 64.1 | first release (internal run "mix E", step 5,650) |
|
| 105 |
|
| 106 |
## Evaluation
|
|
@@ -159,7 +163,9 @@ All results are self-run and not verified by a third party.
|
|
| 159 |
- **Prompt injection:** an instruction planted in the state changes the answer about 9% of the time on our injection
|
| 160 |
set. Keep untrusted text clearly delimited.
|
| 161 |
- **Calibration:** ECE is about 0.05-0.06 on choice and yes/no questions and about 0.12 on score questions. For score
|
| 162 |
-
questions, fit a temperature on your own data.
|
|
|
|
|
|
|
| 163 |
- **Private tests:** part of our public-suite advantage comes from adapting to the question formats, so the 0.3 private
|
| 164 |
tests may score lower.
|
| 165 |
|
|
|
|
| 21 |
|
| 22 |
[Inference guide](INFERENCE.md) · [中文说明](README.zh.md) · weights Apache-2.0
|
| 23 |
|
| 24 |
+
> **Newer version:** [KnowLine-4B-Gen2](https://huggingface.co/PelaAI/KnowLine-4B-Gen2) is out, with a Decision
|
| 25 |
+
> Index 0.3 public score of 62.54, and at or above this model on all of our own evaluations.
|
| 26 |
+
|
| 27 |
Highlights:
|
| 28 |
|
| 29 |
- **Decision Index 0.3, public suite: 60.47** (self-run). For comparison: Jev 1.13 scores 57.96 (board), and the best
|
|
|
|
| 104 |
|
| 105 |
| model | date | DI 0.3 public | DI 0.2.1 | golden held-out (en / zh-Hans / zh-Hant) | notes |
|
| 106 |
|---|---|---|---|---|---|
|
| 107 |
+
| [KnowLine-4B-Gen2](https://huggingface.co/PelaAI/KnowLine-4B-Gen2) | 2026-10-08 | 62.54 | — | 69.7 / 71.2 / 64.8 | this model at 0.5 + mix F steps 4,500 and 5,000 at 0.25 each, weight average |
|
| 108 |
| KnowLine-4B-Gen1 (this model) | 2026-10-07 | 60.47 | 60.92 | 68.7 / 69.9 / 64.1 | first release (internal run "mix E", step 5,650) |
|
| 109 |
|
| 110 |
## Evaluation
|
|
|
|
| 163 |
- **Prompt injection:** an instruction planted in the state changes the answer about 9% of the time on our injection
|
| 164 |
set. Keep untrusted text clearly delimited.
|
| 165 |
- **Calibration:** ECE is about 0.05-0.06 on choice and yes/no questions and about 0.12 on score questions. For score
|
| 166 |
+
questions, fit a temperature on your own data. On the Decision Index 0.3 public suite, computed the board's way, ECE
|
| 167 |
+
is about 0.08-0.09 and 3.3% of answers are wrong at ≥95% confidence (board median 1.3%): the model is overconfident
|
| 168 |
+
overall.
|
| 169 |
- **Private tests:** part of our public-suite advantage comes from adapting to the question formats, so the 0.3 private
|
| 170 |
tests may score lower.
|
| 171 |
|