Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -34,6 +34,25 @@ Prefix cache (8 endings):
|
|
| 34 |
5-question request: 166ms
|
| 35 |
```
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
## Doom played by the model.
|
| 38 |
|
| 39 |
<video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/typesafe-replica-doom-game-only.mp4"></video>
|
|
|
|
| 34 |
5-question request: 166ms
|
| 35 |
```
|
| 36 |
|
| 37 |
+
## Comparison: Strands Decider 2B and the hosted Jev API
|
| 38 |
+
|
| 39 |
+
Same 39 labelled cases (15 `noul`, 12 `choice`, 12 `score`) sent through
|
| 40 |
+
`/v1/systemone` schema to each system. Local models ran on an Apple
|
| 41 |
+
Silicon Mac (64 GB, MPS, bf16)
|
| 42 |
+
|
| 43 |
+
| metric | this model (Gemma 4 12B, local) | [Strands Decider 2B v19](https://strandsagents.com/blog/introducing-strands-decider/) (local) | [TypeSafe Jev API](https://docs.typesafe.ai/introduction/quickstart) (hosted) |
|
| 44 |
+
|---|---|---|---|
|
| 45 |
+
| `noul` accuracy | 15/15 | 15/15 | 15/15 |
|
| 46 |
+
| `noul` Brier (lower is better) | 0.000003 | 0.0338 | 0.00035 |
|
| 47 |
+
| `choice` accuracy | 11/12 | 12/12 | 12/12 |
|
| 48 |
+
| `score` accuracy (rounded) | 11/12 | 5/12 | 12/12 |
|
| 49 |
+
| `score` mean error (levels) | 0.09 | 0.48 | 0.004 |
|
| 50 |
+
| 1-question p50 latency | 117 ms | 96 ms | 144 ms |
|
| 51 |
+
| 3-question p50 latency | 332 ms | 121 ms | 164 ms |
|
| 52 |
+
| Size | ~24 GB | ~2B params | n/a |
|
| 53 |
+
|
| 54 |
+
Test suite: 141 passed. Small hand-written sample
|
| 55 |
+
|
| 56 |
## Doom played by the model.
|
| 57 |
|
| 58 |
<video controls width="640" src="https://huggingface.co/stephenlb/system-one-model/resolve/main/typesafe-replica-doom-game-only.mp4"></video>
|