AndrewRqy commited on
Commit ·
08be558
1
Parent(s): de38060
Add 3-playthrough sample trace + README (Sharing-is-Caring deliverable)
Browse files
traces/sample/README.md
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Sample LLM traces — Sharing-is-Caring badge
|
| 2 |
+
|
| 3 |
+
These JSONL files are unedited captures of every LLM call from real
|
| 4 |
+
playthroughs of *The Apprentice*. One JSON object per line, schema
|
| 5 |
+
documented in [`../README.md`](../README.md).
|
| 6 |
+
|
| 7 |
+
## Files
|
| 8 |
+
|
| 9 |
+
| File | Records | Calls | Notes |
|
| 10 |
+
|---|---|---|---|
|
| 11 |
+
| `three-playthroughs-fantasy-en.jsonl` | 117 | All `oracle-wizard-lora` (the fine-tune) | Three back-to-back playthroughs in one Python process. Default `Tobin / the Hollow` hero+village, English, fantasy theme. ~422k prompt tokens / ~43k output tokens total. 90 JSON-mode calls (resolutions + fork picks) + 27 text-mode calls (interludes + epilogue + theme expansion). |
|
| 12 |
+
|
| 13 |
+
## Quick exploration
|
| 14 |
+
|
| 15 |
+
Inspect with `jq`:
|
| 16 |
+
|
| 17 |
+
```bash
|
| 18 |
+
# Distinct models served
|
| 19 |
+
jq -r '.model_returned' three-playthroughs-fantasy-en.jsonl | sort -u
|
| 20 |
+
|
| 21 |
+
# Just the responses
|
| 22 |
+
jq -r '.response' three-playthroughs-fantasy-en.jsonl | head -3
|
| 23 |
+
|
| 24 |
+
# Calls by mode
|
| 25 |
+
jq -r '.mode' three-playthroughs-fantasy-en.jsonl | sort | uniq -c
|
| 26 |
+
|
| 27 |
+
# Latency distribution (ms)
|
| 28 |
+
jq -r '.latency_ms' three-playthroughs-fantasy-en.jsonl | sort -n | awk '
|
| 29 |
+
{a[NR]=$1; sum+=$1} END {
|
| 30 |
+
print "min:",a[1],"p50:",a[int(NR/2)],"p95:",a[int(NR*0.95)],"max:",a[NR],"mean:",sum/NR
|
| 31 |
+
}'
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
## How these were generated
|
| 35 |
+
|
| 36 |
+
LLM-call tracing is default-on in this app. Set
|
| 37 |
+
`ORACLES_TRACE_DISABLE=1` to opt out. Each session writes to
|
| 38 |
+
`traces/oracles-trace-<12-char-hex>.jsonl`. The
|
| 39 |
+
[`curate_trace`](../../scripts/curate_trace.py) script copies one of
|
| 40 |
+
those session files into this directory under a friendlier label.
|
traces/sample/three-playthroughs-fantasy-en.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|