Buckets:
| # Lab notebook: reasoning decision model (Qwen3.5-4B) | |
| Times are UTC (IST = UTC + 5:30). Observations as they happened; numbers are from the files named. | |
| ## 2026-10-31 | |
| **00:49** Run started (`run_all.sh`, tmux `run`). Prep reused (identical sets under the Qwen3.5 tokenizer). | |
| **00:49-01:03 Probe** (`results/probe.json`): 128 pool items x 4 chains. Procedural: some chain correct on 100%, majority | |
| 98.4%; tasksource: 75% / 65.6%. Chains median 1,878 tokens, p90 6,414; 4 of 512 looped to 32k. Throughput is tail-bound | |
| (1,957 tok/s overall; 4.5-5.5k tok/s while the queue was full). | |
| - Tasksource "errors" read by hand are mostly labelling conventions, not bad reasoning: SemEval aspect sentiment | |
| ("an absolute must" = neutral), AXG pronoun entailment ("said only he could" -> entailment), a packed ToS clause whose gold | |
| looks wrong. The alpha = 0.5 target is meant to make the readout unsure on exactly these. | |
| - Qwen3.5-4B writes `.c` + `w` ("A.cw") just before `</think>` in about a third of chains: a model quirk, harmless for parsing. | |
| **01:03-02:53 Baseline** (untrained model, `results/predictions/*_base.jsonl`). | |
| - Readout, temperatures fitted on calibration (choice 1.71, yes/no 2.41, score 1.28): procedural 46.8%, tasksource 60.4%, | |
| OpenDecisions 66.5%, LocalLLaMA 63.0%. OpenDecisions Capability 61.9 (paper: 63.6 for zero-shot Qwen3.5-4B). | |
| - Reasoning, one sample, 5,957 items, 16.9M tokens in 85 min (3.3k tok/s, 256 running sequences): | |
| | Set | Readout | Reasoning | Post-chain readout | Readout = reasoning | Letter = post-chain | Median tokens | Loops | Disagreement AUROC | | |
| |---|---|---|---|---|---|---|---|---| | |
| | procedural | 46.8 | 95.1 | 96.0 | 46.4 | 98.1 | 2,078 | 5 | 0.739 | | |
| | tasksource | 60.4 | 73.3 | 73.6 | 67.1 | 98.5 | 1,637 | 7 | 0.769 | | |
| | OpenDecisions | 66.5 | 94.9 | 95.0 | 67.3 | 99.2 | 1,043 | 16 | 0.805 | | |
| | LocalLLaMA | 63.0 | 71.0 | 71.0 | 69.8 | 98.8 | 2,660 | 24 | 0.746 | | |
| | all | 62.7 | 83.9 | 84.1 | 66.2 | 98.9 | 1,626 | 52 | 0.777 | | |
| - Chains lengthen with difficulty: procedural level 0 to 4 median 1,283 to 4,446 tokens (accuracy 100% to 89.8%); | |
| OpenDecisions easy 635 vs hard 2,117 tokens, gap readout vs reasoning 3 points (easy) vs 39 points (hard). | |
| - Every loop is a degenerate repeat on a hard procedural item (e.g. "**Wait, check R7 at ts: 1.** - R7 enabled." x370). | |
| - Where reasoning loses to the readout: taste and convention tasks (humour, sentiment of neutral facts, Italian word sense). | |
| - LocalLLaMA "misses" read by hand sit on split soft gold (0.38 vs 0.33; 0.50 vs 0.45); the reasoner's picks are defensible. | |
| - System view (metrics on base only, `results/metrics_baseline_only.json`): OpenDecisions Capability 61.8 readout only, | |
| 67.2 / 72.3 / 76.5 / 82.8 with 10 / 20 / 30 / 50% routed by margin, 84.4 thinking always. Post-chain answers are | |
| overconfident (temperatures 4.77 / 3.49 / 4.02). Gap recovered at 30% routed: 54% (margin), 55% (learned router); | |
| fixable AUROC 0.58. | |
| - The post-chain pass re-prefilled ~15M tokens (13 min) because the prefix cache had evicted the chains. From the joint | |
| eval on, the readout after the chain is taken right after each chain from its exact token ids. | |
| **02:51-03:08 Generation, first segment** (512 running sequences): about 19 items/min, 3.0-3.3k finished tok/s against | |
| 3.4k at 256 sequences in the baseline: the GPU is compute-bound (100%, 382 W), so more concurrency does not help. At that | |
| pace the original deadline (05:01) yields about 2,500 items, and joint training on them takes about 40 min of its 75. | |
| Stopped at 03:08 (380 items kept, no partial lines), reverted to vLLM's A100 batching defaults and restarted with the | |
| deadline at 05:41 (window 8.67 h) so the freed training time goes to data. A first relaunch failed in 1 s: bash integer | |
| arithmetic cannot take `TOTAL_H=8.67`; the end time is now computed in Python. | |
| **03:09-03:16** Engine stats (now on) showed `Running: 128 reqs, Waiting: ~850, KV cache 40%`: the in-process async engine | |
| falls back to vLLM's global 128-sequence default. Restarted at 03:16 with `max_num_seqs=256` (412 items kept): | |
| 4.7k tok/s steady at 70% KV, against 4.1-4.6k at 128 (+12%). The GPU is close to compute-bound from about 128 sequences. | |
| Correction: the earlier "512 is no faster" comparison set a ramp-up window and a lagging completion counter against a | |
| steady-state window, and is withdrawn. Expected generation by 05:41: about 3,500 items. | |
| **05:41-06:03 Generation end** (`results/generation.json`): 3,169 items, 12,676 chains, 41.0M tokens; chains median 1,945, | |
| p90 6,661; parse 98.8%; 154 chains (1.2%) looped to 32k, mostly on natural-language judgement items (safety and | |
| helpfulness ratings, AI-text detection, fact checks) rather than procedural ones as in eval. 2,818 items (89%) have a correct | |
| chain; majority vote correct on 85.2%. Votes unanimous on 96% of procedural and 86% of tasksource items; on 15.5% of | |
| tasksource items the reasoner is unanimous against gold (labelling conventions). Shortest correct chain median 920 | |
| tokens vs 1,502 for an average sample. | |
| **06:03-06:54 Joint training** (`results/adapters/joint`): 5,669 examples (2,673 reasoner, 2,996 readout; 318 dropped by the | |
| timing probe, which measured during kernel compilation), 207/207 steps, 8.74M tokens, 46 min at 3.14k tok/s, 19.8 GB peak. | |
| Chain loss flat at about 0.19 throughout (the model's own chains); readout loss about 1.0 at the start and 0.6-0.9 later, but | |
| per-window values are confounded by length-grouped batching (windows hold 108 to 700 readout examples of different kinds). | |
| **06:56** Gold-only ablation: 3,169 readout examples, 1.71M tokens, 37 steps, no drop. | |
| **07:07** Token-id post-chain readout checked on Qwen3.5-0.8B: on 9 closed chains it equals the text-based readout | |
| (max difference in probability 0.0001, argmax 9/9) and the generated letter (9/9). The joint eval uses it. | |
| **07:23-08:09 Joint eval, first segment** (256 sequences). Readout (calibration temperatures 1.28 / 1.18 / 1.05, against | |
| 1.71 / 2.41 / 1.28 for base): procedural 46.8 -> 71.9, tasksource 60.4 -> 66.4, OpenDecisions 66.5 -> 72.2, | |
| LocalLLaMA 63.0 -> 66.5 (all 62.7 -> 69.5); OpenDecisions Capability 61.8 -> 66.7 (Intelligence 41.2 -> 52.8, | |
| Calibration 82.5 -> 80.7). Reasoning on the first 2,564 items, same items as base: accuracy equal or higher on every set, | |
| chains 20-30% shorter, readout = own reasoning 72.9 vs 47.2 (procedural), 76.2 vs 67.7 (tasksource), 83.7 vs 78.8 | |
| (OpenDecisions). Zero loops on items where base looped 13 times (e.g. a level-4 needle lookup: base ran to 32k tokens, | |
| joint answered correctly in 1,139). From about 2,400 items on the KV cache reached 99.5% with preemption (140 running, | |
| 116 waiting, 1.9k tok/s): stopped at 08:09 and resumed at 128 sequences (`MAX_NUM_SEQS=128`), all 2,564 records kept. | |
| **08:14** At 128 sequences the eval decodes at 4.2-4.6k tok/s (KV 30-40%). The token-id post-chain readouts do not hit the | |
| prefix cache (hit rate 0-4%; prompt throughput 2-5k tok/s): for this hybrid model vLLM 0.32 does not keep a finished | |
| request's generated blocks reusable. The readouts stay exact (checked at 07:07) but cost a full prefill, now interleaved with | |
| decoding (decode drops to ~2.7k tok/s while they run), so the saving expected at 02:53 did not materialise. | |
| **08:48 Correction.** "Zero loops" for the joint reasoner held only for the first ~2,600 items. On 4,237 items both | |
| models have finished, base looped 37 times and joint 25 (OpenDecisions 16 vs 17, LocalLLaMA 9 vs 0 so far). Same items: | |
| reasoning 86.9 -> 87.4%, readout = own reasoning 65.3 -> 74.7%, median chain 1,402 -> 1,184 tokens. | |
| **08:40** Dashboard published (Mercity design language), rebuilt from the run files and republished each update cycle. | |
| ## V2 | |
| **10:09** V2 generation started in `runs/v2` (256 sequences, 0.95 GPU memory, no speculative decoding by decision); | |
| KV cache 1.89M tokens; ~4.8-5.5k tok/s early. V2 pool: 41,994 readout questions (18,000 procedural, 23,994 tasksource | |
| from 717 sources, 3,292 of them from 25 soft-label sources capped at 150), dev slice 999. | |
| **10:20** Repetition filter retuned: a "line repeated 3+ times" rule would drop 2,388 healthy chains (rule quotes, | |
| per-record enumeration). No closed chain contains a true loop; about a dozen near-loops repeat a line 20+ times | |
| ("If A is self-contradictory, B follows." x240). The joint model's extra eval loops therefore come from training, not | |
| from the data; RL's loop penalty is the fix. | |
| **10:45** Chain audit (50 shortest-correct training chains, 25 procedural + 25 tasksource, checked against the evidence; | |
| `runs/v2/data/chain_audit_50.jsonl`): 39 sound, 9 minor slips off the decisive step, 2 flawed (n1 misreads the prior rule | |
| of policy_under_uncertainty; n28 guesses a MuTual pronoun and claims "I found a similar example online"), 0 uncheckable. | |
| Both flawed chains had 4/4 correct samples. Recurring habits: invented look-ups ("Wait, I found similar examples | |
| online"), dataset-convention guessing, repeated "final check" passes, '.cw' before `</think>` (9/50). | |
| Decisions: no judge model (4% flawed); drop chains with invented look-ups (precision checked on 10 random matches, all | |
| genuine; ~6% of candidate chains, almost all tasksource); strip '.cw'. Chains that merely name a dataset are kept. | |
| **11:20** OpenRouter teacher (user's key, $7): Qwen3.5-35B-A3B (`:nitro`; fp4 providers excluded, `require_parameters`, | |
| price cap $2/M out). Pilot 6 x 2: 12/12 closed, 9/12 correct, chains ~same length as the 4B's, raw reasoning returned in | |
| `message.reasoning`. Full run takes tasksource items from the back of the pool (local generation works from the front). | |
| **11:33** Teacher done: 662 items x 4 chains, $6.60, 11.5 min (Alibaba 1,981 / Parasail 667 requests), 2,645/2,648 | |
| closed, 72% correct. Per item correct-of-4: 0:21% 1:4% 2:4% 3:6% 4:64%; the 4B on its own tasksource items: 0:21% 1:5% | |
| 2:5% 3:8% 4:61%. A 9x larger sibling misses the same share, so 0/4 items are mostly label noise or ill-posed: of 5 read, | |
| 3 ambiguous gold (xglue relevance Bad vs Good, dynasent positive vs mixed, pragmeval premise type), 1 near impossible | |
| (TuringBench: which of 20 generators wrote this), 1 ordinal scale shown shuffled (relevance levels are not marked | |
| `score`, a prep issue to fix later). Teacher chains are shorter (median 1,083 vs 1,567 tokens) and never repetitive | |
| (0.0% vs 1.4%), but invent look-ups more (17% of samples). 835 training chains from 454 items survive the filters. | |
| **11:40** Speed benchmark written (`bench_speed.py`, runs first in run_v2.sh). Smoke on 0.8B in the GPU's 4.5 GB slack: | |
| no-checkpoint losses bit-identical to checkpointed; Liger off by 0.6% loss / 6.5% gradient vs HF bf16 -> equivalence is | |
| now judged against an fp32 reference (a setup may not be more than 1.5x further from fp32 than the current one). | |
| Sweep shortened to 1,500 questions / 500 dev, ties to the lower LR. Invented-recall filter widened (+1.8% of chains). | |
| **11:48** Pipeline re-read: the speed benchmark is now optional (a crash leaves the current setup and the pipeline | |
| goes on; tested in isolation). Disk: /workspace had 29 GB free and V2 writes ~27 GB (three merged 4B checkpoints plus | |
| adapters and predictions). Moved the three v1 merged checkpoints (24 GB, rebuildable from their adapters in ~2 min each) | |
| to the container disk under /root/offload_checkpoints, symlinked in place: 52 GB free. Benchmark scratch (~1.5 GB of | |
| gradients) goes to /root/bench_tmp, not results/ (which syncs to the public bucket). | |
| **11:52** RL plan checked against data and TRL 1.17.1 (every config field exists). The 658 RL-eligible items (1-3 of | |
| the 4B's 4 chains correct) have long chains: median 5,280 tokens, mean 6,985, 21% past 8,192, p95 30K (loops). At 32 | |
| prompts x 8 per step that is ~1.3M generated tokens a step, 100 steps ~15 h, not 4. Now: rounds of 128 chains (16 x 8) | |
| feeding 4 optimizer steps of 32 (TRL steps_per_generation; its importance-sampling correction covers the lag), one | |
| chain per forward (an 8K chain's full-vocabulary logits are ~4 GB in bf16), vLLM at 0.5 of the GPU: ~1.6 min/step, | |
| 120 steps ~3.2 h, each eligible prompt used at most once. --model must be a text-only checkpoint (TRL pushes weights | |
| into vLLM by name; the hub repo is the multimodal layout) and rl_grpo.py now refuses anything else; a text-only | |
| Qwen3.5-0.8B copy for the feasibility test is in checkpoints/qwen3.5-0.8b-text. Teacher records are left out of RL | |
| prompt selection (their 1-3 of 4 says nothing about the 4B's uncertainty). | |
| **12:00** Dashboard: V2 section (stages, chain sources local vs teacher, benchmark, sweep, V2 joint curves, results). | |
| Plan doc: Phase A/B steps updated with the teacher chains, benchmark, short sweep and RL round structure. | |
| **12:25** Chains with answers, all sources (no outside reasoning datasets used): 6,377 questions, 25,508 chains, 25,232 | |
| with an answer, 20,660 correct; 9,617 chains on 5,064 questions survive the training filters (4B v1: 4,973 / 2,611; | |
| 4B V2 so far: 3,809 / 1,999; teacher: 835 / 454). RL selection (`rl_select.py`, rules only): 771 mixed, 645 kept | |
| (233 exact, 359 hard-label with >= 2 of 4, 53 soft-label). Reward unit check: +1, -1, +0.2 (60/40 split), -1.5 (open). | |
| **13:31** V2 generation ended (deadline 13:09 + drain); gen.jsonl 7,333 records after the teacher merge. | |
| **13:45** Speed benchmark (results/speed_bench.json): A (current) 3,023 tok/s; B Liger 3,368 (+11%) but loss error vs | |
| fp32 1.49% against A's 0.64% (limit 0.97%) -> not equivalent (its gradient error is lower than A's, 0.287 vs 0.331; | |
| the rule is the rule); C no-checkpointing <= 8K 4,474 (+48%), equivalent, but 74.9 GB peak (limit 72 GB) -> out; | |
| D 5,205 (+72%) fails on B's loss error. Chosen: A, no flags. V2 joint estimate ~51.7M tokens vs ~53M that fit in | |
| 300 min at 3,000 tok/s: fits, just. Sweep 1e-4 (1,500 questions, 17 steps, 4.4 min): dev acc 0.614, NLL 0.880, ECE 0.091. | |
| **14:00** Sweep: 1e-4 acc 0.614 / NLL 0.880 / ECE 0.091; 2e-4 0.650 / 0.828 / 0.055; 3e-4 0.638 / 0.870 / 0.056 -> | |
| lr 2e-4. User asked for config C (+48%) despite the 72 GB line (deviation #26): pipeline stopped between stages, | |
| train.py gained an out-of-memory redo (checkpointed, gradients restored; forced-OOM test on the 0.8B passed), | |
| restarted 14:02:54 with `--micro-tokens 8192 --ckpt-above 8192` and expandable segments. (The stop helper's | |
| `pkill -f` matched its own command line and exited 144 after the stop had already worked; nothing else was hit.) | |
| **16:49** V2 joint done: 1,309/1,309 steps, 47.8M tokens, 2 h 46 min at ~4,950 tok/s, peak 71.5 GiB, 0 out-of-memory | |
| redos. Held-out dev (999 questions: 504 procedural, 495 tasksource): acc 0.813, NLL 0.504, ECE 0.023 (sweep 2e-4 on | |
| 500 of them: 0.650 / 0.828 / 0.055). Training readout loss settled ~0.54 (v1 final 0.87); chain loss ended 0.22 (start | |
| 0.235). Overlap check: dev ids are disjoint from the pool, but 61/999 dev items (56 procedural) are exact duplicates | |
| of a pool item (same evidence, criterion, options; procedural generators repeat small instances), so dev acc without | |
| them is >= 0.801 even if all 61 were right. Eval: eval_proc has 50/526 (9.5%) exact duplicates of a V2 pool item (v1's | |
| trained items: 36, 6.8%); eval_ts, OpenDecisions and LocalLLaMA have none. Procedural eval will be reported with and | |
| without those items for every model. | |
| **18:30** Thinking-mode eval (V2 joint) at 1,901 / 2,013 (1,498 eval subset + 515 calibration), parse 97.3%, 52 looped so | |
| far (v1's eval run ended at 96.9% / 104 of 3,393). On the first 1,058 finished (short-biased): base 89.3, v1 89.9, | |
| V2 90.3. Chain openings ("Thinking Process:" vs "The user ...") split ~50/50 in V2, v1 and the 4B's own generation | |
| alike, so no teacher style shift. RL data (deviation #27): `rl_fresh.py` built 6,000 never-trained tasksource | |
| candidates in 21 s (3,000 from 53 reasoning sources, 3,000 from 651 others; 389 skipped as content seen in pool/held); | |
| rl_select.py now screens them with the V2 readout (gold 0.15-0.85) and mixes 288 fresh (192 reasoning) + 192 | |
| in-domain; rl_grpo.py takes them via --fresh-pool. Offline check: 480 prompts built, 288 fresh. | |
| **18:43** V2 thinking mode, full eval subset (1,498, same items as v1): **81.2% vs v1 84.4%, base 84.0** (proc 94.7 / 97.3, | |
| ts 70.3 / 73.7, LocalLLaMA 66.4 / 70.9, OD 94.2 / 96.4). The regression is the loops: V2 looped (hit the 32K cap, no | |
| answer) on 5.1% vs v1 1.4%, base 0.9%; v1 answered 49 of V2's 76 looped items correctly (3.3 points, the whole gap); | |
| on items neither model looped, v1 86.5 vs V2 85.7 (within noise). Loops are re-verification cycles: the chain states | |
| its answer ("The answer is B. Okay, I'm done.") then "Wait, I should check ..." repeats; 67 of 76 end in an exact | |
| repeat. Median chain tokens 1,340 vs 1,323. Correction: the 18:03 read ("thinking mode held up", 90.3 vs 89.9) used | |
| the first-finished items, which excludes loops by construction; and flat training chain loss could not show this | |
| (it scores imitation of closed chains, not whether sampling stops). RL's -1.5 for an open chain at 8,192 tokens and the | |
| length penalty above 4,096 target exactly this; post-RL loop rate is now the first gate. | |
| **19:00** Phase A finished (pipeline exit 0). metrics_v2: v2_joint readout all 0.727 (v1 0.695), OD Capability 67.3 | |
| (66.7), ECE 0.033 (v1_gold 0.019 is still the best calibrated); agreement 78.3% (75.7), disagreement AUROC 0.839 | |
| (0.780); routing on the 1,498 reasoning items: readout 76.0 -> 81.8 at f = 0.3 (v1 70.3 -> 78.0), 1,749 mean | |
| tokens (v1 1,166; loops cost 32K each), gap recovered 0.70 (v1 0.52). Ablations on eval: abl_gold 71.7 / ECE 0.031, | |
| abl_mix 70.5 / ECE 0.047 (mix +3.1 procedural, -1.6 OD, -2.3 LocalLLaMA): votes hurt out of distribution, the | |
| opposite of the in-distribution dev reading. Ablation checkpoints uploaded to run2_v2/checkpoints/ (sizes verified) | |
| and deleted locally. | |
| **19:05** RL smoke test on Qwen3.5-0.8B text-only (4 steps, 16 chains/step, 1,024-token cap): vLLM colocate, | |
| reward, backward, LoRA save all fine; 30 s/step; exit 0. | |
| **19:10** rl_select.py with V2 readout: in-domain 592 kept (167 already sure, 5 sure-wrong labels dropped); fresh | |
| 3,129 of 6,000 in the 0.15-0.85 band (2,322 sure right, 549 sure wrong or noisy). Final 480 = 192 fresh reasoning + | |
| 96 fresh general + 192 in-domain. | |
| **19:12** RL on the 4B started: rl_grpo.py --model checkpoints/v2_joint_merged, 120 steps, 128 chains per | |
| generation round (16 prompts x 8) feeding 4 optimizer steps of 32, 8,192-token cap, lr 1e-5, vLLM at 0.5. | |
| **19:26** RL round 1 (sequence_mask IS): reward -0.42, 26.6% of chains unfinished at 8,192, mean 4,458 tokens, IS | |
| ratio mean 0.55 / min 0, round time 537 s. Stopped and restarted with token_truncate IS (deviation #28). Round time | |
| means 120 steps would take ~5-6 h rather than 3.2 h; the LR is constant after 5 warm-up steps and checkpoints land | |
| every 20 steps, so the run can stop at any checkpoint without a schedule mismatch. | |
| **20:43** RL step 22 (6 rounds, 96 questions; checkpoint-20 saved). Per round (reward / unfinished at 8,192 / mean | |
| tokens): -0.42/27%/4,458, -0.11/23%/4,188, +0.64/5%/2,702, +0.31/11%/2,869, +0.31/12%/3,745, +0.11/20%/4,538. | |
| Rounds draw different questions, so per question: share right ~ readout p_gold + round gives +0.044 per round | |
| (se 0.022), unfinished -0.016 per round (se 0.015), reward +0.096 per round (se 0.051). Questions in the mix as | |
| planned (40 fresh / 24 in-domain in rounds 1-4). Loops in RL chains are indecision cycles between two options | |
| ("Wait, I'll consider ... B. Okay, I'll choose C."); hardest families: StepGame/SpaRTQA spatial puzzles and | |
| procedural event_triggered_execution (the family V2 looped on in the eval). Entropy 0.14-0.17, no collapse. | |
| **21:03** RL step 28 (7 rounds, 111 questions). Difficulty-adjusted per round: share right +0.028 (se 0.018), | |
| unfinished -0.011 (se 0.012). Finished chains: length flat (log chars +0.009 per round, se 0.015, adjusted for | |
| readout p_gold and prompt length), but "Wait" per finished chain falls 10.2 -> 6.1 (-0.41 per round, se 0.16). | |
| Within a question, right finished chains are shorter than wrong ones (median 12.5K vs 14.9K chars). Signal: 40-50% | |
| of questions split right/wrong (in-domain 60%, fresh reasoning 31%). By family (first vs second half of rounds): | |
| logic/rules 46% -> 77% right, spatial 67% -> 61%, multilingual 50% -> 44% (small n). Dashboard: RL section gains | |
| "Wait" per finished chain and share right among finished chains. | |
| **21:23** RL step 35 (9 rounds, 142 questions). Difficulty-adjusted per round: share right +0.014 (se 0.012; was | |
| +0.044 at 6 rounds), unfinished -0.012 (se 0.008), finished length +1.5% (se 1.0%), "Wait" per finished chain -0.23 | |
| (se 0.11). The early estimates shrank as rounds accumulated; the decisiveness effect is the only one near 2 se. | |
| **21:29** Post-RL eval armed: tmux `stop60` waits for checkpoint-60 (trainer_state + adapter), stops the RL session | |
| by name, then runs runs/v2/run_rl_eval.sh (merge checkpoint-60 into V2 joint -> checkpoints/rl60_merged; evaluate.py | |
| predict exactly as V2: readout on all eval items, thinking mode on the 1,498 subset + calibration, latency; | |
| metrics_v2.py with rl60 beside base / v1 / V2 -> results/metrics_rl.json). Expected: stop ~22:40, results ~23:50 UTC. | |
| **21:35** RL rate change (deviation #29, user's call): tmux `switch40` waits for checkpoint-40, stops the 1e-5 run by | |
| session name, continues 20 steps at 2.5e-5 from the step-40 adapter (fresh AdamW, the 320 questions not yet used, | |
| --exclude-seen tested offline: 480 -> 320), then runs run_rl_eval.sh on checkpoints/rl_grpo_b/final as rl60. | |
| Round 10 (steps 37-40): reward -0.35, 23% unfinished, 4,950 tokens (a hard batch). | |
| **21:44** Phase b first launch (21:40) used --seed 19, which reshuffles option orders, so --exclude-seen matched only 54 | |
| of the 160 used prompts by text; stopped after 3.5 min (still loading; log kept as rl_grpo_b_seed19_aborted.log) and | |
| relaunched with --seed 17: 160 excluded, 320 left. tmux `phaseb` runs it and then the post-RL eval. | |
| **22:20** RL step 50 (13 rounds, 205 questions; phase-b checkpoint-10 = step 50 saved). Difficulty-adjusted per round: | |
| share right +0.001 (se 0.007), unfinished -0.004 (se 0.004), finished length flat, "Wait" per finished chain -0.22 | |
| (se 0.06; 10.2 -> 5.2). Unfinished by family, first vs second half of rounds: procedural 46% -> 35%, logic 23% -> 13%, | |
| spatial 17% -> 9%, other tasksource 8% -> 12%. Correction: the 21:00 "logic 46% -> 77%" did not hold (68% -> 64% with | |
| 13 rounds and babi/lsat/entailment counted as logic). Late third of rounds ~4 points below average on share right after | |
| adjustment (~1 se). Entropy 0.13 at 2.5e-5 (0.15-0.17 at 1e-5); no answer collapse: distinct answers per question | |
| 1.31 (phase a) vs 1.50 (phase b), letter shares A/B/C 35/40/9% vs 23/40/21%. Entropy is not used as a stop rule. | |
| **22:28** Checkpoint-40 eval armed (tmux `evalrl40`: waits for the step-60 eval driver, then run_rl_eval.sh with | |
| CKPT=checkpoint-40 NAME=rl40); metrics_rl.json now takes every evaluated RL model. Dashboard pipeline section: view | |
| switch (V2 + RL / v1 / both) for the Gantt and stage table, and an "Up next" table from runs/v2/results/upcoming.json. | |
| **22:40** User going to sleep (back ~05:30-06:30 UTC); asked to start the 2B regardless of RL (deviation #30). | |
| runs/v2_2b/run_2b.sh: probe Qwen3.5-2B + MiniCPM5-2B -> pick (rule in the script) -> 2-step smoke (falls back to Qwen) | |
| -> sweep -> V2 joint on V2 data -> merge -> base + trained eval -> metrics_2b.json; tmux `night` chains it after the | |
| rl60 eval, then the rl40 eval. MiniCPM5-2B checked on CPU: LlamaForCausalLM, 42 layers, 131K context, Qwen-style | |
| <think> template, A-Z single tokens; its template writes "<s>", so vllm_utils now strips a leading BOS before vLLM | |
| adds its own (no change for Qwen, whose tokenizer has no BOS). stage() in run_2b.sh returns the command's status so a | |
| failure is never marked done. | |
| **23:03** RL step 60 done 22:47 (RL_EXIT 0), merged into checkpoints/rl60_merged (Qwen3_5ForCausalLM). rl60 fast readout on | |
| eval: all 72.5% (V2 72.7), proc 89.5 (89.4), ts 72.3 (71.8), OD 73.9 (74.2), LocalLLaMA 66.3 (66.7); ECE 0.030 (0.033), NLL | |
| 0.672 (0.669): RL left fast mode where it was, so no readout re-alignment is needed. Thinking eval running: 934 of 2,013 | |
| finished at 8 min (V2 took ~20 min for 1,055), median chain 800 tokens so far; early finishers exclude loops by | |
| construction, so no accuracy or loop read until it completes. | |
| **23:20** rl60 thinking mode, the same 1,498 items: **83.1%** (V2 81.2, v1 84.4, base 84.0); looped **0.2%** (3 items; V2 5.1%, | |
| v1 1.4%, base 0.9%); median chain 1,096 tokens (V2 1,340), mean 1,802 (V2 3,748: loops gone, half the compute); eval | |
| finished in 23 min (V2 57). By set: proc 97.0 (V2 94.7), ts 72.0 (70.3), LocalLLaMA 68.7 (66.4), OD 95.8 (94.2). Paired: | |
| RL right / V2 wrong 91, the reverse 62. On V2's 76 looped items RL answers 47 right and loops on 1. On items neither | |
| looped, V2 85.5 vs RL 84.4: RL fixed the loops and gave up ~1 point where V2 already finished. RL fast readout 72.5 (V2 | |
| 72.7): no re-alignment needed. Decision on more RL: not tonight (the loop fix is in; in-training accuracy was flat; the | |
| small loss on finished items argues for a better setup rather than more steps of this one); the GPU goes to Jevals | |
| and the 2B, then the rl40 eval separates the two learning rates (after_2b.sh). | |
| Jevals (jevals.com, suite 0.1.0): 3 boards x 300 items rebuilt from the source datasets and matched to every published | |
| state hash (banking77 state key "message", pubmedqa "context"); jevals_eval.py scores with their rules (Brier / RPS | |
| Decision Score vs label prior, accuracy, 10-bin ECE, 2,000-sample item bootstrap). Our pool holds 40 Banking77 and 40 | |
| HelpSteer2-helpfulness train-split items (exact text overlap with the Jevals items: 0 and 2 of 300); PubMedQA is unseen. | |
| **23:29** Jevals thinking mode added (`jevals_eval.py think`): one chain per item, then the readout after the chain, | |
| post-chain temperatures from the model's own calibration chains; Banking77 reasons over the fast readout's 24-option | |
| shortlist (gold outside it = wrong); an unfinished chain falls back to the fast readout. Overnight driver reordered | |
| (tmux `night3`): Jevals fast for the 4B models -> Jevals thinking for rl60 -> 2B pipeline -> after_2b.sh (Jevals | |
| thinking for V2 / base / v1, Jevals for the 2B, rl40 eval). Base 4B fast, raw: Banking77 DS 56.2, PubMedQA 48.1, | |
| HelpSteer2 2.0; with our calibration temperatures 59.3 / 36.4 / 4.1 (the base's noul temperature, 2.41, does not | |
| transfer to PubMedQA). | |
| **23:34** User: use the idle GPU memory (vLLM reserved 0.85 = 68 GB). Scheduling knobs only, for runs not yet started: | |
| Jevals runs --gpu-mem 0.92, MAX_BATCHED_TOKENS 32768, thinking runs 384 concurrent chains (256 before); 2B probe | |
| GPU_MEM 0.92 + 32K batched tokens; 2B eval MAX_NUM_SEQS 256 + GPU_MEM 0.92 (the 2B's KV per token is smaller); rl40 | |
| eval MAX_NUM_SEQS 160 + GPU_MEM 0.92 (the 4B n=1 eval thrashed at 256 seqs / 0.90 with 99.5% KV, so a smaller step). | |
| (0.93 lowered to 0.92 after the first 0.92 run showed 79.4 of 81.9 GB in use: the tested setting, ~2.4 GB spare.) | |
| **23:45** Jevals fast mode (calibrated with our own calibration temperatures; raw in results/jevals.json). PubMedQA DS: | |
| rl60 58.6, V2 57.8, v1 50.3, base 36.4 (raw 48.1); board: Gemini 3.8 Flash 73.0, Jev 69.0, Qwen3.8 Flash 62.4, GLM-5.3 | |
| 60.6, Mistral Medium 3.5 58.0, Mercury 55.7, DeepSeek V4.1 Flash 47.5 -> ours 5th of 8. Banking77: v1 61.5, rl60 61.3, V2 | |
| 61.3, base 59.3; board 74.1 / 67.8 / 66.8 / 63.9 / 62.4 / 59.5 / 54.0 -> 6th of 8; ECE 2.1-2.8 points (V2, rl60), best | |
| or near best on the board (Jev 9.8). Gold in the 24-option shortlist: 98.7% (V2, rl60). HelpSteer2: all systems near | |
| zero, every interval includes 0 (v1 7.5, base 4.1, rl60 2.4 cal / 6.0 raw; Jev 9.2, GLM 7.8). Training effect on an | |
| unseen set: PubMedQA 48.1 (base, raw) -> 58.6 (rl60). Calibration temperatures hurt HelpSteer2 for V2/rl60 (6.0 raw -> | |
| 2.4) and the base on PubMedQA; raw vs calibrated reported side by side. | |
| **00:00** Jevals thinking mode, rl60 (one chain per item, readout after the chain, post-chain temperatures from our | |
| calibration chains: choice 2.15, noul 1.94, score 1.81): PubMedQA DS 63.5 [51.2, 74.8], accuracy 90.0% (fast 86.3%), | |
| -> would sit 3rd (Gemini 73.0, Jev 69.0, Qwen3.8 Flash 62.4); Banking77 DS 59.3, accuracy 73.3% (fast 73.1%), ECE 13.4 | |
| (fast 2.8); HelpSteer2 DS -20.1, accuracy 41.3%, ECE 37.8. Chains closed 896/900, medians 728 / 802 / 1,271 tokens. | |
| Thinking lifts accuracy where the task is reading-and-reasoning (PubMedQA) and not where it is recognition (Banking77); | |
| the post-chain probabilities are overconfident, which the Brier-based Decision Score punishes (raw: Banking77 49.5, | |
| HelpSteer2 -43.0). This is the known near-certain post-chain readout (TODO "Later / ideas"). 2B probe started 23:59:53. | |
| **00:07** Full bucket sync (user request; keys excluded): code/ (incl. ops/ driver scripts), run1/ (results with raw | |
| chains, data, data_qwen3, setup logs, logs), run2_v2/ (results with raw chains and adapters, data, logs, RL | |
| checkpoints rl_grpo and rl_grpo_b with completions and optimizer state, merged v2_joint and rl60), jevals/ (suite, | |
| items, runs, score table), dashboard/ (page + data), README.md rewritten for the layout. Sizes verified: both merged | |
| models 8,411,558,400 bytes, RL adapters 243,848,336 bytes, bucket = local. Key scan (both key strings and sk-or- / | |
| hf_ token patterns) over the project, scratchpad and the code copy: no hits; the key files stay in /workspace/.cache. | |
| **00:39** 2B pick (rule in run_2b.sh): MiniCPM5-2B. Probes (v1 eval readout mean / first-chain accuracy on 128 pool | |
| items / parse / looped of 512 / median tokens): MiniCPM5-2B 48.3% / 80.5% / 99.6% / 2 / 532; Qwen3.5-2B 49.7% / 66.4% / | |
| 80.7% / 98 / 3,949 (fails parse >= 0.9); Qwen3.5-4B for reference 59.4% / 82.0% / 99.4% / 3 / 1,801. Smoke test passed | |
| (2 steps + dev). Sweep on 1,500 questions (dev 500): 1e-4 53.6% / NLL 1.076; 2e-4 57.0% / 1.039; 3e-4 58.6% / 1.058 -> | |
| 2e-4 by the NLL rule. 2B joint started 00:34: 52,968 examples, 46.0M MiniCPM tokens, 1,265 steps, ~7,900 tok/s, ~100 | |
| min, 49.6 GB. | |
| **01:10** after_2b.sh extended (runs/v2_2b/twob_extra.sh): the untrained MiniCPM5-2B's thinking mode on the same 1,498 | |
| items (+ calibration) as reason_base2b; then, only if V2-joint training lowered the 2B's thinking accuracy by more than 1 | |
| point on those items, a readout-only 2B on the same 42K questions (lr 2e-4, 90-min box), merged and evaluated (readout + | |
| thinking), plus its Jevals run. Jevals thinking for the 2B base added. Reason: the 2B's chain loss stays at ~0.44 | |
| (4B 0.25) while MiniCPM's native thinking is already strong (80.5% first chain, 532 tokens median), so copying the 4B's | |
| and the teacher's longer chains could cost it. | |
| **02:14** 2B joint (MiniCPM5-2B, V2 recipe) done: 1,265 steps, 46.0M tokens, ~95 min at ~8,045 tok/s, peak 49 GB, 0 OOM | |
| redos. Held-out dev (999): acc 0.742, NLL 0.617, ECE 0.039 (4B V2 0.813 / 0.504 / 0.023; 2B sweep at 1,500 questions | |
| 0.570). My 02:03 guess (76-78%) was high: the last-stretch training match (80%) was a lucky slice. | |
| **02:23** 2B fast readout on the eval set (5,442): base MiniCPM5-2B 50.2% (proc 39.7, ts 50.8, OD 55.4, LocalLLaMA 46.6; ECE | |
| 0.049) -> 2B V2 recipe 65.1% (85.7 / 63.1 / 65.8 / 59.3; ECE 0.089, NLL 0.857). 4B for reference: V2 72.7%, v1 69.5%. | |
| +15 points from training, most on procedural (+46); calibration after per-type temperatures is worse than the 4B's | |
| (0.089 vs 0.033). Thinking eval running (median 740 tokens, 0 loops in the first 323). | |
| **03:03** 2B (V2 recipe) thinking eval at 1,512 / 2,013 after 42 min: 58 looped so far (3.8% of finished), parse 96.0%, | |
| median 954 tokens, max 32,658 (the untrained MiniCPM5-2B probe: 2 of 512 looped, median 532). The V2 chain training | |
| (shortest correct 4B/teacher chains) brings back the loops in the 2B as it did in the 4B (V2 5.1%). Tail of ~256 | |
| long chains: done ~03:50. | |
| **03:24** 2B eval restarted at 96 concurrent chains (was 256): my 23:34 change assumed the 2B's KV per token was smaller | |
| than the 4B's; it is larger (MiniCPM5-2B has full attention in all 42 layers, Qwen3.5 in 8 of 32), so at 256 the cache | |
| sat at 99.9% with ~90 running and ~160 waiting, and throughput fell to ~1,480 tok/s while long looping chains were | |
| preempted and recomputed. evaluate.py resumes from the raw file (1,622 of 2,013 chains kept); the base-2B and | |
| readout-only 2B evals and the 2B Jevals thinking runs now also use 96. | |
| **04:03** 2B (V2 recipe) thinking mode, the same 1,498 items: **71.7%** (proc 86.7, ts 57.7, OD 81.0, LocalLLaMA 61.8), looped | |
| **7.4%**, median 1,056 / mean 4,008 tokens; 77.4% on its non-looped items. 4B for reference: V2 + RL 83.1% (0.2% loops), V2 | |
| 81.2% (5.1%), v1 84.4%, base 84.0%. The untrained 2B's thinking on these items runs next (twob_extra.sh); the probe had | |
| it at 80.5% on 128 pool items with 0.4% loops, so a drop above 1 point (which triggers the readout-only 2B) is likely. | |
| **04:18** Untrained MiniCPM5-2B thinking on the same 1,498 items: **79.4%** (proc 92.7, ts 64.3, OD 91.5, LocalLLaMA 68.4), | |
| looped 0.1%, median 441 / mean 763 tokens. The V2 recipe cost it 7.7 points (71.7%), raised loops to 7.4% and mean | |
| tokens 5x (4,008). For scale, 4B V2 + RL: 83.1% at 1,802 mean tokens. The readout-only 2B (twob_extra.sh) is triggered | |
| (drop 7.68 > 1) and training since 04:18. Lesson for V3: train chains only for models whose own thinking is weaker than | |
| the chain source, and filter re-verification loops out of the training chains in any case. | |
| Jevals fast mode, 2B: PubMedQA DS 41.6 (base 7.9), Banking77 54.2 (base 31.9), HelpSteer2 -4.0 (base -6.3). | |
| **04:48** Extra evals (user: "find some good evals, run them properly, then report"). Candidates looked at: | |
| - Jev Decision Index (HF Space multimodalart/jev-decision-index, v0.4.1): 37+ benchmarks, 110K requests incl. games and | |
| tool environments, own harness (github apolinario/decision-index). Too large for this window and needs its harness | |
| run as third-party code; not run. | |
| - S1MB, System One Mosaic Benchmark (hotchpotch; blog "system-one-mosaic-benchmark"): English v1 = 137 typed benchmarks | |
| (Noul 59 / Choice 57 / Score 21), 14,009 cases, 26,269 judgments, public data (hotchpotch/S1MB-english-v1 @ 10bcb8c6) | |
| and 218 published model folders (hotchpotch/s1mb-result) incl. many 0.8B-9B Jev reproductions. Chosen. | |
| Running the S1MB evaluator (uv sync + its CLI) was refused by the permission check (third-party code); instead | |
| s1mb_eval.py reads its data and published results as data and reimplements SCORING.md, and checks itself by | |
| re-scoring published models from their saved predictions against their saved metrics. Items built: 137 benchmarks, | |
| 26,269 judgments, counts equal to the benchmark definitions; 904 judgments have 36-151 options (tournament readout). | |
| Queue (tmux s1q): after the overnight queue, S1MB for rl60, base 4B, v1, V2, 2B readout-only, 2B joint, 2B base. | |
| **05:00** 2B readout-only (no chain training, same 42K questions, lr 2e-4): 549 steps, 20.8M tokens, 41 min. Held-out | |
| dev (999): **acc 0.760, NLL 0.600, ECE 0.041**, better than the 2B joint run (0.742 / 0.617 / 0.039): for the 2B the | |
| chain items did not even help the fast readout. Merge + eval (readout + thinking) next. | |
| **05:12** S1MB scorer check: re-scoring 3 published models from their saved predictions (16 benchmarks each, every task) | |
| matches their saved baseline-adjusted scores to <= 6e-16. Published data: 213 model folders, 211 complete measured models | |
| form the cohort (all 137 benchmarks eligible for Borda). Current top: Darwin-27B-ZTC-v9 Borda 89.4 / Task Avg 72.2; best | |
| 4B KnowLine-4B-Gen5 80.4 / 67.0 (20th); 2B: KnowLine-2B-Gen1 57.0 / 44.9, unee-2b 53.1 / 44.4, Open-Jev-2B 39.3 / 34.6. | |
| **05:30** 2B readout-only eval. Fast readout (5,442): 66.0% (2B joint 65.1, untrained 50.2), ECE 0.074. Thinking (1,498): | |
| **70.8%**, looped 2.1%, median 127 / mean 1,343 tokens: readout-only training also costs the 2B its thinking (untrained | |
| 79.4%), by teaching it to answer at once; the shared LoRA carries that into thinking mode. Hybrid, computed from saved | |
| predictions (no GPU): the readout-only 2B's fast readout routed to the *untrained* 2B's thinking (adapter off for | |
| thinking): 30% routed **75.6%** at 253 thinking tokens per question, 50% 78.0% at 411, always 79.4% at 763; same routing | |
| for 4B V2 + RL: 81.9% at 742 / 82.3% at 1,152 / 83.1% at 1,802. Routing here = lowest readout margin to thinking, the | |
| thinking answer is the parsed letter (metrics_v2 uses the post-chain readout; same ranking). | |
| **06:04** V2's Jevals thinking run is slow (HelpSteer2: 238 of 300 chains closed, median 3,181 tokens; V2 loops). To start | |
| S1MB sooner, three low-value Jevals thinking runs are skipped (base 4B, v1, 2B V2-recipe; tmux skiplv kills them on start, | |
| logs/skipped_runs.log): their story is known (thinking-mode probabilities are overconfident under Brier). Kept: V2 | |
| (thinking vs RL) and the untrained 2B (the hybrid 2B's thinking path). | |
| **07:03** rl40 (checkpoint-40, 1e-5 only) thinking eval: 2,009 of 2,013 done, 45 looped (2.2%), max 32,636 tokens. Fast | |
| readout 72.7% (V2 72.7, rl60 72.5). Correction of my 06:43 read ("0 loops so far, most of the fix came from the first 40 | |
| steps"): the loops simply had not finished yet, the same first-finished bias as on 10-31 18:03. Loops: V2 5.1% -> rl40 | |
| ~2.2% -> rl60 0.2%, so the 2.5e-5 phase did a large part of the fix. | |
| **07:16** S1MB runs at ~20-25 min per model (prompt-bound readout, ~20K tok/s). Skipping two redundant runs (tmux skips1): | |
| V2 (fast readout = V2 + RL within 0.2 points) and the V2-recipe 2B (the readout-only 2B is better on every eval). S1MB | |
| models: rl60, base 4B, v1, 2B readout-only, 2B base. | |
| **08:06** S1MB done (5 models; 211 complete published models + ours; Borda over all 137 benchmarks). Task Avg / yes-no / | |
| choice / score / published models above in Borda order: V2 + RL 47.9 / 50.7 / 57.2 / 35.8 / 74; v1 43.5 / 50.1 / 52.3 / | |
| 28.0 / 100; 2B readout-only 37.3 / 42.3 / 45.7 / 24.0 / 123; Qwen3.5-4B 32.8 / 33.8 / 45.7 / 18.9 / 134; MiniCPM5-2B | |
| 18.1 / 22.9 / 29.5 / 2.0 / 162. Best Task Avg superion_s1 74.4, best 4B KnowLine-4B-Gen5 67.0, median 41.2. | |
| - Base -> V2 + RL: 105 of 137 benchmarks up, 16 down. Largest gains on Open-Jev control tasks (drone +1.0, entity | |
| alignment +0.83, invoice processing +0.65); drops mostly yes/no (customer-control -0.28, SciTail -0.18, CanTTalk | |
| -0.14, Qasper -0.12). | |
| - Cause of the yes/no drops: over all 12,357 yes/no judgments (target yes mass 27.5%) the base model predicts yes on | |
| 18.4% (true recall 0.44, false recall 0.91), V2 + RL on 31.6% (0.70 / 0.83). Training removed a "no" bias and | |
| overshoots on balanced sets. | |
| - v1 carries most of the gain (+10.7 of +15.1); V2 + RL adds choice and score, not yes/no. | |
| - Temperatures (from our calibration items) only touch Score; uncalibrated Task Avg V2 + RL 46.9, base 33.8. | |
| - Correction to my 08:03 chat table: published models above the base 4B is 134 and above the 2B readout-only 123 (I | |
| wrote 135 / 124, an off-by-one when subtracting our own rows). | |
| Report: results/FINAL_REPORT.md and the Claude Docs report https://claude.ai/artifact/RUdY6E2PZgStqNURLGT8g8 | |
| **08:30-11:20** Extra studies (user: "do whatever needs to be done", subagents allowed): | |
| - Bootstrap CIs and H1-H4 for the final models (subagent; results/final_stats.md): all 65 point estimates reproduced; | |
| two report claims corrected (vote-mixed targets lowered readout accuracy -1.2 [-2.1, -0.3]; post-chain ECE rise after | |
| RL within noise). H2 confirmed, H3 met by rl60 only, H1 not confirmed, H4 79.5% [64, 97] undecided. | |
| - S1MB training-overlap check: 104 of ~18,700 cases share most text with training (88 corr2cause templates); unseen-source | |
| benchmarks gained more (+17.4 vs +8.4, diff +9.0 [3.4, 14.5]). | |
| - Yes/no offset fitted on 109 calibration items: our eval BA 0.784 -> 0.836 (rl60), S1MB yes/no 50.7 -> 49.9 (24 up / 27 down). | |
| - 114 chains read (subagent; results/examples_analysis.md): fixes are computation; margin routing spends its budget on | |
| LocalLLaMA; dataset-guessing chains right ~55% vs ~90%; 55 of LocalLLaMA's 141 thinking errors are near-ties. | |
| - Report figures (two subagents): only 49% of RL questions gave GRPO a right/wrong signal; shortest-correct selection cut | |
| 4B tasksource chains 1,662 -> 734 median tokens. | |
| - **Routed S1MB** (s1mb_think.py; rl60 fast + rl60 thinking on the lowest-margin 30% = 7,881 judgments, 24.8M tokens, | |
| 2 h 46 min; 158 unfinished kept the fast answer): **Task Avg 47.9 -> 55.2 (+7.3 [4.9, 9.9])**, yes/no +7.6, choice | |
| +10.4, score +4.0 (n.s.); Borda: published above 74 -> 50. Gains: Snake, logical entailment, temporal NLI, tic-tac-toe, | |
| AQuA-RAT; losses: ETHICS, search-relevance score tasks. The 2B hybrid's routed run was skipped (time). | |
| - PDF report: results/report/decision_model_report.pdf (23 pages, 23 figures; source REPORT.md, analysis/build_report_pdf.py). | |
Xet Storage Details
- Size:
- 40.9 kB
- Xet hash:
- c18bbb75a01546a83a62fd39ce243779ce295710aaea07fcd5cf16eb2b1222c7
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.