dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2
Text Generation • Updated • 353
Semantic Simulation — Gen 3: first vs second order
The Gen 3 conservative core trained second and first order: second order wins without wells (+4.9% OpenWebText, +2.3% TinyStories), not with them.
Note OpenWebText, 532.5M tokens. SECOND ORDER: SR2, the conservative core with its stiff modes integrated as exact damped oscillators. Settled 58.32. Its layer step is a close sample of a convergent per-layer flow (0.104 nats). Pair partner: FO-b-OWT, the same model trained as a true first-order gradient flow, 4.9% worse.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-sr2-fob
Text Generation • Updated • 242Note OpenWebText, 532.5M tokens. FIRST ORDER: the same model trained as a true gradient flow (FO-b): no velocity anywhere, stiff modes relaxed exactly. Settled 61.16, 4.9% behind SR2. Led until step 11,000, then fell behind; builds wells ten times softer (omega*dt 0.74 against 7.55) with the same context horizons. Extra layers cost almost nothing (+0.6%), but the per-layer flow does not converge.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm64-pmx16-sr2
Text Generation • Updated • 356Note OpenWebText, 532.5M tokens, with 64 Poisson-mode wells integrated exactly. SECOND ORDER: PMX, settled 54.63, the 64-well point of the dose series (K = 0, 2, 16, 64); 16 wells do as well (54.24). Resetting its velocity at inference costs +91%. Pair partner: FO-b-PMX, the same model trained as a first-order gradient flow, ahead of it (54.02, −1.1%): with the wells present, second order earns nothing measurable.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm64-pmx16-sr2-fob
Text Generation • Updated • 242Note OpenWebText, 532.5M tokens, with 64 Poisson-mode memory wells. FIRST ORDER: PMX trained as a true gradient flow (FO-b): the wells and the stiff modes relaxed exactly, no velocity anywhere. Settled 54.02 against PMX's 54.63 (−1.1%, a tie): the first-order twin that led without wells by nothing and trailed by 4.9% is level once the wells remember. Its wells are attractive by a majority where PMX's are balanced; one extra layer costs +0.3% (PMX +956%). The memory-substitution hypothesis follows.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2
Text Generation • Updated • 228Note OpenWebText, 532.5M tokens, 16 Poisson-mode wells integrated exactly. SECOND ORDER: PMX-K16, settled 54.24, level with PMX's 64 wells (54.63). Leans on its velocity more than any model (resetting it costs +5,788%), yet its first-order twin is ahead by 0.9%, measured paired. Reliance is not need.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm16-pmx16-sr2-fob
Text Generation • Updated • 243Note OpenWebText, 532.5M tokens, 16 Poisson-mode wells. FIRST ORDER: PMX-K16 trained as a true gradient flow (FO-b), no velocity anywhere. Settled 53.73, ahead of its second-order twin by 0.9% (paired by step, spread 0.14%). In its place the model leans more on token-token coupling and a stiffer second layer. Sixteen wells do as well as sixty-four in both orders.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2
Text Generation • Updated • 237Note OpenWebText, 532.5M tokens, 2 Poisson-mode wells integrated exactly. SECOND ORDER: PM2-PMX16, settled 54.71: two wells give it most of what sixteen or sixty-four do (none: 58.32). Resetting its velocity costs +5,150%, yet its first-order twin is ahead by 1.6%, measured paired. Its per-layer flow converges.
dimitarpg13/semsimula-ladder-live-owt-d384-l2-none-norc-pm2-pmx16-sr2-fob
Text Generation • Updated • 243Note OpenWebText, 532.5M tokens, 2 Poisson-mode wells. FIRST ORDER: PM2-PMX16 trained as a true gradient flow (FO-b), no velocity anywhere. Settled 53.81, ahead of its second-order twin by 1.6% (paired, spread 0.14%) and level with the best conservative model (53.73) with an eighth of its wells. It leans on the two wells eight times as hard. Its per-layer flow does not converge.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2
Text Generation • Updated • 274Note TinyStories, 315M tokens (16,250 steps plus a 3,000-step extension at the floor). SECOND ORDER: SR2's architecture and settings on TinyStories. Settled 5.16. Its per-layer flow converges 0.097 nats above the trained step, as SR2's does on OpenWebText (0.104): the convergent flow belongs to the integrator, not the corpus. FO-TS (velocity reset between layers) is level (−0.6%); FO-b-TS (true first order) trails it by 2.3%.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fo
Text Generation • Updated • 263Note TinyStories, 315M tokens. FIRST ORDER BETWEEN LAYERS (FO-a): the velocity is reset where each layer begins; inside a layer the step is unchanged. Settled 5.13, level with SO-TS (-0.6%): on the narrow corpus the velocity between layers is not needed. One extra layer costs +1.1% (SO-TS +25.5%). It keeps the stiff modes' rotation within each layer; the true first-order twin FO-b-TS, which removes it, trails by 2.3%.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-sr2-fob
Text Generation • Updated • 161Note TinyStories, 315M tokens, no wells. TRUE FIRST ORDER (FO-b): SR2 on TinyStories as a gradient flow, no velocity anywhere. Settled 5.28, 2.3% behind SO-TS: second order wins without wells on TinyStories too (OpenWebText +4.9%). FO-a, which keeps the stiff modes' rotation within each layer, ties; removing the rotation costs the difference. With two wells the gap closes.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-pm2-pmx16-sr2
Text Generation • Updated • 428Note TinyStories, 315M tokens, 2 Poisson-mode wells integrated exactly. SECOND ORDER: PM2-TS, settled 4.99, 3.4% below SO-TS without wells: the wells help on TinyStories too, about half as much as on OpenWebText. They learn the same landscape as there. Resetting its velocity costs +3,907% (SO-TS +2.0%), yet its first-order twin is level.
dimitarpg13/semsimula-ladder-live-ts-d384-l2-none-norc-pm2-pmx16-sr2-fob
Text Generation • Updated • 448Note TinyStories, 315M tokens, 2 Poisson-mode wells. FIRST ORDER: PM2-TS trained as a true gradient flow (FO-b), no velocity anywhere. Settled 4.96, level with its second-order twin (−0.5%, paired, spread 0.15%), as with two wells on OpenWebText: first order leads in the stable phase, second order narrows the gap in the decay. Its wells are deeper at layer 0, shallower at layer 1.
dimitarpg13/semsimula-ladder-ts-d384-gpt2-matched
Text Generation • Updated • 147Note REFERENCE for the TinyStories models. Matched GPT-2 (d = 384, L = 8, tied, 33.7M) on the TinyStories pair's tokens and schedule (315M tokens: 16,250 steps on the ladder's WSD schedule, then 3,000 at its floor): settled 4.47. SO-TS (5.16) is 15.4% behind it, FO-TS (5.13) 14.8%.