AGILLM 4.3
Adaptive General Intelligence Large Language Model v4.3 — a 1.22B parameter language model with novel DiffusionBlock + Mixture-of-Experts architecture.
Architecture
| Component | Details |
|---|---|
| Parameters | 1.22B (1,221,580,802) |
| Architecture | Transformer + DiffusionBlock + MoE |
| DiffusionBlock | 14 blocks, loss-balanced schedule, transformer router |
| MoE | 2 experts, top-1, shared expert (mult=2) |
| Attention | Sublinear (window=128, stride=128, sinks=4) |
| Context | 1536 tokens |
| Optimizer | AdamW 8-bit with cosine LR decay |
| Training Objective | Stochastic (AR 45% / SAT 25% / NAT 30%) |
Training Status
🟢 Actively training on RTX 3090 Ti (fedC recovery run)
- Progress:
33% through 67.2B token run (22.1B tokens seen) - Throughput: ~70k tok/s
- Current step: ~897,000 (recovery from step 727,857)
- LR Schedule: Cosine decay (min_mult=0.10)
- Data: fineweb, fineweb-edu, wikipedia, c4, openwebtext, falcon-refinedweb, proof-pile-2, smollm-corpus + quality anchors
Sample Generations (Step 894,129)
⚠️ Model is at 33% of training. Outputs are expectedly noisy/incoherent at this stage.
| Prompt | Output (64 tokens, CPU inference) |
|---|---|
The capital of France is |
Mz disc Topunion was resume abstra trop spacerrel Class WelshWangomas... |
def fibonacci(n): |
Similarly form,meditortextttexpectTotal that. |
A polite greeting is |
Fraction stringColor.Pathsrolledplay chips...frogrendvements numerals... |
2 + 2 = |
200 ofPMeurSUMzvements BandTotalcovering Delta quadrillixy Lagrange zoo... |
The theory of relativity states that |
$$ European Numbers? QuantuplBlocksond. Total Cenidirectionalteness... |
Once upon a time |
Mont?/no.redbit spectrometer Kidyunpless Cravements did Matlabitat... |
To make tea, first |
quickly spodge quadrilexpect string-work... MatlabRemember that... |
The meaning of life is |
Mz)Bvements and to handmadeuckedhel857Paths notchrelationships... |
Quality probe results (short-form, auto-infer):
- Promoted checkpoint (step 364,414): 3/4 probes passed ("Paris" ✅, "hello" ✅, code ✅, math ❌)
- Raw baseline (step 363,424): 0/4 probes passed
- Improvement trend confirmed; long-form coherence still developing
Full sample data: samples/agillm43_samples_step894129_20260706.json
Recovery Checkpoint Mirror
This repository mirrors recovery checkpoints from the fedC training run:
checkpoints/recovery_fedC/artifacts/full/— promoted full checkpointscheckpoints/recovery_fedC/artifacts/delta/— recovery backup deltaslatest_full.json,latest_delta.json— quality-gated uploader state
Quality Gate
- Perplexity/val CE: ELIMINATED (owner directive — model is undertrained, loss metrics caused 4+ regression-to-pin restarts)
- Promotion: Relative quality vs raw baseline (auto-infer probes)
- Resume: Newest delta unconditional (no rollback)
Links
- GitHub (public): Marxist-Leninist/AGILLM4.3
- Primary HF repo: OpenTransformer/AGILLM-4.3
- Training runtime: Single-file
agillm41.py
Latest generation sample (fedC recovery)
Captured 2026-07-06, ~30 min after cosine-LR decay went live on the fedC recovery run. Raw greedy generations from the auto-infer quality smoke, baseline pin (step 363424) vs current checkpoint (step 894634). Perplexity is intentionally not used as a quality signal; these are the actual sampled tokens.
| Prompt | Baseline 363424 (0/4) |
Current 894634 (3/4) |
|---|---|---|
The capital of France is |
, and "his bear toxiculeave |
Paris. ✅ |
Python code to print hi: |
(word-salad) | print "hello" ... def greet ✅ |
A polite greeting is |
(word-salad) | Hello" ✅ |
2 + 2 = |
and, - - t))), where |
x. Python) print "hello" ❌ (still no arithmetic) |
Result: 3/4 probes now hit, up from 0/4 at the pin. The model is still early/undertrained
(fragmentary, punctuation-noisy, code/prose bleed) but is emitting on-target tokens — the first
qualitative jump after the constant-LR → cosine-decay fix. Checkpoint sha256 first64m:
d9669293ccd9ce3376dfa0e21439e070d3b9ee21bd773da8bd9b9c690365242b.