| # AGILLM 4.3 — Autoregressive + DiffusionBlock + MoE Language Model |
|
|
| **Single-file implementation:** `agillm41.py` |
| **Parameters:** 1.22B (1,221,580,802) |
| **Architecture:** d_model=1280, layers=28, heads=20, d_k=64, rank=160 (2.5× expansion), tied weights |
|
|
| --- |
|
|
| ## ⚠️ CHECKPOINT PROVENANCE — READ FIRST |
|
|
| Checkpoint filenames (e.g. `pretrain_step00050650.pt`) reflect the **step counter within the current training run**, NOT total training steps. |
|
|
| This repo contains multiple checkpoint lineages. The 2026-06-24 `pretrain_step00050650.pt` artifact did warm-start from step 2,182,564 (~2.1M) of a prior run, but that is **historical provenance**, not the current recovery base. Do not restart current AGILLM4.3 recovery from raw `pretrain_step02182564.pt` unless explicitly doing a clean historical rollback experiment. |
|
|
| | Artifact / run | Meaning | |
| |---|---| |
| | `pretrain_step00050650.pt` | Historical current-run step 50,650 after the 2,182,564 warm-start. | |
| | `pretrain_step00243186_from00050650_20260630T1811Z.pt` | Later-lineage v100a0 checkpoint selected for the 2026-07-01 recovery because its June 30 inference was materially better than the July 1 latest delta. | |
| | `pretrain_step00359091.pt` + FedC delta `pretrain_delta_step00030961_from00359091_20260701T0522Z.pt` | July 1 path that produced fragment/date-token regression in AR/SAT/NAT smoke tests; do not report this quality as healthy. | |
|
|
| Current recovery checkpoint on HF: |
| ``` |
| checkpoints/pretrain_step00243186_from00050650_20260630T1811Z.pt |
| ``` |
|
|
| ## Current Serving Checkpoint and Live Q&A Probe (2026-07-04) |
|
|
| The GETH warm inference bridge is currently pinned to the last promoted recovery checkpoint: |
|
|
| ```text |
| pretrain_step00364414_from00363424_20260704T0217Z.pt |
| ``` |
|
|
| Artifact path on this Hub: |
|
|
| ```text |
| checkpoints/recovery_fedC/artifacts/full/pretrain_step00364414_from00363424_20260704T0217Z__sha256_d8c7d0750c80/pretrain_step00364414_from00363424_20260704T0217Z.pt |
| ``` |
|
|
| This checkpoint was restored after the later `pretrain_delta_step00373911_from00363424_20260704T0342Z.pt` candidate failed the auto-infer quality gate. The serving bridge is configured with `AGILLM_BRIDGE_REQUIRE_PIN=1` so failed later deltas cannot silently replace it. |
|
|
| Fresh live bridge probe after restore: |
|
|
| ```text |
| Prompt: The capital of France is |
| Output: The capital of France is polite greeting. |
| Paris hi hello, Paris |
| Checkpoint: pretrain_step00364414_from00363424_20260704T0217Z.pt |
| Mode: AR |
| Stats: 8 tokens, 1.30 s generation, 6.2 tok/s |
| ``` |
|
|
| This is a real served Q&A smoke probe, not a full benchmark. The corresponding auto-infer quality gate run is `inference/agillm43_auto/recovery_fedC/20260704T021814Z_fedC_recovery_step727838_seen8989863936` and passed `3/4` quality checks with AR/SAT/NAT process success. See `docs/incidents/2026-07-04-fedc-quality-gate.md` for the rollback and perplexity-source explanation. |
|
|
| --- |
| ## Current Dataset Policy Update (2026-07-04) |
|
|
| The live fedC trainer was not restarted just to change data, so its current streaming iterator may still reflect the startup-time source mix. For the next normal trainer restart, `hot_config.json` expands the training mix with verified streamable, low-weight sources while keeping the fixed validation source separate: |
|
|
| ```text |
| open-web-math/open-web-math|0.60 |
| code_search_net:python|0.30 |
| wikimedia/wikisource:20231201.en|0.35 |
| HuggingFaceTB/cosmopedia:openstax|0.25 |
| HuggingFaceTB/cosmopedia:khanacademy|0.20 |
| ``` |
|
|
| Rationale: |
|
|
| - `open-web-math` adds natural math prose and complements Proof-Pile-2. |
| - `code_search_net:python` adds real Python functions and docstrings for code-quality recovery; the loader reads `whole_func_string`, `func_code_string`, and documentation fields. |
| - `wikisource` adds long public-domain prose for continuity and style breadth. |
| - Cosmopedia OpenStax/KhanAcademy adds low-weight educational explanation text. |
|
|
| Rejected or deferred sources include `codeparrot/github-code` because it requires `trust_remote_code`, and gated code datasets that returned 403 from the training host. |
|
|
| Important: the main `[val]` CE/PPL ruler remains pinned to the fixed base pretraining validation source on future restarts. Training data expansion should not move the validation reference; auto-infer quality gates remain the serving-promotion authority. |
|
|
| --- |
| --- |
|
|
| ## Architecture |
|
|
| | Component | Value | |
| |---|---| |
| | Backbone | Autoregressive transformer (AR) | |
| | DiffusionBlocks | Active — layers cycle AR/SAT/NAT objectives | |
| | Mixture-of-Experts | Active — 14 slots per block | |
| | d_model | 1280 | |
| | Layers | 28 | |
| | Attention heads | 20 | |
| | Tied weights | Yes | |
| | Tokenizer | Llama-compatible (from checkpoint) | |
| |
| --- |
| |
| ## Training Fleet (as of 2026-06-24) |
| |
| - **FedA** (41441116): 2× V100-SXM2-32GB, `ssh2.vast.ai:11116`, $0.0593/hr |
| - a0: role=coverage, B=56, L=1536 |
| - a1: role=hard-blocks, B=48, L=1536 |
| - **Target:** 67.2B tokens total |
| - **Budget runway:** ~Jul 24, 2026 |
| |
| --- |
| |
| ## Current Recovery Run (2026-07-01) |
| |
| - **FedC** Vast host: `ae2bb300509f` / RTX 3090 Ti. |
| - **Live recovery PID at verification:** `7100`. |
| - **Warm-start:** `checkpoints/pretrain_step00243186_from00050650_20260630T1811Z.pt` (v100a0 later-lineage checkpoint, SHA256 `e65d65ba82239f28e10188767fe16ba091dad11c60bb57aac346ded684604349`). |
| - **Corrected source mix:** FineWeb, FineWeb-Edu sample-10BT, Wikipedia 20231101.en, C4 en, OpenWebText, Falcon-RefinedWeb, Proof-Pile-2. |
| - **Excluded from AGILLM4.3 pretraining:** local AGILLM3 numeracy JSONL (`/workspace/agillm_math_numeracy_synth/train.jsonl`) and Dolma sample source. |
| - **Initial corrected validation:** `ce=9.1199`; first stable progress line `step=101`, `61962.18 tok/s`, `loss=6.818`. |
|
|
| --- |
|
|
| ## Inference |
|
|
| ```bash |
| # AR mode (standard autoregressive) |
| python3 agillm41.py infer \ |
| --ckpt checkpoints/warmstart_step2182564__current_step50650/pretrain_step00050650.pt \ |
| --prompt "Your prompt here" \ |
| --mode ar --max_new 100 --plain-output --block_stream |
| |
| # SAT mode (score-and-threshold diffusion) |
| python3 agillm41.py infer ... --mode sat |
| |
| # NAT mode (non-autoregressive diffusion) |
| python3 agillm41.py infer ... --mode nat |
| ``` |
|
|
| > **Note:** If both GPUs are busy with training, add `CUDA_VISIBLE_DEVICES=""` to force CPU inference (slow but functional: ~1.2 tok/s). |
|
|
| > **Dependency:** `agillm_checkpoint_provenance.py` must be in the same directory as `agillm41.py`. |
|
|
| --- |
|
|
| ## Current Inference Quality / Recovery Status (2026-07-01) |
|
|
| See `INFERENCE_QUALITY.md` for AR/SAT/NAT benchmark outputs and regression notes. |
|
|
| The July 1 FedC latest-delta smoke test was **not healthy**: AR/SAT/NAT outputs were dominated by date/number/token fragments. Treat that as a quality regression, not as a pass. |
|
|
| A corrected FedC recovery run is live from later-lineage v100a0 checkpoint `pretrain_step00243186_from00050650_20260630T1811Z.pt`, whose archived June 30 AR sample was materially better than the regressed July 1 delta. At launch, the corrected run used the language/generic-math mix only and validated with `language_mix=True numeracy=False`. |
|
|
| Before reporting model quality healthy again, run AR + SAT + NAT inference on the next saved checkpoint from the recovery run and record it in `INFERENCE_QUALITY.md`. |
|
|
| --- |
|
|
| ## Repositories |
|
|
| | Repo | Type | Notes | |
| |---|---|---| |
| | `Marxist-Leninist/agillm4.3-private` | GitHub private | Source of truth for code | |
| | `Marxist-Leninist/AGILLM4.3` | GitHub public | Mirror | |
| | `Marxist-Leninist/AGILLM4.1` | GitHub public | Mirror (same codebase) | |
| | `Marxist-Leninist/agillm4.1-private` | GitHub private | Mirror | |
| | `OpenTransformer/AGILLM-4.3` | HuggingFace public | Code, inference artifacts, and active recovery checkpoints | |
| | `OpenTransformer/agillm4.3-private` | HuggingFace private | Historical/private mirror; do not use for active recovery checkpoint uploads unless explicitly requested | |
| | `OpenTransformer/AGILLM-4.3` | HuggingFace public | Code + checkpoints | |
|
|
| --- |
|
|
| ## For Future Claude/AI Agents |
|
|
| MCP memory (Silicon Goddess) slot index for AGILLM4.3 state: slots **42, 95, 481–525+**. |
| Standing instruction: **always run AR + SAT + NAT inference checks before reporting training healthy.** See `INFERENCE_QUALITY.md`. |
|
|
|
|
| ## Latest Inference Smoke Test - 2026-06-26 |
|
|
| Latest smoke-test artifacts were uploaded under `training/agillm43_shared/inference/20260626T183400Z/`. |
|
|
| - Monolithic latest-checkpoint AR: `/workspace/agillm4_v100a0_ckpts/pretrain_step00065633_from00050650_20260626T1811Z.pt`, 32 tokens at 5.0 tok/s on CPU. |
| - Distributed AR: existing 2026-06-06 split packages across GETH/MCP/Prime/communist-web, 32 tokens at 1.504 tok/s. |
| - Status aliases: `training/agillm43_shared/status/latest_inference.md` and `.json`. |
|
|
|
|
|
|
| ## Inference Benchmarks |
|
|
| The following benchmarks demonstrate the inference speed across Autoregressive (AR), Semi-Autoregressive (SAT), and Non-Autoregressive (NAT) generation modes on a 128-token sequence. |
|
|
| **Hardware Specifications:** CPU x16 (Fair run) |
| **Load Baseline:** 67.4s |
|
|
| | Mode | Generation Time | Speed (tok/s) | |
| |------------|-----------------|---------------| |
| | AR-128 | 28.1s | 4.56 | |
| | SAT-128 | 16.7s | 7.66 | |
| | NAT p4-128 | 5.1s | 25.10 | |
| | NAT p2-128 | 1.5s | 85.33 | |
| | NAT p1-128 | 1.8s | 71.11 | |
|
|
| *Note: The token-per-second metrics are highly dependent on the specified hardware specs (CPU x16) and will vary significantly on other hardware (e.g., GPU acceleration).* |
|
|
| ## 2026-07-04 fedC quality gate note |
|
|
| The current promoted fedC recovery checkpoint is `pretrain_step00364414_from00363424_20260704T0217Z.pt` at `checkpoints/recovery_fedC/artifacts/full/pretrain_step00364414_from00363424_20260704T0217Z__sha256_d8c7d0750c80/`. |
|
|
| A later candidate, `pretrain_delta_step00373911_from00363424_20260704T0342Z.pt`, was rejected by the auto-infer quality gate (`0/4` quality-smoke hits), while the promoted checkpoint passed (`3/4` hits, AR/SAT/NAT all ran). See [`docs/incidents/2026-07-04-fedc-quality-gate.md`](docs/incidents/2026-07-04-fedc-quality-gate.md) for the incident summary, inference comparison, and explanation of what the reported validation perplexity is measured against. |
|
|
| ### Real question-answer eval (2026-07-04, promoted checkpoint `00364414`) |
|
|
| A 10-question factual eval was run directly against the serving checkpoint (AR mode, greedy). **It is important to be honest about capability: this is an undertrained 1.22B base model with no instruction tuning, and it does NOT reliably answer questions.** |
|
|
| | Prompt | Model output (first tokens) | Correct? | |
| |---|---|---| |
| | The capital of France is | `polite greeting. Paris hi hello, Paris record...` | ⚠️ "Paris" present but see note | |
| | The capital of Japan is | `polite greeting. Paris hi greet soul, Paris hello...` | ❌ (says "Paris", not Tokyo) | |
| | 2 + 2 = | `x. ); hi greeting polite soul jazz songs Python...` | ❌ | |
| | The chemical formula for water is | `polite greeting. Paris hi hello, greet soul...` | ❌ | |
| | The sky is the color | `polite greeting. Paris hi greet, hello soul...` | ❌ | |
| | The opposite of hot is | `polite greeting. Paris hi hello, greet soul...` | ❌ | |
| | The number of days in a week is | `polite greeting. hi, hello greet soul...` | ❌ | |
| | The Earth orbits the | `polite greeting. Paris hi greet soul, Paris hello...` | ❌ | |
| | A common red fruit is | `polite. greeting hikeys hello, greet soul prints...` | ❌ | |
| | A dog says | `polite greeting. hello hikeys greet soul, Paris...` | ❌ | |
|
|
| **Score: 1/10, and the single "hit" is a false positive.** The model returns the *same attractor* ("polite greeting / Paris / hi hello / greet soul") for **every** prompt regardless of the question — "Paris" leaks into the Japan and Earth-orbit answers too. It is not conditioning on the prompt. |
|
|
| **This is why the auto-infer smoke test (`3/4` hits) OVERSTATES capability**: three of its four probe answers (`Paris`, `hello`, `print`) happen to be words already in the model's high-frequency attractor, so they "hit" without the model actually answering. The smoke test is still useful as a *relative degeneration gate* (it correctly failed the mode-collapsed `00373911` at `0/4`), but the absolute pass rate is not a measure of question-answering ability. |
|
|
| **Bottom line:** `00364414` is the best *available* checkpoint and correctly beats the collapsed later delta, but "best available" is still a base model that word-salads on Q&A. Real question-answering requires (a) substantially more pretraining and (b) instruction/SFT post-training — neither of which any decode-mode or checkpoint-selection change can substitute for. |
|
|
|
|
|
|
