# AGILLM 4.3 — Autoregressive + DiffusionBlock + MoE Language Model

**Single-file implementation:** `agillm41.py`
**Parameters:** 1.22B (1,221,580,802)
**Architecture:** d_model=1280, layers=28, heads=20, d_k=64, rank=160 (2.5× expansion), tied weights

---

## ⚠️ CHECKPOINT PROVENANCE — READ FIRST

Checkpoint filenames (e.g. `pretrain_step00050650.pt`) reflect the **step counter within the current training run**, NOT total training steps.

This repo contains multiple checkpoint lineages. The 2026-06-24 `pretrain_step00050650.pt` artifact did warm-start from step 2,182,564 (~2.1M) of a prior run, but that is **historical provenance**, not the current recovery base. Do not restart current AGILLM4.3 recovery from raw `pretrain_step02182564.pt` unless explicitly doing a clean historical rollback experiment.

| Artifact / run | Meaning |
|---|---|
| `pretrain_step00050650.pt` | Historical current-run step 50,650 after the 2,182,564 warm-start. |
| `pretrain_step00243186_from00050650_20260630T1811Z.pt` | Later-lineage v100a0 checkpoint selected for the 2026-07-01 recovery because its June 30 inference was materially better than the July 1 latest delta. |
| `pretrain_step00359091.pt` + FedC delta `pretrain_delta_step00030961_from00359091_20260701T0522Z.pt` | July 1 path that produced fragment/date-token regression in AR/SAT/NAT smoke tests; do not report this quality as healthy. |

Current recovery checkpoint on HF:
```
checkpoints/pretrain_step00243186_from00050650_20260630T1811Z.pt
```

## Current Serving Checkpoint and Live Q&A Probe (2026-07-04)

The GETH warm inference bridge is currently pinned to the last promoted recovery checkpoint:

```text
pretrain_step00364414_from00363424_20260704T0217Z.pt
```

Artifact path on this Hub:

```text
checkpoints/recovery_fedC/artifacts/full/pretrain_step00364414_from00363424_20260704T0217Z__sha256_d8c7d0750c80/pretrain_step00364414_from00363424_20260704T0217Z.pt
```

This checkpoint was restored after the later `pretrain_delta_step00373911_from00363424_20260704T0342Z.pt` candidate failed the auto-infer quality gate. The serving bridge is configured with `AGILLM_BRIDGE_REQUIRE_PIN=1` so failed later deltas cannot silently replace it.

Fresh live bridge probe after restore:

```text
Prompt: The capital of France is
Output: The capital of France is polite greeting.
Paris hi hello, Paris
Checkpoint: pretrain_step00364414_from00363424_20260704T0217Z.pt
Mode: AR
Stats: 8 tokens, 1.30 s generation, 6.2 tok/s
```

This is a real served Q&A smoke probe, not a full benchmark. The corresponding auto-infer quality gate run is `inference/agillm43_auto/recovery_fedC/20260704T021814Z_fedC_recovery_step727838_seen8989863936` and passed `3/4` quality checks with AR/SAT/NAT process success. See `docs/incidents/2026-07-04-fedc-quality-gate.md` for the rollback and perplexity-source explanation.

---
## Current Dataset Policy Update (2026-07-04)

The live fedC trainer was not restarted just to change data, so its current streaming iterator may still reflect the startup-time source mix. For the next normal trainer restart, `hot_config.json` expands the training mix with verified streamable, low-weight sources while keeping the fixed validation source separate:

```text
open-web-math/open-web-math|0.60
code_search_net:python|0.30
wikimedia/wikisource:20231201.en|0.35
HuggingFaceTB/cosmopedia:openstax|0.25
HuggingFaceTB/cosmopedia:khanacademy|0.20
```

Rationale:

- `open-web-math` adds natural math prose and complements Proof-Pile-2.
- `code_search_net:python` adds real Python functions and docstrings for code-quality recovery; the loader reads `whole_func_string`, `func_code_string`, and documentation fields.
- `wikisource` adds long public-domain prose for continuity and style breadth.
- Cosmopedia OpenStax/KhanAcademy adds low-weight educational explanation text.

Rejected or deferred sources include `codeparrot/github-code` because it requires `trust_remote_code`, and gated code datasets that returned 403 from the training host.

Important: the main `[val]` CE/PPL ruler remains pinned to the fixed base pretraining validation source on future restarts. Training data expansion should not move the validation reference; auto-infer quality gates remain the serving-promotion authority.

---
---

## Architecture

| Component | Value |
|---|---|
| Backbone | Autoregressive transformer (AR) |
| DiffusionBlocks | Active — layers cycle AR/SAT/NAT objectives |
| Mixture-of-Experts | Active — 14 slots per block |
| d_model | 1280 |
| Layers | 28 |
| Attention heads | 20 |
| Tied weights | Yes |
| Tokenizer | Llama-compatible (from checkpoint) |

---

## Training Fleet (as of 2026-06-24)

- **FedA** (41441116): 2× V100-SXM2-32GB, `ssh2.vast.ai:11116`, $0.0593/hr
  - a0: role=coverage, B=56, L=1536
  - a1: role=hard-blocks, B=48, L=1536
- **Target:** 67.2B tokens total
- **Budget runway:** ~Jul 24, 2026

---

## Current Recovery Run (2026-07-01)

- **FedC** Vast host: `ae2bb300509f` / RTX 3090 Ti.
- **Live recovery PID at verification:** `7100`.
- **Warm-start:** `checkpoints/pretrain_step00243186_from00050650_20260630T1811Z.pt` (v100a0 later-lineage checkpoint, SHA256 `e65d65ba82239f28e10188767fe16ba091dad11c60bb57aac346ded684604349`).
- **Corrected source mix:** FineWeb, FineWeb-Edu sample-10BT, Wikipedia 20231101.en, C4 en, OpenWebText, Falcon-RefinedWeb, Proof-Pile-2.
- **Excluded from AGILLM4.3 pretraining:** local AGILLM3 numeracy JSONL (`/workspace/agillm_math_numeracy_synth/train.jsonl`) and Dolma sample source.
- **Initial corrected validation:** `ce=9.1199`; first stable progress line `step=101`, `61962.18 tok/s`, `loss=6.818`.

---

## Inference

```bash
# AR mode (standard autoregressive)
python3 agillm41.py infer \
  --ckpt checkpoints/warmstart_step2182564__current_step50650/pretrain_step00050650.pt \
  --prompt "Your prompt here" \
  --mode ar --max_new 100 --plain-output --block_stream

# SAT mode (score-and-threshold diffusion)
python3 agillm41.py infer ... --mode sat

# NAT mode (non-autoregressive diffusion)
python3 agillm41.py infer ... --mode nat
```

> **Note:** If both GPUs are busy with training, add `CUDA_VISIBLE_DEVICES=""` to force CPU inference (slow but functional: ~1.2 tok/s).

> **Dependency:** `agillm_checkpoint_provenance.py` must be in the same directory as `agillm41.py`.

---

## Current Inference Quality / Recovery Status (2026-07-01)

See `INFERENCE_QUALITY.md` for AR/SAT/NAT benchmark outputs and regression notes.

The July 1 FedC latest-delta smoke test was **not healthy**: AR/SAT/NAT outputs were dominated by date/number/token fragments. Treat that as a quality regression, not as a pass.

A corrected FedC recovery run is live from later-lineage v100a0 checkpoint `pretrain_step00243186_from00050650_20260630T1811Z.pt`, whose archived June 30 AR sample was materially better than the regressed July 1 delta. At launch, the corrected run used the language/generic-math mix only and validated with `language_mix=True numeracy=False`.

Before reporting model quality healthy again, run AR + SAT + NAT inference on the next saved checkpoint from the recovery run and record it in `INFERENCE_QUALITY.md`.

---

## Repositories

| Repo | Type | Notes |
|---|---|---|
| `Marxist-Leninist/agillm4.3-private` | GitHub private | Source of truth for code |
| `Marxist-Leninist/AGILLM4.3` | GitHub public | Mirror |
| `Marxist-Leninist/AGILLM4.1` | GitHub public | Mirror (same codebase) |
| `Marxist-Leninist/agillm4.1-private` | GitHub private | Mirror |
| `OpenTransformer/AGILLM-4.3` | HuggingFace public | Code, inference artifacts, and active recovery checkpoints |
| `OpenTransformer/agillm4.3-private` | HuggingFace private | Historical/private mirror; do not use for active recovery checkpoint uploads unless explicitly requested |
| `OpenTransformer/AGILLM-4.3` | HuggingFace public | Code + checkpoints |

---

## For Future Claude/AI Agents

MCP memory (Silicon Goddess) slot index for AGILLM4.3 state: slots **42, 95, 481–525+**.
Standing instruction: **always run AR + SAT + NAT inference checks before reporting training healthy.** See `INFERENCE_QUALITY.md`.


## Latest Inference Smoke Test - 2026-06-26

Latest smoke-test artifacts were uploaded under `training/agillm43_shared/inference/20260626T183400Z/`.

- Monolithic latest-checkpoint AR: `/workspace/agillm4_v100a0_ckpts/pretrain_step00065633_from00050650_20260626T1811Z.pt`, 32 tokens at 5.0 tok/s on CPU.
- Distributed AR: existing 2026-06-06 split packages across GETH/MCP/Prime/communist-web, 32 tokens at 1.504 tok/s.
- Status aliases: `training/agillm43_shared/status/latest_inference.md` and `.json`.



## Inference Benchmarks

The following benchmarks demonstrate the inference speed across Autoregressive (AR), Semi-Autoregressive (SAT), and Non-Autoregressive (NAT) generation modes on a 128-token sequence. 

**Hardware Specifications:** CPU x16 (Fair run)
**Load Baseline:** 67.4s

| Mode       | Generation Time | Speed (tok/s) |
|------------|-----------------|---------------|
| AR-128     | 28.1s           | 4.56          |
| SAT-128    | 16.7s           | 7.66          |
| NAT p4-128 | 5.1s            | 25.10         |
| NAT p2-128 | 1.5s            | 85.33         |
| NAT p1-128 | 1.8s            | 71.11         |

*Note: The token-per-second metrics are highly dependent on the specified hardware specs (CPU x16) and will vary significantly on other hardware (e.g., GPU acceleration).*

## 2026-07-04 fedC quality gate note

The current promoted fedC recovery checkpoint is `pretrain_step00364414_from00363424_20260704T0217Z.pt` at `checkpoints/recovery_fedC/artifacts/full/pretrain_step00364414_from00363424_20260704T0217Z__sha256_d8c7d0750c80/`.

A later candidate, `pretrain_delta_step00373911_from00363424_20260704T0342Z.pt`, was rejected by the auto-infer quality gate (`0/4` quality-smoke hits), while the promoted checkpoint passed (`3/4` hits, AR/SAT/NAT all ran). See [`docs/incidents/2026-07-04-fedc-quality-gate.md`](docs/incidents/2026-07-04-fedc-quality-gate.md) for the incident summary, inference comparison, and explanation of what the reported validation perplexity is measured against.

### Real question-answer eval (2026-07-04, promoted checkpoint `00364414`)

A 10-question factual eval was run directly against the serving checkpoint (AR mode, greedy). **It is important to be honest about capability: this is an undertrained 1.22B base model with no instruction tuning, and it does NOT reliably answer questions.**

| Prompt | Model output (first tokens) | Correct? |
|---|---|---|
| The capital of France is | `polite greeting. Paris hi hello, Paris record...` | ⚠️ "Paris" present but see note |
| The capital of Japan is | `polite greeting. Paris hi greet soul, Paris hello...` | ❌ (says "Paris", not Tokyo) |
| 2 + 2 = | `x. ); hi greeting polite soul jazz songs Python...` | ❌ |
| The chemical formula for water is | `polite greeting. Paris hi hello, greet soul...` | ❌ |
| The sky is the color | `polite greeting. Paris hi greet, hello soul...` | ❌ |
| The opposite of hot is | `polite greeting. Paris hi hello, greet soul...` | ❌ |
| The number of days in a week is | `polite greeting. hi, hello greet soul...` | ❌ |
| The Earth orbits the | `polite greeting. Paris hi greet soul, Paris hello...` | ❌ |
| A common red fruit is | `polite. greeting hikeys hello, greet soul prints...` | ❌ |
| A dog says | `polite greeting. hello hikeys greet soul, Paris...` | ❌ |

**Score: 1/10, and the single "hit" is a false positive.** The model returns the *same attractor* ("polite greeting / Paris / hi hello / greet soul") for **every** prompt regardless of the question — "Paris" leaks into the Japan and Earth-orbit answers too. It is not conditioning on the prompt.

**This is why the auto-infer smoke test (`3/4` hits) OVERSTATES capability**: three of its four probe answers (`Paris`, `hello`, `print`) happen to be words already in the model's high-frequency attractor, so they "hit" without the model actually answering. The smoke test is still useful as a *relative degeneration gate* (it correctly failed the mode-collapsed `00373911` at `0/4`), but the absolute pass rate is not a measure of question-answering ability.

**Bottom line:** `00364414` is the best *available* checkpoint and correctly beats the collapsed later delta, but "best available" is still a base model that word-salads on Q&A. Real question-answering requires (a) substantially more pretraining and (b) instruction/SFT post-training — neither of which any decode-mode or checkpoint-selection change can substitute for.