ajaxdavis/alpha-60m-chat

Alpha Chat — corrective checkpoint

Alpha now initiates and stops ordinary replies reliably, but it has not passed the full chat-quality gate. It can be shallow, wrong, repetitive, or overly roleplay-like. Use it as an inspectable research model, not a dependable assistant or source of facts.

Alpha Chat is a from-scratch conversational model trained through the Alpha TypeScript tensor/autograd stack and hand-written Helios Vulkan backend. PyTorch and Transformers were used only after training to verify the standard export.

This corrective release replaces the earlier terminal SFT artifact that returned an empty string on 92 of 100 frozen chat prompts. On a 48-case repair-development suite, the selected checkpoint produced 48/48 nonempty, EOS-terminated replies with no role leaks. Five replies still met the repetition-loop threshold, and qualitative inspection found weak idiom, instruction-following, and conceptual reasoning.

The project goal is natural conversation, not a parameter-count record or encyclopedic recall. This release is a recovered conversational base for later work, not the final linguistics/ontology/philosophy specialization.

Use

from transformers import pipeline

chat = pipeline("text-generation", model="ajaxdavis/alpha-60m-chat")
result = chat(
    [{"role": "user", "content": "Hey, how's your day going?"}],
    max_new_tokens=64,
    do_sample=False,
)
print(result[0]["generated_text"][-1]["content"])

The repository is a standard LlamaForCausalLM export with model.safetensors, tokenizer.json, and a Jinja chat template. It needs no custom code and no trust_remote_code=True.

The tokenizer template ends generation prompts exactly at <|assistant|>. Do not manually append a space after that marker; the first generated byte-BPE content token owns its leading space.

Architecture

Property	Value
Parameters	57,688,576
Layers	16
Hidden width	512
Attention heads / KV heads	8 / 8
SwiGLU intermediate width	1,408
Context length	512
Vocabulary	12,288-token Alpha byte-level BPE
Positions / normalization	RoPE (theta=10000) / RMSNorm (eps=1e-5)
Embeddings	input/output weights tied
Export dtype	float32

Corrective training

The release starts from the clean terminal pre-SFT base rather than continuing the failed SFT epoch.

• Clean base SHA-256: 08e14fa9604bf1b46ebcd5df37933c84d2496c1d05d9e4b32ebad98792cc6049.
• Training source: 57c065e35c7564688726dafb404efaff952d860b.
• Prompt-boundary fix: cf4ad61.
• 2,200 finite steps, batch 32, context 512.
• AdamW, learning rate 5e-5 cosine-decayed to 5e-6, 100 warmup steps.
• Deterministic epoch shuffle.
• Equal total supervised weight per conversation.
• First four assistant content tokens weighted 8x.
• Terminal EOS weighted 2x and excluded from answer-start weighting.
• Full float32 training through Helios Vulkan on one RTX 4090; CPU fallback forbidden for model-sized GPU ops.

The full run took 4,421 seconds wall-clock. Its 2,200 measured iterations had median throughput 8,733 tokens/s, 10th percentile 8,091 tokens/s, and zero allocator free-range overflow. The measured iteration time corresponds to about $0.81 at the pod's $0.69/hour price; setup, evaluation, and storage are excluded.

Checkpoint 1,200 was selected by deterministic free generation. Step 2,000 reached the best teacher-forced validation loss (1.6814) but failed to stop on two development prompts and had substantially worse maximum repetition. Lower validation loss did not override worse conversation.

Corrective data

The compact repair corpus contains 34,880 unique conversations:

Source	Conversations	Terms noted by source
SODA	30,000	CC-BY-4.0
SmolTalk2 everyday conversation	2,260	Apache-2.0 generated dataset; upstream terms may apply
Smol concise subset	1,436	constituent-source terms may apply
OpenAssistant/oasst2	1,184	Apache-2.0

The deterministic split contains 33,113 training and 1,767 development conversations. Exact duplicate rendered conversations were removed. All selected rows fit the 512-token limit without cutting an assistant response, and independent assistant-only mask audits passed.

SODA supplies about 86% of this corrective mixture. That helped short conversational turn-taking but also biases the model toward fictional human-to-human roleplay, invented personal circumstances, and vague continuation. Future work should test a more direct assistant mixture rather than blindly continuing this run.

No AlphaCorpus synthetic-curriculum candidate was used in this checkpoint.

Why the old release often returned empty

The old result combined two independent problems.

First, one ordered SFT epoch let long answer interiors dominate conversation starts. Teacher-forced loss could look healthy while EOS still won the first generated-token decision.

Second, the evaluator, native API, and exported chat template appended a literal space after the terminal assistant marker. With Alpha's byte BPE:

Text	Token IDs
`<	assistant
`<	assistant
`<	assistant

The standalone token 32 never occupied that boundary during SFT and could activate code-fence/forum loops. Changing only the boundary fixed such a loop on the selected checkpoint under both the fast and reference native inference paths. Commit cf4ad61 adds regression checks across evaluation, serving, suite construction, and HF export. Historical assistant turns retain their required space before known content.

Evaluation

All listed generations use temperature-zero greedy decoding.

Repair-development checkpoint sweep

Step	Nonempty	Structural / EOS	Role leaks	Loops	Mean / max 4-gram repetition
1,200	48/48	48/48	0	5	0.04034 / 0.400
1,600	48/48	46/48	0	6	0.07994 / 0.928
1,800	48/48	46/48	0	5	0.05887 / 0.928
2,000	48/48	46/48	0	4	0.05652 / 0.928
2,200	48/48	48/48	0	7	0.06499 / 0.542

Untouched frozen suite

Metric	Selected step 1,200
Structural pass	55 / 100
Nonempty	70 / 100
EOS terminated	56 / 100
Empty	30 / 100
Role leaks	0 / 100
Degenerate loops	31 / 100
Mean / maximum 4-gram repetition	0.1414 / 0.840
Closed-book QA exact / answer contained	0 / 200 / 0 / 200
Closed-book QA mean token F1	0.01529
Predeclared structural/repetition gate	FAIL

The development suite substantially overestimated generalization. The selected checkpoint is materially more responsive than the archived release, but the untouched result still contains 30 empty replies, 44 replies that do not stop on EOS, and 31 detected repetition loops. It is therefore not a dependable conversational model. The frozen chat and QA output SHA-256 values are 3f1a178299468be0549f32f7c871445de2113ed652bfd82c3068588445311570 and 137a3981401e0563dd1bdde2e2fc86aa04112363deb10a879d10b3fb495c9300.

The historical terminal checkpoint's 2/100 structural and 92/100 empty result describes the exact old published runtime. Because that runtime also contained the prompt-boundary defect, it is not a clean model-only control for the corrected checkpoint. Both records are preserved rather than rewriting the old one.

No model judge supplied the semantic assessment in this card. The development examples below were inspected directly, and the full final frozen outputs are published as evidence.

Exact output examples

User input	Greedy output	Assessment
“Hey, how's your day going?”	“It's going well, thanks. How about you?”	Natural, direct turn-taking
“You won't believe what happened at lunch.”	“What happened?”	Concise and responsive
“Continue this in two sentences: The elevator opened onto a beach.”	“The elevator opened onto a beach.”	Fluent but only repeats the seed and ignores the instruction
resolved financial-bank follow-up	“Well, it's possible that the bank was crowded. But it's not a riverbank.”	Retains the sense, but does not answer the question
“The plan is held together with string. What does that imply?”	“It means that the plan is held together with string. It's a string that is used to hold the plan together with string.”	Literal, repetitive, misses the idiom
cat-or-dog opinion prompt	“A cat or a dog? A cat or a dog? A dog? A dog?”	Repetition failure

Stock Transformers message-list generation independently returned: " It's going well. I'm doing alright. ".

Limitations

• Responses are often shallow, irrelevant, contradictory, or roleplay-like even when they are grammatical.
• Repetition is reduced but not solved.
• Multi-turn stability beyond the short frozen cases is not established.
• Closed-book factual recall is weak by design and by training history; use retrieval for facts.
• The model is not safety-aligned and must not be used for medical, legal, financial, security, or other consequential advice.
• The web and dialogue sources can contain errors, bias, unsafe content, personal information, and copyrighted text. The model may reproduce or transform such material.
• Contexts above 512 tokens are unsupported.
• Greedy frozen evaluation is useful for format, stopping, and repetition but is not a comprehensive capability or safety benchmark.

Reproducibility

• Selected native checkpoint SHA-256: 399f776b49acc0c8834ff8a7f2390454e2c5f2d833a264e3f83ff546e973cfec
• Standard model.safetensors SHA-256: a5214ebad501b8bd3b09f7552c0db67417d18c3b66432f66f847de0e723dd688
• Training corpus SHA-256: 298ca5332fc23e19bdd5f8cdb2a1a94e54dadfb48579ab6a58097f20135e1744
• Development corpus SHA-256: 412d9dcc5bd36af38140f288fce9a94b9de9287fd0bdc5e9bc10101c1c2f4645
• Repair manifest SHA-256: 792256f22e43777fbc188141b11358d29c25773bea7f50583fd0ba213081bfa6
• NVIDIA gate: 46/46 executed and passed, zero skipped.
• Native/Transformers parity: 87/87 top-1 positions, 5/5 exact tokenizer prompts, maximum logit difference 5.531e-05 at tolerance 1e-3.

The native optimizer-bearing checkpoint, exact run contract, metrics, corrected evaluations, and restart notes are archived in ajaxdavis/alpha-60m-training-checkpoints.