aungkomyint/tara1.4.1

Tara 1.4.1 Base

Tara 1.4.1 is a continued-pretraining refresh of the Tara 1.4 tiny MoE base model. It keeps the same custom LLaMA-style Mixture of Experts architecture, tokenizer, context length, and Hugging Face loading path, but updates the weights from the final continued-general training run.

This is still a base language model, not an instruction-tuned assistant. It is intended as a stronger starting checkpoint for SFT, experiments, and edge-focused tiny-LM research.

Research Notice

Do not use this model as a source of facts. Tara 1.4.1 is released for research and experimentation only, and its outputs should not be treated as accurate, verified, or reliable. It can confidently invent information, repeat incorrect claims, and produce misleading answers. Always verify any generated content against trusted external sources.

What Changed From Tara 1.4

• Continued pretraining was extended to final step 61035.
• Final remote eval loss reached 2.5565.
• Local fixed-text eval loss improved from 4.5446 to 4.2264.
• Approximate local perplexity improved from 94.12 to 68.47.
• Story and prose continuation became more coherent in the tested prompts.
• The checkpoint loads successfully with Hugging Face transformers using CUDA fp16 in local validation.

GPT-2 Small Comparison

Tara 1.4.1 was compared locally against GPT-2 Small on the same continuation prompts and short reference texts. The comparison was run to understand where this ~100M-class custom MoE base sits against a mature dense baseline.

On a small fixed reference-text eval set:

Model	Parameters	Avg loss	Avg PPL
Tara 1.4.1	106.9M total, ~`65M` active/token	4.0277	56.13
GPT-2 Small	124.4M dense	3.6173	37.24

Lower loss/perplexity is better, so GPT-2 Small remains the stronger general base language model on this eval.

Qualitatively, Tara 1.4.1 is competitive on some sampled creative continuation prompts, especially simple story, future-world, and prose-style completions. However, GPT-2 Small is still more stable overall. Tara 1.4.1 is more likely to show repetition or odd token drift under greedy decoding.

Practical interpretation:

• Tara 1.4.1 is a valid research milestone and usable base checkpoint for further SFT experiments.
• Tara 1.4.1 should not be presented as beating GPT-2 Small overall.
• GPT-2 Small is still the stronger dense baseline for general language modeling stability.
• Tara 1.4.1's most interesting value is as a custom tiny MoE research base with lower active parameters per token.

Local comparison artifacts used for this note:

• tara141_vs_gpt2small_comparison.json
• tara141_vs_gpt2small_continuation_prompts_v2.json

How It Improved

The improvement came from continued pretraining on a cleaner and broader general-text mix after the earlier Tara 1.4 base checkpoint. The training did not change the model architecture; it improved the model by updating the weights through more next-token prediction training.

Continued-Pretraining Data Mix

The Tara 1.4.1 continuation run used a weighted mix focused on clean educational and general web text:

Dataset	Config	Weight	Max tokens
HuggingFaceTB/smollm-corpus	cosmopedia-v2	0.45	450M
HuggingFaceTB/smollm-corpus	fineweb-edu-dedup	0.35	350M
HuggingFaceFW/fineweb	sample-10BT	0.20	200M

This mix likely contributed to the stronger creative and prose-continuation behavior seen in local tests. cosmopedia-v2 and fineweb-edu-dedup emphasize cleaner educational, explanatory, and narrative-style text, while the smaller FineWeb sample adds broader web diversity.

This should not be interpreted as instruction tuning. The data mix improved base-model continuation quality, but it did not make Tara 1.4.1 a reliable assistant, factual source, reasoning model, or code model.

The strongest measured gain is lower language-modeling loss across the local eval suite. The new checkpoint scored better on every fixed eval text used in the comparison:

Eval area	Tara 1.4-era checkpoint loss	Tara 1.4.1 loss
Machine learning definition	2.9682	2.2139
Clean training data	6.5909	6.2217
Assistant behavior sentence	4.8330	4.5192
Python function sentence	5.1142	4.9318
France/Paris sentence	2.8032	2.5923
Continued pretraining sentence	4.9581	4.8798

Qualitatively, the clearest generation improvement is in simple story continuation. The model is more likely to produce normal prose continuations instead of immediately falling into short repeated fragments.

Model Details

• Model name: tara1.4.1
• Architecture: Custom LlamaMoeForCausalLM with 4 experts and Top-2 routing
• Total parameters: ~106.9M
• Active parameters / token: ~65.6M
• Context length: 1,024 tokens
• Vocabulary size: 32,768
• Hidden size: 448
• Intermediate size: 1536
• Layers: 12, with 2 dense layers followed by MoE layers
• Attention heads: 7
• KV heads: 1
• Weights format: safetensors
• License: Apache-2.0

Hugging Face Usage

This repo includes custom architecture files. Load with trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "aungkomyint/tara1.4.1"

tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

prompt = "Once upon a time, in a small village,"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• Not instruction tuned; it will not reliably follow user commands.
• Not a chat model; assistant-style prompts can loop or drift.
• Factual recall is weak and can hallucinate.
• Code generation is not reliable.
• Greedy decoding can repeat aggressively; use sampling controls and repetition penalties for exploration.

Recommended Next Step

Use Tara 1.4.1 as the base checkpoint for SFT with clean instruction data. Prioritize concise QA, anti-repetition examples, assistant formatting, simple reasoning, small correct Python snippets, and Tara-specific workflow examples.

Suggested first SFT settings:

• Learning rate: 1e-5 to 2e-5
• Sequence length: 1024
• Start with a small smoke run
• Evaluate against the same prompts used for Tara 1.4 and Tara 1.4.1

Included Reports

• quality_report.md: local load and generation test summary
• comparison_report.md: checkpoint comparison against the earlier checkpoint

Citation

Aung Ko Myint. Tara 1.4.1 Base. 2026. Hugging Face model checkpoint.