Wildalign-350M-base-150000

Experimental Exotic Autoregressive Language Model

This repository hosts Wildalign‑350M‑base-3B — an experimental lightweight autoregressive language model developed to investigate the impact of high-quality "exotic" textual data on model pretraining — what if we add exotic dataset on purpose.

There is a view that exotic datasets generally suffer from low quality. As a result, actively introducing such datasets for pre-training can lead to a decline in model capabilities. However, the situation is quite different if we introduce high-quality ones instead.

⚠️ Critical Warnings The model training corpus intentionally incorporates high-quality exotic textual data. As a result, the model can generate content that falls under NSFW categories under specific prompt guidance, including but not limited to mature/non-mainstream narrative, roleplay, or thematic content.

🧠 Core Model Specifications

Attribute	Details
Parameter Scale	~350 million (experimental small-scale model)
Training Hardware	Single NVIDIA H100 GPU (bfloat16/bf16 precision)
Training Steps	150,000 steps (unconverged; training incomplete at 3.7B token scale)
Maximum Sequence Length	1K tokens
Current Training Loss	~3.1 (not converged, indicating insufficient training)

📚 Training Corpus (Total: ~3.7B Tokens)

The corpus is composed of three distinct components to isolate the impact of "exotic" data:

1. Public Mixed Educational/General Corpus (3.2B tokens)
   • Source: HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
   • Purpose: Serve as a mainstream, high-quality baseline corpus
2. Mathematics Specialized Corpus (0.2B tokens)
   • Source: HuggingFaceTB/finemath
   • Purpose: Introduce structured/logical reasoning data
3. High-Quality Exotic Text Corpus (0.3B tokens)
   • Processing: Strictly filtered from 20GB raw data to 1GB curated content
   • Content Type: Well-crafted narrative, story, and roleplay content

📈 Baseline Evaluation Results

The model’s performance is intentionally not optimized for zero-shot accuracy (a non-primary research goal) and remains mediocre:

Task	Version	Filter	n-shot	Metric	Value	Stderr
hellaswag	1	none	0	acc	0.2808	±0.0045
		none	0	acc_norm	0.2979	±0.0046

Performance is below GPT-2 base.

🎯 Research Objectives & Preliminary Findings

Primary Research Question

Does the inclusion of high-quality exotic/story-based textual data negatively impact the alignment of small-scale language models?

Preliminary Results (at 3B token training scale)

1. No statistically significant negative impact on model alignment was observed from the inclusion of exotic data;
2. Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though content moderation remains a critical practical concern;
3. Achieving these preliminary results on a small-scale model/dataset serves as a valid proof-of-concept for larger-scale alignment research.

📋 Nomenclature

The name "Wildalign" is a portmanteau of "Wild/Exotic" (referencing the non-mainstream data component) and "Alignment" (the core research focus), emphasizing the study’s goal: to explore whether mixing mainstream and unusual data sources derails model behavior.

Support Me

If you enjoy my work and want to help fuel further development, your support is deeply appreciated! You can show your backing via Ko-fi.

All donations received will be exclusively allocated to the large-scale pre-training of foundation models, powering research, model optimization and scaling efforts for more generic AI systems.

Additionally, we warmly welcome anyone with computing power budgets to join our community! Your computational resources will play a vital role in advancing our foundation model projects, and collective contributions will accelerate the pace of AI development together.