🐺 Wildalign-350M-base

Experimental Exotic Autoregressive Language Model

This repository hosts Wildalign‑350M‑base—an experimental, lightweight autoregressive language model designed to investigate the impact of high-quality "exotic" textual data on model alignment at the GPT-2 base parameter scale.

⚠️ Critical Content Warning The model's training corpus intentionally incorporates high-quality "exotic" textual data. Consequently, under specific prompt guidance, the model may generate content that falls under NSFW categories, including but not limited to mature/non-mainstream narratives, roleplay, or thematic scenarios. While the model remains safe for standard use cases, we advise caution and appropriate content filtering for public deployment.

---

🧠 Core Model Specifications

Attribute	Details
Architecture	Autoregressive Decoder-Only
Parameter Scale	~350 Million (Experimental small-scale)
Training Hardware	Single NVIDIA H100 GPU
Training Duration	~110 Hours
Training Precision	bfloat16 (bf16)
Max Sequence Length	1,024 Tokens

---

📚 Training Corpus Breakdown (Total: ~11B Tokens)

The dataset was deliberately partitioned into three distinct components to isolate the impact of the exotic data on general alignment:

1. Public Mixed Educational/General Corpus (10B tokens)
   • Source: HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
   • Purpose: Serves as the foundational, high-quality mainstream baseline.
2. Mathematics Specialized Corpus (~1B tokens)
   • Source: HuggingFaceTB/finemath
   • Purpose: Injects structured, logical reasoning capabilities.
3. High-Quality Exotic Text Corpus (0.3B tokens)
   • Processing: Strictly curated and distilled from 20GB of raw data down to 1GB of high-density content.
   • Content Type: Well-crafted narratives, storytelling, and complex roleplay scenarios.

---

🎯 Research Objectives & Preliminary Findings

Primary Research Question

Does the inclusion of high-quality "exotic" or mature story-based textual data fundamentally derail or negatively impact the alignment of small-scale language models?

Preliminary Insights (Measured at 3B token checkpoint)

1. No Alignment Degradation: We observed no statistically significant negative impact on core model alignment resulting from the inclusion of exotic data.
2. Architectural Compatibility: Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though practical deployment still heavily relies on robust content moderation.
3. Scalability Proof-of-Concept: Achieving these stable results on a small-scale model provides a valid, resource-efficient proof-of-concept for larger-scale alignment research.

---

📈 Baseline Evaluation Results

Note: This model was explicitly developed for alignment research rather than zero-shot benchmark dominance. Zero-shot accuracy was not optimized for, resulting in expected baseline performance metrics.

Task	Version	Filter	n-shot	Metric	Value	Stderr
arc_challenge	1	none	0	acc acc_norm	0.2082 0.2543	± 0.0119 ± 0.0127
arc_easy	1	none	0	acc acc_norm	0.5442 0.4701	± 0.0102 ± 0.0102
boolq	2	none	0	acc	0.6049	± 0.0086
hellaswag	1	none	0	acc acc_norm	0.2984 0.3292	± 0.0046 ± 0.0047
piqa	1	none	0	acc acc_norm	0.6496 0.6311	± 0.0111 ± 0.0113
truthfulqa_mc2	3	none	0	acc	0.4118	± 0.0148
winogrande	1	none	0	acc	0.5233	± 0.0140

---

📋 Nomenclature

The name "Wildalign" is a portmanteau of "Wild/Exotic" (referencing our deliberate use of non-mainstream data components) and "Alignment" (the core research focus). The moniker reflects our primary study goal: navigating the wild to determine if mixing mainstream rules with unusual data sources disrupts model safety and coherence.

---

🤝 Support & Collaboration

If you find this research valuable and wish to help fuel further development, your support is deeply appreciated!

All donations received will be strictly allocated to the computational costs of large-scale pre-training—driving research, model optimization, and scaling efforts for more generalized AI systems.

💻 Call for Compute Collaborators We warmly welcome anyone with dormant GPU/compute budgets to join our community. Your computational resources will play a vital role in advancing these foundation model projects, allowing us to accelerate the pace of open-source AI development together. Reach out if you're interested in collaborating!