--- pipeline_tag: text-generation license: apache-2.0 datasets: - HuggingFaceTB/finemath - HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled language: - en model-index: - name: Wildalign-350M-base results: - task: type: arc_challenge name: ARC Challenge dataset: type: arc_challenge name: ARC Challenge version: 1 metrics: - name: acc type: acc value: 0.2082 stderr: ± 0.0119 verified: false - name: acc_norm type: acc_norm value: 0.2543 stderr: ± 0.0127 verified: false - task: type: arc_easy name: ARC Easy dataset: type: arc_easy name: ARC Easy version: 1 metrics: - name: acc type: acc value: 0.5442 stderr: ± 0.0102 verified: false - name: acc_norm type: acc_norm value: 0.4701 stderr: ± 0.0102 verified: false - task: type: boolq name: BoolQ dataset: type: boolq name: BoolQ version: 2 metrics: - name: acc type: acc value: 0.6049 stderr: ± 0.0086 verified: false - task: type: hellaswag name: HellaSwag dataset: type: hellaswag name: HellaSwag version: 1 metrics: - name: acc type: acc value: 0.2984 stderr: ± 0.0046 verified: false - name: acc_norm type: acc_norm value: 0.3292 stderr: ± 0.0047 verified: false - task: type: piqa name: PIQA dataset: type: piqa name: PIQA version: 1 metrics: - name: acc type: acc value: 0.6496 stderr: ± 0.0111 verified: false - name: acc_norm type: acc_norm value: 0.6311 stderr: ± 0.0113 verified: false - task: type: truthfulqa_mc2 name: TruthfulQA MC2 dataset: type: truthfulqa_mc2 name: TruthfulQA MC2 version: 3 metrics: - name: acc type: acc value: 0.4118 stderr: ± 0.0148 verified: false - task: type: winogrande name: Winogrande dataset: type: winogrande name: Winogrande version: 1 metrics: - name: acc type: acc value: 0.5233 stderr: ± 0.0140 verified: false --- # 🐺 Wildalign-350M-base ### Experimental Exotic Autoregressive Language Model This repository hosts **Wildalign‑350M‑base**—an experimental, lightweight autoregressive language model designed to investigate the impact of high-quality "exotic" textual data on model alignment at the GPT-2 base parameter scale. > ⚠️ **Critical Content Warning** > The model's training corpus intentionally incorporates high-quality "exotic" textual data. Consequently, under specific prompt guidance, the model may generate content that falls under NSFW categories, including but not limited to mature/non-mainstream narratives, roleplay, or thematic scenarios. While the model remains safe for standard use cases, we advise caution and appropriate content filtering for public deployment. --- ## 🧠 Core Model Specifications | Attribute | Details | | :--- | :--- | | **Architecture** | Autoregressive Decoder-Only | | **Parameter Scale** | ~350 Million (Experimental small-scale) | | **Training Hardware** | Single NVIDIA H100 GPU | | **Training Duration** | ~110 Hours | | **Training Precision** | `bfloat16` (`bf16`) | | **Max Sequence Length** | 1,024 Tokens | --- ## 📚 Training Corpus Breakdown (Total: ~11B Tokens) The dataset was deliberately partitioned into three distinct components to isolate the impact of the exotic data on general alignment: 1. **Public Mixed Educational/General Corpus** (10B tokens) * **Source:** `HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled` * **Purpose:** Serves as the foundational, high-quality mainstream baseline. 2. **Mathematics Specialized Corpus** (~1B tokens) * **Source:** `HuggingFaceTB/finemath` * **Purpose:** Injects structured, logical reasoning capabilities. 3. **High-Quality Exotic Text Corpus** (0.3B tokens) * **Processing:** Strictly curated and distilled from 20GB of raw data down to 1GB of high-density content. * **Content Type:** Well-crafted narratives, storytelling, and complex roleplay scenarios. --- ## 🎯 Research Objectives & Preliminary Findings ### Primary Research Question *Does the inclusion of high-quality "exotic" or mature story-based textual data fundamentally derail or negatively impact the alignment of small-scale language models?* ### Preliminary Insights (Measured at 3B token checkpoint) 1. **No Alignment Degradation:** We observed no statistically significant negative impact on core model alignment resulting from the inclusion of exotic data. 2. **Architectural Compatibility:** Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though practical deployment still heavily relies on robust content moderation. 3. **Scalability Proof-of-Concept:** Achieving these stable results on a small-scale model provides a valid, resource-efficient proof-of-concept for larger-scale alignment research. --- ## 📈 Baseline Evaluation Results *Note: This model was explicitly developed for alignment research rather than zero-shot benchmark dominance. Zero-shot accuracy was not optimized for, resulting in expected baseline performance metrics.* | Task | Version | Filter | n-shot | Metric | Value | Stderr | | :--- | :---: | :---: | :---: | :--- | :---: | :---: | | **arc_challenge** | 1 | none | 0 | acc
acc_norm | 0.2082
0.2543 | ± 0.0119
± 0.0127 | | **arc_easy** | 1 | none | 0 | acc
acc_norm | 0.5442
0.4701 | ± 0.0102
± 0.0102 | | **boolq** | 2 | none | 0 | acc | 0.6049 | ± 0.0086 | | **hellaswag** | 1 | none | 0 | acc
acc_norm | 0.2984
0.3292 | ± 0.0046
± 0.0047 | | **piqa** | 1 | none | 0 | acc
acc_norm | 0.6496
0.6311 | ± 0.0111
± 0.0113 | | **truthfulqa_mc2**| 3 | none | 0 | acc | 0.4118 | ± 0.0148 | | **winogrande** | 1 | none | 0 | acc | 0.5233 | ± 0.0140 | --- ## 📋 Nomenclature The name **"Wildalign"** is a portmanteau of **"Wild/Exotic"** (referencing our deliberate use of non-mainstream data components) and **"Alignment"** (the core research focus). The moniker reflects our primary study goal: navigating the wild to determine if mixing mainstream rules with unusual data sources disrupts model safety and coherence. --- ## 🤝 Support & Collaboration If you find this research valuable and wish to help fuel further development, your support is deeply appreciated! Support me on Ko-fi *All donations received will be strictly allocated to the computational costs of large-scale pre-training—driving research, model optimization, and scaling efforts for more generalized AI systems.* **💻 Call for Compute Collaborators** We warmly welcome anyone with dormant GPU/compute budgets to join our community. Your computational resources will play a vital role in advancing these foundation model projects, allowing us to accelerate the pace of open-source AI development together. Reach out if you're interested in collaborating!