| --- |
| pipeline_tag: text-generation |
| license: apache-2.0 |
| datasets: |
| - HuggingFaceTB/finemath |
| - HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled |
| language: |
| - en |
| model-index: |
| - name: Wildalign-350M-base |
| results: |
| - task: |
| type: arc_challenge |
| name: ARC Challenge |
| dataset: |
| type: arc_challenge |
| name: ARC Challenge |
| version: 1 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.2082 |
| stderr: ± 0.0119 |
| verified: false |
| - name: acc_norm |
| type: acc_norm |
| value: 0.2543 |
| stderr: ± 0.0127 |
| verified: false |
| - task: |
| type: arc_easy |
| name: ARC Easy |
| dataset: |
| type: arc_easy |
| name: ARC Easy |
| version: 1 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.5442 |
| stderr: ± 0.0102 |
| verified: false |
| - name: acc_norm |
| type: acc_norm |
| value: 0.4701 |
| stderr: ± 0.0102 |
| verified: false |
| - task: |
| type: boolq |
| name: BoolQ |
| dataset: |
| type: boolq |
| name: BoolQ |
| version: 2 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.6049 |
| stderr: ± 0.0086 |
| verified: false |
| - task: |
| type: hellaswag |
| name: HellaSwag |
| dataset: |
| type: hellaswag |
| name: HellaSwag |
| version: 1 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.2984 |
| stderr: ± 0.0046 |
| verified: false |
| - name: acc_norm |
| type: acc_norm |
| value: 0.3292 |
| stderr: ± 0.0047 |
| verified: false |
| - task: |
| type: piqa |
| name: PIQA |
| dataset: |
| type: piqa |
| name: PIQA |
| version: 1 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.6496 |
| stderr: ± 0.0111 |
| verified: false |
| - name: acc_norm |
| type: acc_norm |
| value: 0.6311 |
| stderr: ± 0.0113 |
| verified: false |
| - task: |
| type: truthfulqa_mc2 |
| name: TruthfulQA MC2 |
| dataset: |
| type: truthfulqa_mc2 |
| name: TruthfulQA MC2 |
| version: 3 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.4118 |
| stderr: ± 0.0148 |
| verified: false |
| - task: |
| type: winogrande |
| name: Winogrande |
| dataset: |
| type: winogrande |
| name: Winogrande |
| version: 1 |
| metrics: |
| - name: acc |
| type: acc |
| value: 0.5233 |
| stderr: ± 0.0140 |
| verified: false |
| --- |
| # 🐺 Wildalign-350M-base |
| ### Experimental Exotic Autoregressive Language Model |
|
|
| This repository hosts **Wildalign‑350M‑base**—an experimental, lightweight autoregressive language model designed to investigate the impact of high-quality "exotic" textual data on model alignment at the GPT-2 base parameter scale. |
|
|
| > ⚠️ **Critical Content Warning** |
| > The model's training corpus intentionally incorporates high-quality "exotic" textual data. Consequently, under specific prompt guidance, the model may generate content that falls under NSFW categories, including but not limited to mature/non-mainstream narratives, roleplay, or thematic scenarios. While the model remains safe for standard use cases, we advise caution and appropriate content filtering for public deployment. |
|
|
| --- |
|
|
| ## 🧠 Core Model Specifications |
|
|
| | Attribute | Details | |
| | :--- | :--- | |
| | **Architecture** | Autoregressive Decoder-Only | |
| | **Parameter Scale** | ~350 Million (Experimental small-scale) | |
| | **Training Hardware** | Single NVIDIA H100 GPU | |
| | **Training Duration** | ~110 Hours | |
| | **Training Precision** | `bfloat16` (`bf16`) | |
| | **Max Sequence Length** | 1,024 Tokens | |
|
|
| --- |
|
|
| ## 📚 Training Corpus Breakdown (Total: ~11B Tokens) |
|
|
| The dataset was deliberately partitioned into three distinct components to isolate the impact of the exotic data on general alignment: |
|
|
| 1. **Public Mixed Educational/General Corpus** (10B tokens) |
| * **Source:** `HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled` |
| * **Purpose:** Serves as the foundational, high-quality mainstream baseline. |
| 2. **Mathematics Specialized Corpus** (~1B tokens) |
| * **Source:** `HuggingFaceTB/finemath` |
| * **Purpose:** Injects structured, logical reasoning capabilities. |
| 3. **High-Quality Exotic Text Corpus** (0.3B tokens) |
| * **Processing:** Strictly curated and distilled from 20GB of raw data down to 1GB of high-density content. |
| * **Content Type:** Well-crafted narratives, storytelling, and complex roleplay scenarios. |
|
|
| --- |
|
|
| ## 🎯 Research Objectives & Preliminary Findings |
|
|
| ### Primary Research Question |
| *Does the inclusion of high-quality "exotic" or mature story-based textual data fundamentally derail or negatively impact the alignment of small-scale language models?* |
|
|
| ### Preliminary Insights (Measured at 3B token checkpoint) |
| 1. **No Alignment Degradation:** We observed no statistically significant negative impact on core model alignment resulting from the inclusion of exotic data. |
| 2. **Architectural Compatibility:** Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though practical deployment still heavily relies on robust content moderation. |
| 3. **Scalability Proof-of-Concept:** Achieving these stable results on a small-scale model provides a valid, resource-efficient proof-of-concept for larger-scale alignment research. |
|
|
| --- |
|
|
| ## 📈 Baseline Evaluation Results |
|
|
| *Note: This model was explicitly developed for alignment research rather than zero-shot benchmark dominance. Zero-shot accuracy was not optimized for, resulting in expected baseline performance metrics.* |
|
|
| | Task | Version | Filter | n-shot | Metric | Value | Stderr | |
| | :--- | :---: | :---: | :---: | :--- | :---: | :---: | |
| | **arc_challenge** | 1 | none | 0 | acc <br> acc_norm | 0.2082 <br> 0.2543 | ± 0.0119 <br> ± 0.0127 | |
| | **arc_easy** | 1 | none | 0 | acc <br> acc_norm | 0.5442 <br> 0.4701 | ± 0.0102 <br> ± 0.0102 | |
| | **boolq** | 2 | none | 0 | acc | 0.6049 | ± 0.0086 | |
| | **hellaswag** | 1 | none | 0 | acc <br> acc_norm | 0.2984 <br> 0.3292 | ± 0.0046 <br> ± 0.0047 | |
| | **piqa** | 1 | none | 0 | acc <br> acc_norm | 0.6496 <br> 0.6311 | ± 0.0111 <br> ± 0.0113 | |
| | **truthfulqa_mc2**| 3 | none | 0 | acc | 0.4118 | ± 0.0148 | |
| | **winogrande** | 1 | none | 0 | acc | 0.5233 | ± 0.0140 | |
| |
| --- |
| |
| ## 📋 Nomenclature |
| |
| The name **"Wildalign"** is a portmanteau of **"Wild/Exotic"** (referencing our deliberate use of non-mainstream data components) and **"Alignment"** (the core research focus). The moniker reflects our primary study goal: navigating the wild to determine if mixing mainstream rules with unusual data sources disrupts model safety and coherence. |
| |
| --- |
| |
| ## 🤝 Support & Collaboration |
| |
| If you find this research valuable and wish to help fuel further development, your support is deeply appreciated! |
| |
| <a href="https://ko-fi.com/wildmann" target="_blank"><img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support me on Ko-fi" height="36"></a> |
| |
| *All donations received will be strictly allocated to the computational costs of large-scale pre-training—driving research, model optimization, and scaling efforts for more generalized AI systems.* |
| |
| **💻 Call for Compute Collaborators** |
| We warmly welcome anyone with dormant GPU/compute budgets to join our community. Your computational resources will play a vital role in advancing these foundation model projects, allowing us to accelerate the pace of open-source AI development together. Reach out if you're interested in collaborating! |
