File size: 7,952 Bytes
bb38e98 eca202a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 | ---
pipeline_tag: text-generation
license: apache-2.0
datasets:
- HuggingFaceTB/finemath
- HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
language:
- en
model-index:
- name: Wildalign-350M-base
results:
- task:
type: arc_challenge
name: ARC Challenge
dataset:
type: arc_challenge
name: ARC Challenge
version: 1
metrics:
- name: acc
type: acc
value: 0.2082
stderr: ± 0.0119
verified: false
- name: acc_norm
type: acc_norm
value: 0.2543
stderr: ± 0.0127
verified: false
- task:
type: arc_easy
name: ARC Easy
dataset:
type: arc_easy
name: ARC Easy
version: 1
metrics:
- name: acc
type: acc
value: 0.5442
stderr: ± 0.0102
verified: false
- name: acc_norm
type: acc_norm
value: 0.4701
stderr: ± 0.0102
verified: false
- task:
type: boolq
name: BoolQ
dataset:
type: boolq
name: BoolQ
version: 2
metrics:
- name: acc
type: acc
value: 0.6049
stderr: ± 0.0086
verified: false
- task:
type: hellaswag
name: HellaSwag
dataset:
type: hellaswag
name: HellaSwag
version: 1
metrics:
- name: acc
type: acc
value: 0.2984
stderr: ± 0.0046
verified: false
- name: acc_norm
type: acc_norm
value: 0.3292
stderr: ± 0.0047
verified: false
- task:
type: piqa
name: PIQA
dataset:
type: piqa
name: PIQA
version: 1
metrics:
- name: acc
type: acc
value: 0.6496
stderr: ± 0.0111
verified: false
- name: acc_norm
type: acc_norm
value: 0.6311
stderr: ± 0.0113
verified: false
- task:
type: truthfulqa_mc2
name: TruthfulQA MC2
dataset:
type: truthfulqa_mc2
name: TruthfulQA MC2
version: 3
metrics:
- name: acc
type: acc
value: 0.4118
stderr: ± 0.0148
verified: false
- task:
type: winogrande
name: Winogrande
dataset:
type: winogrande
name: Winogrande
version: 1
metrics:
- name: acc
type: acc
value: 0.5233
stderr: ± 0.0140
verified: false
---
# 🐺 Wildalign-350M-base
### Experimental Exotic Autoregressive Language Model
This repository hosts **Wildalign‑350M‑base**—an experimental, lightweight autoregressive language model designed to investigate the impact of high-quality "exotic" textual data on model alignment at the GPT-2 base parameter scale.
> ⚠️ **Critical Content Warning**
> The model's training corpus intentionally incorporates high-quality "exotic" textual data. Consequently, under specific prompt guidance, the model may generate content that falls under NSFW categories, including but not limited to mature/non-mainstream narratives, roleplay, or thematic scenarios. While the model remains safe for standard use cases, we advise caution and appropriate content filtering for public deployment.
---
## 🧠 Core Model Specifications
| Attribute | Details |
| :--- | :--- |
| **Architecture** | Autoregressive Decoder-Only |
| **Parameter Scale** | ~350 Million (Experimental small-scale) |
| **Training Hardware** | Single NVIDIA H100 GPU |
| **Training Duration** | ~110 Hours |
| **Training Precision** | `bfloat16` (`bf16`) |
| **Max Sequence Length** | 1,024 Tokens |
---
## 📚 Training Corpus Breakdown (Total: ~11B Tokens)
The dataset was deliberately partitioned into three distinct components to isolate the impact of the exotic data on general alignment:
1. **Public Mixed Educational/General Corpus** (10B tokens)
* **Source:** `HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled`
* **Purpose:** Serves as the foundational, high-quality mainstream baseline.
2. **Mathematics Specialized Corpus** (~1B tokens)
* **Source:** `HuggingFaceTB/finemath`
* **Purpose:** Injects structured, logical reasoning capabilities.
3. **High-Quality Exotic Text Corpus** (0.3B tokens)
* **Processing:** Strictly curated and distilled from 20GB of raw data down to 1GB of high-density content.
* **Content Type:** Well-crafted narratives, storytelling, and complex roleplay scenarios.
---
## 🎯 Research Objectives & Preliminary Findings
### Primary Research Question
*Does the inclusion of high-quality "exotic" or mature story-based textual data fundamentally derail or negatively impact the alignment of small-scale language models?*
### Preliminary Insights (Measured at 3B token checkpoint)
1. **No Alignment Degradation:** We observed no statistically significant negative impact on core model alignment resulting from the inclusion of exotic data.
2. **Architectural Compatibility:** Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though practical deployment still heavily relies on robust content moderation.
3. **Scalability Proof-of-Concept:** Achieving these stable results on a small-scale model provides a valid, resource-efficient proof-of-concept for larger-scale alignment research.
---
## 📈 Baseline Evaluation Results
*Note: This model was explicitly developed for alignment research rather than zero-shot benchmark dominance. Zero-shot accuracy was not optimized for, resulting in expected baseline performance metrics.*
| Task | Version | Filter | n-shot | Metric | Value | Stderr |
| :--- | :---: | :---: | :---: | :--- | :---: | :---: |
| **arc_challenge** | 1 | none | 0 | acc <br> acc_norm | 0.2082 <br> 0.2543 | ± 0.0119 <br> ± 0.0127 |
| **arc_easy** | 1 | none | 0 | acc <br> acc_norm | 0.5442 <br> 0.4701 | ± 0.0102 <br> ± 0.0102 |
| **boolq** | 2 | none | 0 | acc | 0.6049 | ± 0.0086 |
| **hellaswag** | 1 | none | 0 | acc <br> acc_norm | 0.2984 <br> 0.3292 | ± 0.0046 <br> ± 0.0047 |
| **piqa** | 1 | none | 0 | acc <br> acc_norm | 0.6496 <br> 0.6311 | ± 0.0111 <br> ± 0.0113 |
| **truthfulqa_mc2**| 3 | none | 0 | acc | 0.4118 | ± 0.0148 |
| **winogrande** | 1 | none | 0 | acc | 0.5233 | ± 0.0140 |
---
## 📋 Nomenclature
The name **"Wildalign"** is a portmanteau of **"Wild/Exotic"** (referencing our deliberate use of non-mainstream data components) and **"Alignment"** (the core research focus). The moniker reflects our primary study goal: navigating the wild to determine if mixing mainstream rules with unusual data sources disrupts model safety and coherence.
---
## 🤝 Support & Collaboration
If you find this research valuable and wish to help fuel further development, your support is deeply appreciated!
<a href="https://ko-fi.com/wildmann" target="_blank"><img src="https://ko-fi.com/img/githubbutton_sm.svg" alt="Support me on Ko-fi" height="36"></a>
*All donations received will be strictly allocated to the computational costs of large-scale pre-training—driving research, model optimization, and scaling efforts for more generalized AI systems.*
**💻 Call for Compute Collaborators**
We warmly welcome anyone with dormant GPU/compute budgets to join our community. Your computational resources will play a vital role in advancing these foundation model projects, allowing us to accelerate the pace of open-source AI development together. Reach out if you're interested in collaborating! |