pipeline_tag: text-generation
license: apache-2.0
datasets:
- HuggingFaceTB/finemath
- HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled
language:
- en
model-index:
- name: Wildalign-350M-base
results:
- task:
type: arc_challenge
name: ARC Challenge
dataset:
type: arc_challenge
name: ARC Challenge
version: 1
metrics:
- name: acc
type: acc
value: 0.2082
stderr: ± 0.0119
verified: false
- name: acc_norm
type: acc_norm
value: 0.2543
stderr: ± 0.0127
verified: false
- task:
type: arc_easy
name: ARC Easy
dataset:
type: arc_easy
name: ARC Easy
version: 1
metrics:
- name: acc
type: acc
value: 0.5442
stderr: ± 0.0102
verified: false
- name: acc_norm
type: acc_norm
value: 0.4701
stderr: ± 0.0102
verified: false
- task:
type: boolq
name: BoolQ
dataset:
type: boolq
name: BoolQ
version: 2
metrics:
- name: acc
type: acc
value: 0.6049
stderr: ± 0.0086
verified: false
- task:
type: hellaswag
name: HellaSwag
dataset:
type: hellaswag
name: HellaSwag
version: 1
metrics:
- name: acc
type: acc
value: 0.2984
stderr: ± 0.0046
verified: false
- name: acc_norm
type: acc_norm
value: 0.3292
stderr: ± 0.0047
verified: false
- task:
type: piqa
name: PIQA
dataset:
type: piqa
name: PIQA
version: 1
metrics:
- name: acc
type: acc
value: 0.6496
stderr: ± 0.0111
verified: false
- name: acc_norm
type: acc_norm
value: 0.6311
stderr: ± 0.0113
verified: false
- task:
type: truthfulqa_mc2
name: TruthfulQA MC2
dataset:
type: truthfulqa_mc2
name: TruthfulQA MC2
version: 3
metrics:
- name: acc
type: acc
value: 0.4118
stderr: ± 0.0148
verified: false
- task:
type: winogrande
name: Winogrande
dataset:
type: winogrande
name: Winogrande
version: 1
metrics:
- name: acc
type: acc
value: 0.5233
stderr: ± 0.0140
verified: false
🐺 Wildalign-350M-base
Experimental Exotic Autoregressive Language Model
This repository hosts Wildalign‑350M‑base—an experimental, lightweight autoregressive language model designed to investigate the impact of high-quality "exotic" textual data on model alignment at the GPT-2 base parameter scale.
⚠️ Critical Content Warning The model's training corpus intentionally incorporates high-quality "exotic" textual data. Consequently, under specific prompt guidance, the model may generate content that falls under NSFW categories, including but not limited to mature/non-mainstream narratives, roleplay, or thematic scenarios. While the model remains safe for standard use cases, we advise caution and appropriate content filtering for public deployment.
🧠 Core Model Specifications
| Attribute | Details |
|---|---|
| Architecture | Autoregressive Decoder-Only |
| Parameter Scale | ~350 Million (Experimental small-scale) |
| Training Hardware | Single NVIDIA H100 GPU |
| Training Duration | ~110 Hours |
| Training Precision | bfloat16 (bf16) |
| Max Sequence Length | 1,024 Tokens |
📚 Training Corpus Breakdown (Total: ~11B Tokens)
The dataset was deliberately partitioned into three distinct components to isolate the impact of the exotic data on general alignment:
- Public Mixed Educational/General Corpus (10B tokens)
- Source:
HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled - Purpose: Serves as the foundational, high-quality mainstream baseline.
- Source:
- Mathematics Specialized Corpus (~1B tokens)
- Source:
HuggingFaceTB/finemath - Purpose: Injects structured, logical reasoning capabilities.
- Source:
- High-Quality Exotic Text Corpus (0.3B tokens)
- Processing: Strictly curated and distilled from 20GB of raw data down to 1GB of high-density content.
- Content Type: Well-crafted narratives, storytelling, and complex roleplay scenarios.
🎯 Research Objectives & Preliminary Findings
Primary Research Question
Does the inclusion of high-quality "exotic" or mature story-based textual data fundamentally derail or negatively impact the alignment of small-scale language models?
Preliminary Insights (Measured at 3B token checkpoint)
- No Alignment Degradation: We observed no statistically significant negative impact on core model alignment resulting from the inclusion of exotic data.
- Architectural Compatibility: Exotic long-form narrative content is not intrinsically problematic for long-attention architectures, though practical deployment still heavily relies on robust content moderation.
- Scalability Proof-of-Concept: Achieving these stable results on a small-scale model provides a valid, resource-efficient proof-of-concept for larger-scale alignment research.
📈 Baseline Evaluation Results
Note: This model was explicitly developed for alignment research rather than zero-shot benchmark dominance. Zero-shot accuracy was not optimized for, resulting in expected baseline performance metrics.
| Task | Version | Filter | n-shot | Metric | Value | Stderr |
|---|---|---|---|---|---|---|
| arc_challenge | 1 | none | 0 | acc acc_norm |
0.2082 0.2543 |
± 0.0119 ± 0.0127 |
| arc_easy | 1 | none | 0 | acc acc_norm |
0.5442 0.4701 |
± 0.0102 ± 0.0102 |
| boolq | 2 | none | 0 | acc | 0.6049 | ± 0.0086 |
| hellaswag | 1 | none | 0 | acc acc_norm |
0.2984 0.3292 |
± 0.0046 ± 0.0047 |
| piqa | 1 | none | 0 | acc acc_norm |
0.6496 0.6311 |
± 0.0111 ± 0.0113 |
| truthfulqa_mc2 | 3 | none | 0 | acc | 0.4118 | ± 0.0148 |
| winogrande | 1 | none | 0 | acc | 0.5233 | ± 0.0140 |
📋 Nomenclature
The name "Wildalign" is a portmanteau of "Wild/Exotic" (referencing our deliberate use of non-mainstream data components) and "Alignment" (the core research focus). The moniker reflects our primary study goal: navigating the wild to determine if mixing mainstream rules with unusual data sources disrupts model safety and coherence.
🤝 Support & Collaboration
If you find this research valuable and wish to help fuel further development, your support is deeply appreciated!
All donations received will be strictly allocated to the computational costs of large-scale pre-training—driving research, model optimization, and scaling efforts for more generalized AI systems.
💻 Call for Compute Collaborators We warmly welcome anyone with dormant GPU/compute budgets to join our community. Your computational resources will play a vital role in advancing these foundation model projects, allowing us to accelerate the pace of open-source AI development together. Reach out if you're interested in collaborating!