GODELEV
/

Archaea-74M

@@ -8,149 +8,205 @@ pipeline_tag: text-generation
 ---
 # Archaea-74M
-Archaea-74M is a small decoder-only transformer language model trained on BetterDataset-2M.
-The project focuses on efficient pretraining of a compact language model using a curated multi-source dataset and a modern transformer architecture with Grouped Query Attention (GQA).
 ---
-# Model Information
-| Attribute         | Value                             |
-| ----------------- | --------------------------------- |
-| Model Name        | Archaea-74M                       |
-| Parameters        | ~74M                              |
-| Architecture      | Decoder-Only Transformer          |
-| Base Architecture | LLaMA-style                       |
-| Attention         | Grouped Query Attention (GQA)     |
-| Dataset           | BetterDataset-2M                  |
-| Context Length    | 1024                              |
-| Vocabulary        | GPT-2 Tokenizer (~50k vocabulary) |
-| Framework         | PyTorch + Transformers            |
-| License           | Apache-2.0                        |
 ---
-# Architecture
-Archaea-74M uses a decoder-only transformer architecture implemented with `LlamaForCausalLM`.
-## Configuration
-| Parameter         | Value |
-| ----------------- | ----- |
-| Hidden Size       | 512   |
-| Intermediate Size | 1408  |
-| Layers            | 8     |
-| Attention Heads   | 8     |
-| KV Heads          | 2     |
-| Context Length    | 1024  |
-| Activation        | SiLU  |
-| RMSNorm Epsilon   | 1e-5  |
-## Attention System
-The model uses Grouped Query Attention (GQA):
-* Query Heads: 8
-* Key/Value Heads: 2
-* GQA Ratio: 4:1
-This configuration reduces memory requirements while maintaining efficient attention computation.
 ---
-# Dataset
-## BetterDataset-2M
-Dataset:
-https://huggingface.co/datasets/GODELEV/BetterDataset-2M
-Archaea-74M was trained on BetterDataset-2M, a custom dataset assembled from multiple text sources into a unified corpus.
-The dataset contains a mixture of:
-* General text
-* Conversational data
-* Instruction-oriented samples
-* Knowledge-focused content
-* Technical text
-* Code-related content
-The training pipeline automatically identifies the text field and tokenizes samples using the GPT-2 tokenizer before packing them into contiguous 1024-token sequences.
----
-# Training Configuration
-## Optimization
-| Parameter         | Value                    |
-| ----------------- | ------------------------ |
-| Optimizer         | AdamW                    |
-| Scheduler         | OneCycleLR               |
-| Learning Rate     | 6e-4                     |
-| Weight Decay      | 0.1                      |
-| Gradient Clipping | 1.0                      |
-| Precision         | bfloat16 Mixed Precision |
-| Compilation       | torch.compile            |
-## Batch Configuration
-| Parameter                   | Value |
-| --------------------------- | ----- |
-| Sequence Length             | 1024  |
-| Micro Batch Size            | 16    |
-| Gradient Accumulation Steps | 8     |
-| Effective Batch Size        | 128   |
 ---
-# Training Statistics
-| Metric              | Value   |
-| ------------------- | ------- |
-| Total Steps Trained | 18,800  |
-| Planned Steps       | 25,000  |
-| Initial Loss        | 10.9223 |
-| Final Loss          | 2.9488  |
-| Best Loss           | 2.8071  |
-| Final Perplexity    | 19.08   |
-| Best Perplexity     | 16.56   |
-Training was stopped before the planned completion point in order to continue experimentation and improvements in future training runs.
----
-# Training Visualizations
-## Training Loss Curve
-![Training Loss Curve](Archaea74M_Training_Loss_Curve.png)
-The figure above shows the complete training trajectory, including the raw loss values and a smoothed moving-average trend.
----
-## Learning Rate Schedule
-![Learning Rate Schedule](Archaea74M_Learning_Rate_Schedule.png)
-The model was trained using AdamW with a OneCycleLR scheduler featuring an initial warmup phase followed by gradual decay.
 ---
-# Usage
-## Installation
 ```bash
 pip install torch transformers
 ```
-## Loading the Model
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -167,7 +223,7 @@ model = AutoModelForCausalLM.from_pretrained(
 )
 ```
-## Text Generation
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -176,18 +232,11 @@ import torch
 model_id = "GODELEV/Archaea-74M"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    device_map="auto"
-)
 prompt = "The future of artificial intelligence"
-inputs = tokenizer(
-    prompt,
-    return_tensors="pt"
-).to(model.device)
 with torch.no_grad():
     output = model.generate(
@@ -204,9 +253,9 @@ print(tokenizer.decode(output[0], skip_special_tokens=True))
 ---
-# Repository Structure
-```text
 Archaea-74M/
 ├── config.json
 ├── generation_config.json
@@ -221,29 +270,36 @@ Archaea-74M/
 ---
-# Limitations
-Archaea-74M is a pretrained base model and has not undergone instruction tuning, preference optimization, or extensive benchmark evaluation.
-Known limitations include:
-* Hallucinations
-* Limited reasoning capability compared to larger models
-* Inconsistent factual accuracy
-* Sensitivity to prompt wording
-* Limited evaluation coverage
-The model should not be used for medical, legal, financial, or safety-critical decision making.
 ---
-# Citation
 ```bibtex
 @misc{archaea74m,
-  title={Archaea-74M},
-  author={Akshit Kumar},
-  year={2026},
-  publisher={Hugging Face}
 }
 ```

 ---
 # Archaea-74M
+Archaea-74M is a decoder-only causal language model with approximately 74 million parameters, pretrained from scratch on [BetterDataset-2M](https://huggingface.co/datasets/GODELEV/BetterDataset-2M). The model uses a LLaMA-style architecture with Grouped Query Attention (GQA) and was trained using bfloat16 mixed precision with `torch.compile` enabled.
+Training was stopped at 18,800 of a planned 25,000 steps to allow for architecture and data pipeline iteration in future runs.
 ---
+## Model Card
+| Attribute | Value |
+|---|---|
+| Model ID | `GODELEV/Archaea-74M` |
+| Parameters | ~74 million |
+| Architecture | Decoder-only Transformer (LLaMA-style) |
+| Attention | Grouped Query Attention (GQA) |
+| Context Length | 1024 tokens |
+| Tokenizer | GPT-2 (~50,257 vocabulary) |
+| Training Precision | bfloat16 Mixed Precision |
+| License | Apache-2.0 |
+| Framework | PyTorch + HuggingFace Transformers |
 ---
+# Archaea-74M
+Archaea-74M is a decoder-only causal language model with approximately 74 million parameters, pretrained from scratch on [BetterDataset-2M](https://huggingface.co/datasets/GODELEV/BetterDataset-2M). The model uses a LLaMA-style architecture with Grouped Query Attention (GQA) and was trained using bfloat16 mixed precision with `torch.compile` enabled.
+Training was stopped at 18,800 of a planned 25,000 steps to allow for architecture and data pipeline iteration in future runs.
+---
+## Model Card
+| Attribute | Value |
+|---|---|
+| Model ID | `GODELEV/Archaea-74M` |
+| Parameters | ~74 million |
+| Architecture | Decoder-only Transformer (LLaMA-style) |
+| Attention | Grouped Query Attention (GQA) |
+| Context Length | 1024 tokens |
+| Tokenizer | GPT-2 (~50,257 vocabulary) |
+| Training Precision | bfloat16 Mixed Precision |
+| License | Apache-2.0 |
+| Framework | PyTorch + HuggingFace Transformers |
+---
+## Architecture
+Archaea-74M implements `LlamaForCausalLM` with the following configuration:
+### Transformer Configuration
+| Parameter | Value |
+|---|---|
+| Hidden Size | 512 |
+| Intermediate Size | 1408 |
+| Number of Layers | 8 |
+| Attention Heads | 8 |
+| KV Heads (GQA) | 2 |
+| GQA Ratio | 4:1 |
+| Activation Function | SiLU |
+| Normalization | RMSNorm (eps = 1e-5) |
+| Context Length | 1024 tokens |
+### Grouped Query Attention
+The model uses Grouped Query Attention with 8 query heads mapped to 2 key/value heads (a 4:1 ratio). This reduces KV cache memory footprint at inference time relative to standard multi-head attention while preserving representational capacity in the query projection.
 ---
+## Training
+### Dataset
+Archaea-74M was trained on [BetterDataset-2M](https://huggingface.co/datasets/GODELEV/BetterDataset-2M), a multi-source corpus assembled from:
+- General web text
+- Conversational data
+- Instruction-oriented samples
+- Knowledge-focused content
+- Technical and code-related text
+Samples were tokenized using the GPT-2 tokenizer and packed into contiguous 1024-token sequences.
+### Optimization
+| Parameter | Value |
+|---|---|
+| Optimizer | AdamW |
+| Learning Rate Scheduler | OneCycleLR |
+| Peak Learning Rate | 6e-4 |
+| Weight Decay | 0.1 |
+| Gradient Clipping | 1.0 |
+| Sequence Length | 1024 |
+| Micro Batch Size | 16 |
+| Gradient Accumulation Steps | 8 |
+| Effective Batch Size | 128 |
+| Compilation | `torch.compile` |
+### Training Statistics
+| Metric | Value |
+|---|---|
+| Total Steps Trained | 18,800 / 25,000 |
+| Initial Loss | 10.9223 |
+| Final Loss | 2.9488 |
+| Best Loss | 2.8071 |
+| Final Perplexity | 19.08 |
+| Best Perplexity | 16.56 |
+### Training Loss Curve
+<img src="Archaea74M_Training_Loss_Curve.png" width="600"/>
+The curve shows the raw per-step loss alongside a smoothed moving average. The loss decrease is consistent throughout training with no notable instability or divergence events.
+### Learning Rate Schedule
+<img src="Archaea74M_Learning_Rate_Schedule.png" width="600"/>
+OneCycleLR applies a linear warmup phase followed by cosine annealing decay to a minimum learning rate. The warmup phase stabilizes early training before the peak learning rate is reached.
 ---
+## Evaluation
+Archaea-74M was evaluated using [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on an NVIDIA L4 GPU (24 GB VRAM) in float16 precision. Full datasets were used with no sample limits. Evaluation was conducted on 2026-06-01.
+### Per-Task Results
+| Benchmark | Few-Shot | Metric | Score | Stderr |
+|---|---|---|---|---|
+| HellaSwag | 10 | acc\_norm | 27.16% | ±0.44% |
+| PIQA | 0 | acc\_norm | 58.60% | ±1.15% |
+| WinoGrande | 5 | acc | 51.14% | ±1.41% |
+| BoolQ | 0 | acc | 56.30% | ±0.87% |
+| ARC-Easy | 25 | acc\_norm | 40.11% | ±1.01% |
+| ARC-Challenge | 25 | acc\_norm | 23.04% | ±1.23% |
+| OpenBookQA | 0 | acc\_norm | 26.00% | ±1.96% |
+| CommonsenseQA | 7 | acc | 18.84% | ±1.12% |
+| LAMBADA (OpenAI) | 0 | acc | 18.05% | ±0.54% |
+| BLiMP | 0 | acc | 74.89% | ±0.14% |
+| MMLU | 5 | acc | 25.07% | ±0.36% |
+### Category Averages
+| Category | Benchmarks Included | Average Score |
+|---|---|---|
+| Commonsense / NLI | HellaSwag, PIQA, WinoGrande, BoolQ, ARC-Easy, ARC-Challenge, OpenBookQA, CommonsenseQA | 37.65% |
+| Language Modelling | LAMBADA (OpenAI) | 18.05% |
+| Linguistic | BLiMP | 74.89% |
+| Knowledge | MMLU | 25.07% |
+| **Overall Average** | All above | **38.11%** |
+### Benchmark Score Distribution
+```
+HellaSwag      [=======·····················]  27.16%
+PIQA           [================············]  58.60%
+WinoGrande     [==============··············]  51.14%
+BoolQ          [===============·············]  56.30%
+ARC-Easy       [===========·················]  40.11%
+ARC-Challenge  [======······················]  23.04%
+OpenBookQA     [=======·····················]  26.00%
+CommonsenseQA  [=====·······················]  18.84%
+LAMBADA        [=====·······················]  18.05%
+BLiMP          [======================······]  74.89%
+MMLU           [=======·····················]  25.07%
+               0%        25%       50%       75%      100%
+```
+### Notes on Failed Tasks
+Two tasks could not be evaluated due to infrastructure issues unrelated to the model:
+- `social_iqa` — HuggingFace deprecated the `social_i_qa.py` dataset loading script. Re-evaluation pending.
+- `arithmetic_2digit` — Task name has been renamed in the current version of lm-eval. Re-evaluation pending with the updated task identifier.
+### Evaluation Environment
+```
+Framework   : EleutherAI lm-evaluation-harness
+Hardware    : NVIDIA L4 (24 GB VRAM), Google Colab
+Precision   : float16
+Batch Size  : 8
+Limit       : None (full datasets)
+Runtime     : ~36 minutes
+```
 ---
+## Usage
+### Installation
 ```bash
 pip install torch transformers
 ```
+### Loading the Model
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 )
 ```
+### Text Generation
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 model_id = "GODELEV/Archaea-74M"
 tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
 prompt = "The future of artificial intelligence"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
 with torch.no_grad():
     output = model.generate(
 ---
+## Repository Structure
+```
 Archaea-74M/
 ├── config.json
 ├── generation_config.json
 ---
+## Limitations
+Archaea-74M is a base pretrained model. It has not undergone instruction tuning, RLHF, preference optimization, or any alignment procedure. The following limitations apply:
+- Outputs may contain hallucinated or factually incorrect content
+- Reasoning capability is constrained by the model's size and training duration (18,800 steps of a planned 25,000)
+- Factual accuracy is inconsistent, particularly on knowledge-intensive tasks (MMLU: 25.07%)
+- The model is sensitive to prompt phrasing; small changes in input can produce substantially different outputs
+- Context length is fixed at 1024 tokens; inputs longer than this will be truncated
+- The model should not be used for medical, legal, financial, or safety-critical applications
+---
+## Future Work
+- Instruction fine-tuning on a curated supervised dataset
+- Re-evaluation on `social_iqa` and `arithmetic` once dataset/task compatibility is resolved
+- Context length extension beyond 1024 tokens
+- Evaluation on additional benchmarks relevant to the SLM category
 ---
+## Citation
 ```bibtex
 @misc{archaea74m,
+  title   = {Archaea-74M},
+  author  = {Akshit Kumar},
+  year    = {2026},
+  publisher = {Hugging Face},
+  url     = {https://huggingface.co/GODELEV/Archaea-74M}
 }
 ```