| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - text-generation |
| - causal-lm |
| - chatml |
| - from-scratch |
| - hydrion |
| - opengcm |
| pipeline_tag: text-generation |
| --- |
| |
|
|
|
|
|  |
|
|
| Hydrion is a 114M-parameter causal language model, pretrained from scratch and fine-tuned for chat, built on a single RTX 3060 plus a handful of rented A100 hours. |
|
|
| This repo (`OpenGCM/Hydrion-Base`) is the base model, non-chat ready version. The instruction model (chat formatting) is available at [`OpenGCM/Hydrion-SFT`](https://huggingface.co/OPENGCM/Hydrion-SFT). |
| ## Model Details |
|
|
| - **Architecture:** Llama-style decoder-only transformer (RMSNorm, rotary position embeddings, SwiGLU MLP, grouped-query attention) |
| - **Parameters:** 114.1M |
| - **Layers:** 12 |
| - **Hidden size:** 768 |
| - **Attention heads:** 12 (4 KV heads, GQA) |
| - **Context length:** 1024 tokens |
| - **Tokenizer:** [`EleutherAI/gpt-neox-20b`](https://huggingface.co/EleutherAI/gpt-neox-20b) |
| - **License:** Apache 2.0 |
|
|
| ## Training |
|
|
| Hydrion was trained in two pretraining stages. |
|
|
| 1. **Initial pretraining** — ~2B tokens on a FineWeb-Edu / Wikipedia mix, trained on a single RTX 3060 (12GB). |
| 2. **Continued pretraining** — an additional ~0.5B tokens on a more diverse mix (FineWeb-Edu, Wikipedia, TinyStories, a code subset, and Dolly), run on a rented A100 to broaden register and topic coverage beyond pure web/encyclopedic text. |
|
|
| Total pretraining exposure: roughly **2.5 billion tokens**. |
|
|
| ## Benchmarks |
|
|
| Evaluated with [`lm-evaluation-harness`](https://github.com/EleutherAI/lm-evaluation-harness) on the base (pre-SFT) checkpoint: |
|
|
| | Benchmark | Metric | Score | |
| |---|---|---| |
| | BLiMP | acc | 80.08% | |
| | ARC-Easy | acc | 47.26% | |
| | ARC-Easy | acc_norm | 43.39% | |
| | WikiText-2 | byte_perplexity | 2.04 | |
| | WikiText-2 | bits_per_byte | 1.03 | |
| | WikiText-2 | word_perplexity | 45.02 | |
| |
| Grammatical judgment (BLiMP) is comparable to models trained on far larger token budgets; factual/reasoning performance (ARC-Easy) is meaningfully weaker, consistent with the relatively small pretraining corpus. |
| |
| ## Usage |
| |
| ```python |
| import torch |
| from transformers import AutoTokenizer, LlamaForCausalLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("OPENGCM/Hydrion-Base") |
| model = LlamaForCausalLM.from_pretrained("OPENGCM/Hydrion-Base", torch_dtype=torch.bfloat16).cuda() |
| model.eval() |
|
|
| prompt = "What is the capital of" |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| |
| with torch.no_grad(): |
| output = model.generate( |
| **inputs, |
| max_new_tokens=150, |
| do_sample=True, |
| temperature=0.7, |
| top_p=0.9, |
| repetition_penalty=1.3, |
| no_repeat_ngram_size=3, |
| eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"), |
| ) |
| |
| response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True) |
| print(response) |
| ``` |
| |
| ## Limitations |
| |
| Hydrion is a small model trained on a modest token budget (~2.5B tokens, versus the trillions used by comparable production small models). It should **not** be relied on for factual accuracy. It reliably produces fluent, grammatically well-formed English and responds in a conversational chat format, but frequently states incorrect facts, fabricates names/dates/attributions, and performs poorly at arithmetic and multi-step reasoning. Treat outputs as unreliable by default — this model is best understood as a demonstration of a working from-scratch training pipeline rather than a usable knowledge source or assistant. |
| |
