README.md

---
license: apache-2.0
tags:
- base-model
- causal-lm
- qwen3
- transformer
language:
- en
pipeline_tag: text-generation
---

# QVAC Genesis I Pretrained Model

## Key Highlights
- **Pretrained on the Largest Synthetic Educational Dataset**  
  This model has been **pretrained on Tether's QVAC Genesis I**, the largest synthetic dataset released for educational LLM pre-training.

  The model was trained **from scratch** on approximately **40B tokens** of multi-domain educational text, using **BF16 mixed precision** and a **4,096-token context window**. Training was made with a **Qwen3-family 1.7B-parameter decoder-only transformer** architecture.

  Checkpoints are provided in standard Hugging Face format for easy inference, continual pre-training, and fine-tuning.

- **Multi-Domain Educational Coverage**  
  Because the model is trained on QVAC Genesis I, it inherits curriculum-aligned coverage across:  
  - Mathematics  
  - Physics  
  - Biology  
  - Medicine  

- **Superior Benchmark Performance**  
  Leveraging QVAC Genesis I as its training foundation, the model consistently outperforms baselines in:  
  - Reasoning tasks  
  - Knowledge assessments  
  - Subject-specific QA  

- **First Publicly Released Education-Specific Pretrained Model**  
  This is the first open-source pretrained model built directly on a rigorously validated synthetic dataset for education, offering deep and comprehensive STEM coverage.
abilities  

## Intended Uses
- Continual pre-training or fine-tuning for educational applications (STEM-focused tutoring, QA systems, curriculum support)  
- Benchmarking reasoning and subject-specific QA performance  
- Research into synthetic dataset–driven LLM training  

---

## Model Details

### Model Description

- **Developed by:** Qvac by Tether
- **Model type:** Decoder-only Transformer (causal LM)
- **Language(s) (NLP):** Primarily English 
- **License:** Apache-2.0
- **Finetuned from model:** **None (trained from scratch)**
- **Intended stage:** **Base pre-trained model** (no SFT / RLHF alignment)

### Dataset Details

- **Repository:** https://huggingface.co/qvac/genesisI-model
- **Paper / Blog :** https://huggingface.co/blog/qvac/genesis-i

---

## Uses

### Direct Use

- General language modeling: next-token prediction, continuation, summarization, drafting.
- Research baseline for scaling, data ablations, or tokenizer studies.

### Downstream Use (recommended)
- **CPT** Continued Pre-Training on more tokens.
- **SFT** for assistants, domain experts, or task-specific models.
- **Preference optimization / RLHF** for safer, more helpful behavior.
- **Adapters/LoRA** for efficient domain specialization.

### Out-of-Scope Use

- High-stakes decision-making (medical/financial/legal).
- Safety-critical or autonomous control systems.
- Unfiltered end-user chat deployment without alignment / safety layers.
- Any use that violates applicable laws or platform policies.

---

## Bias, Risks, and Limitations

- **Bias & toxicity:** May reflect or amplify biases present in web text.
- **Hallucinations:** Can produce confident but incorrect statements or citations.
- **Security / privacy:** May emit continous random strings.
- **Context limit:** 4,096 tokens; longer inputs require chunking.

### Recommendations

- Disclose limitations to downstream users.
- Research Model : Not to be used in production use cases. 

---

## How to Get Started

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "qvac/genesisI-model"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,   # trained with BF16 mixed precision
    device_map="auto"
)

prompt = "Explain precision vs. recall in one paragraph."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    top_p=0.9,
    temperature=0.7
)
print(tok.decode(out[0], skip_special_tokens=True))
````

*Tip: On consumer GPUs, consider loading in `float16` or using 4/8-bit quantization (e.g., bitsandbytes/AutoGPTQ).*

---

## Training Details

### Training Data

* **Size:** ~**40B tokens**, single epoch.
* **Domains:** Mixed general + STEM/technical sources (expository text, problem sets, references).
* **Format:** Hugging Face Datasets (Arrow).
* **Tokenizer:** **Qwen3** tokenizer.
* **Processing:** Normalization, filtering of extremes, document chunking to fit **4096** context, sequence packing where applicable.
* **Dataset Card:** *Coming Soon*

### Training Procedure

#### Preprocessing

* Unicode normalization, whitespace cleanup, control-char stripping.
* Length filtering; chunking to 4096; optional packing to improve throughput.

#### Training Hyperparameters

* **Optimizer:** AdamW (β₁=0.9, β₂=0.95), **weight decay 0.01**
* **Learning rate:** **2e-4** (linear warmup)
* **Warmup:** **600** steps (~10% of max steps)
* **Precision:** **BF16 mixed precision**
* **Gradient clipping:** **1.0**
* **Seed:** **42**
* **Logging:** Every **50** steps
* **Eval:** Every **500** steps (20 iters)
* **Checkpointing:** Every **1000** steps (sharded; full optimizer/state resume)

#### Speeds, Sizes, Times

* **Per-GPU micro-batch:** 4
* **Grad accumulation:** 8
* **World size:** 480 GPUs
* **Effective global batch:** `4 × 8 × 480 = 15,360` samples/step
* **Step time (indicative):** ~**1.5 s/step** (cluster/I-O dependent)

#### Stability & Performance

* Activation checkpointing.
* Fused kernels where available (fused attention/optimizer).
* **FlashAttention-2** on H100.
* `torch.compile` (safe mode) after warmup stability.
* Dynamic loss scaling to mitigate BF16 overflow.
* Fragmentation mitigations (e.g., `max_split_size_mb=512`, expandable segments, GC threshold ~0.8).

---

## Multi-Node GPU Setup

* **Cluster:** ~**60 nodes**, each **8× NVIDIA H100 80GB** (total **480 GPUs**), ~800 GB RAM/node.
* **Scheduler:** Slurm (priority partition, exclusive allocation, 72-hour limit).
* **Launch:** `srun` + PyTorch DDP (world size 480; ranks bound via Slurm env).
* **Storage:** Sharded checkpoints; periodic saves for robust resume.
* **Networking:** NCCL over InfiniBand with UCX

  * `NCCL_IB_DISABLE=0`, `NCCL_IB_HCA="mlx5*"`, `NCCL_SOCKET_IFNAME=<ib0/enoX>`, `NCCL_BLOCKING_WAIT=1`
  * Watchdog ~**720s** for fail-fast on fabric issues
* **I/O:** Async dataset prefetching; pinned FS threads.
* **Observability:** W&B + structured logs (throughput, TFLOPs/GPU, mem, step time).
* **Reproducibility:** Fixed seeds; exact launch scripts/env logged; effective tokens/step reported.

> Final checkpoint converted to **Hugging Face format** for plug-and-play inference.

---

## Evaluation

### Testing Data, Factors & Metrics

* **Testing data:** Standard academic suites (e.g., EleutherAI LM Evaluation Harness).
* **Factors:** Domain/topic (STEM vs. general), task type (multi-choice vs. open-ended).
* **Metrics:** Accuracy (MCQ), EM/F1 (QA), plus task-native metrics.

**Suggested suite (edit as applicable):**

* General knowledge & reasoning: **MMLU (STEM subsets)**, **ARC-E/ARC-C**, **HellaSwag**, **PIQA**, **Winogrande**
* Math/coding (optional): **GSM8K**, **HumanEval**
* Reading comprehension (optional): **BoolQ**, **RACE**

### Results

* *To be released with an evaluated checkpoint and harness version pin.*
  Include tables with exact versions, seeds, and commit hashes.

#### Summary

* Base LM targets broad generalization at 41B tokens.
* Expect material gains after SFT + preference optimization for target tasks.

---

## Technical Specifications

### Model Architecture and Objective

* **Architecture:** Qwen3-style decoder-only Transformer
* **Parameters:** ~**1.7B**
* **Context length:** **4,096** tokens
* **Positional encoding:** *Rotary / relative (specify)*
* **Attention:** Multi-head scaled dot-product; FlashAttention-2 enabled on H100
* **Activation:** *GELU / SiLU (specify)*
* **Norms:** *RMSNorm / LayerNorm (specify)*
* **Objective:** **Causal LM** (next-token prediction)

### Compute Infrastructure

**Hardware**

* 60 nodes × 8× H100 80GB, ~800 GB RAM/node, InfiniBand fabric.

**Software**

* PyTorch ≥ 2.1 (CUDA 12.x), FlashAttention-2, UCX/NCCL
* Slurm for orchestration; W&B for logging
* (Optional) DeepSpeed/Zero-3 for training; HF conversion post-train

---

## Reproducibility (Launch Sketch)

```bash
# Slurm (illustrative)
srun -N 60 -n 480 --ntasks-per-node=8 --gpus-per-task=1 \
  --cpus-per-task=8 --mem=0 \
  bash -lc '
  export NCCL_IB_DISABLE=0
  export NCCL_IB_HCA="mlx5*"
  export NCCL_SOCKET_IFNAME=ib0
  export NCCL_BLOCKING_WAIT=1
  export TORCH_DISTRIBUTED_DEBUG=DETAIL

  python train.py \
    --model qwen3_1p7b_from_scratch \
    --tokenizer qwen3 \
    --data_path /path/to/arrow \
    --context_length 4096 \
    --optimizer adamw --weight_decay 0.01 \
    --lr 2e-4 --warmup_steps 600 \
    --precision bf16-mixed \
    --micro_batch_size 4 \
    --grad_accum_steps 8 \
    --eval_every 500 --log_every 50 \
    --ckpt_every 1000 \
    --activation_checkpointing \
    --flash_attn 2 \
    --compile safe \
    --seed 42
'
```

---

## Conversion & Inference

* Checkpoints are **HF-compatible**: load with `AutoModelForCausalLM`.
* For memory-limited environments, prefer half-precision or 4/8-bit loading.
* Distribute as `safetensors` for integrity.

---


## Changelog

* **v0.1 (2025-11-17):** Initial public release — 40B-token 1-epoch pretrain; HF conversion.