| |
|
| | --- |
| | license: mit |
| | datasets: |
| | - ethanker/nanomind_1m |
| | language: |
| | - en |
| | library_name: transformers |
| | tags: |
| | - gpt |
| | - decoder-only |
| | - llama |
| | - tiny |
| | pipeline_tag: text-generation |
| | --- |
| | |
| | # nanomind-step-002000 (early experiment checkpoint) |
| |
|
| | This is an early checkpoint (step 2,000) from a small decoder-only GPT-style experiment. It is shared primarily for transparency and to help others reproduce or build upon the setup. This checkpoint is not production-ready. |
| |
|
| | ## What this is |
| | - Model: small LLaMA-style decoder-only (RMSNorm, SwiGLU, RoPE, MQA/GQA-compatible) |
| | - Checkpoint: step_002000 from run1 |
| | - Data: curated 1M-doc mix (English), hosted at the public dataset repo: [ethanker/nanomind_1m](https://huggingface.co/datasets/ethanker/nanomind_1m) |
| | - Intended use: research/experimentation only |
| | |
| | ## How it was trained (run1) |
| | - Script: `train_run1.py` (included here) with the exact launch command in `RUN_COMMAND.txt`. |
| | - Key settings used for run1: |
| | - seq_len 2048, hidden_size 512, n_layers 16, n_heads 8, n_kv_heads 1 |
| | - global_batch_size 64, micro_batch_size 1, AdamW lr 1e-3, warmup 2000 |
| | - bf16 autocast, gradient clipping 1.0 |
| | |
| | ## Quick eval snapshot (for context only) |
| | - In-domain ppl (small slice): ~1.06 (expected to be low given early-stage in-domain evaluation) |
| | - Generations: fluent but sometimes regurgitative; this is a very early checkpoint |
| | |
| | ## Optimizations implemented for subsequent runs |
| | These were implemented in the training/data pipeline for future iterations (beyond this checkpoint): |
| | - Near-duplicate filtering (MinHash+LSH) and stronger boilerplate heuristics |
| | - Optional gradient checkpointing and torch.compile for better memory/throughput |
| | - Periodic quick perplexity checks on a small token budget |
| | |
| | References: |
| | - Chinchilla compute-optimal scaling: https://arxiv.org/abs/2203.15556 |
| | - Deduplication improves LMs: https://arxiv.org/abs/2107.06499 |
| | - Dedup mitigates privacy risks: https://arxiv.org/abs/2202.06539 |
| | - FlashAttention-3: https://arxiv.org/abs/2407.08608 |
| | - YaRN long-context: https://arxiv.org/abs/2309.00071 |
| | |
| | ## Load and sample |
| | ```python |
| | from transformers import AutoTokenizer, LlamaForCausalLM |
| | import torch |
| | m = 'ethanker/nanomind-step-002000' |
| | tok = AutoTokenizer.from_pretrained(m, use_fast=True) |
| | model = LlamaForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32) |
| | model.eval().to('cuda' if torch.cuda.is_available() else 'cpu') |
| | |
| | prompt = "Once upon a time," |
| | inputs = tok(prompt, return_tensors='pt').to(model.device) |
| | out = model.generate(**inputs, do_sample=True, top_p=0.9, temperature=0.8, max_new_tokens=128) |
| | print(tok.decode(out[0], skip_special_tokens=True)) |
| | ``` |
| | |
| | ## Files |
| | - `model.safetensors`, tokenizer/config files |
| | - `train_run1.py` (training code snapshot) |
| | - `RUN_COMMAND.txt` (exact command used) |
| | |
| | ## Notes |
| | - Early and exploratory; expect limited generalization and occasional regurgitation. |
| | - Please prefer the referenced dataset repo and scripts for reproducibility and your own experiments. |
| | |
