Add concise experiment model card.

README.md

@@ -0,0 +1,73 @@
+
+---
+license: mit
+datasets:
+- ethanker/nanomind_1m
+language:
+- en
+library_name: transformers
+tags:
+- gpt
+- decoder-only
+- llama
+- tiny
+pipeline_tag: text-generation
+---
+
+# nanomind-step-002000 (early experiment checkpoint)
+
+This is an early checkpoint (step 2,000) from a small decoder-only GPT-style experiment. It is shared primarily for transparency and to help others reproduce or build upon the setup. This checkpoint is not production-ready.
+
+## What this is
+- Model: small LLaMA-style decoder-only (RMSNorm, SwiGLU, RoPE, MQA/GQA-compatible)
+- Checkpoint: step_002000 from run1
+- Data: curated 1M-doc mix (English), hosted at the public dataset repo: [ethanker/nanomind_1m](https://huggingface.co/datasets/ethanker/nanomind_1m)
+- Intended use: research/experimentation only
+
+## How it was trained (run1)
+- Script: `train_run1.py` (included here) with the exact launch command in `RUN_COMMAND.txt`.
+- Key settings used for run1:
+  - seq_len 2048, hidden_size 512, n_layers 16, n_heads 8, n_kv_heads 1
+  - global_batch_size 64, micro_batch_size 1, AdamW lr 1e-3, warmup 2000
+  - bf16 autocast, gradient clipping 1.0
+
+## Quick eval snapshot (for context only)
+- In-domain ppl (small slice): ~1.06 (expected to be low given early-stage in-domain evaluation)
+- Generations: fluent but sometimes regurgitative; this is a very early checkpoint
+
+## Optimizations implemented for subsequent runs
+These were implemented in the training/data pipeline for future iterations (beyond this checkpoint):
+- Near-duplicate filtering (MinHash+LSH) and stronger boilerplate heuristics
+- Optional gradient checkpointing and torch.compile for better memory/throughput
+- Periodic quick perplexity checks on a small token budget
+
+References:
+- Chinchilla compute-optimal scaling: https://arxiv.org/abs/2203.15556
+- Deduplication improves LMs: https://arxiv.org/abs/2107.06499
+- Dedup mitigates privacy risks: https://arxiv.org/abs/2202.06539
+- FlashAttention-3: https://arxiv.org/abs/2407.08608
+- YaRN long-context: https://arxiv.org/abs/2309.00071
+
+## Load and sample
+```python
+from transformers import AutoTokenizer, LlamaForCausalLM
+import torch
+m = 'ethanker/nanomind-step-002000'
+tok = AutoTokenizer.from_pretrained(m, use_fast=True)
+model = LlamaForCausalLM.from_pretrained(m, torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32)
+model.eval().to('cuda' if torch.cuda.is_available() else 'cpu')
+
+prompt = "Once upon a time,"
+inputs = tok(prompt, return_tensors='pt').to(model.device)
+out = model.generate(**inputs, do_sample=True, top_p=0.9, temperature=0.8, max_new_tokens=128)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+## Files
+- `model.safetensors`, tokenizer/config files
+- `train_run1.py` (training code snapshot)
+- `RUN_COMMAND.txt` (exact command used)
+
+## Notes
+- Early and exploratory; expect limited generalization and occasional regurgitation.
+- Please prefer the referenced dataset repo and scripts for reproducibility and your own experiments.