124M - 20.879B
Simple LLM base model; 20,878,526,009 lifetime next-token targets; 9,999,963,390 targets in this continuation.
124,389,632 parameters, 8K context, frozen 32,768-token BPE. Eight H100s, cropped padding, cached RoPE, torch.compile, microbatch two, 32 global windows, BF16 computation with FP32 weights. AdamW moments/RNG preserved; constant learning rate 3e-5. Code 60%, EU 5%, previous non-code sources scaled to 35%. Code documents are fresh after exact/whitespace deduplication and consumed at most once; non-code may reuse older data at most twice in this phase. See data-manifest.json for capacities and limitations.
HumanEval/MBPP lexical screening and parent split checks do not prove semantic contamination absent. This is not instruction-tuned; generation may remain incorrect and repetitive. Full optimizer checkpoints stay on Modal.
Training and validation curves. Custom PyTorch code: install requirements.txt and run python generate.py --prompt 'def add(a, b):' --tokens 64.
- Downloads last month
- 5