slm-125m-base

asinha08/slm-125m-base is a 125M parameter, Llama-compatible, decoder-only causal language model trained from scratch on a legal-first English corpus. It is a base language model, not an instruction-tuned or chat-aligned assistant.

Model Details

Field	Value
Repository	asinha08/slm-125m-base
Project / Modal app	slm125mLIVE
Model type	Decoder-only causal LM, Llama-compatible
Parameters	125,848,320
Context length	1,024 tokens
Vocabulary size	16,384
Tokenizer	Byte-level BPE trained from the final corpus
Hidden size	768
Layers	12
Attention heads	12
KV heads	12
Intermediate size	3,072
Activation	silu / SwiGLU
Positional encoding	RoPE, theta 10,000
Norm	RMSNorm, eps 1e-5
Embeddings	Tied input/output embeddings
Attention bias	false
Saved format	safetensors
Weight dtype	bfloat16

Special tokens:

Token role	Token
BOS	<|bos|>
EOS	<|eos|>
PAD	<|pad|>
UNK	<|unk|>
Extra	<|user|>, <|assistant|>, <|system|>

The extra chat-style tokens are present in the tokenizer, but the model was not instruction tuned.

Intended Use

This model is intended for:

• Research on small language model pretraining.
• Legal-domain language modeling experiments.
• Continued pretraining, supervised fine-tuning, or evaluation pipelines.
• Text continuation/completion tests.

This model is not intended for:

• Legal advice or compliance decisions.
• Production legal automation without task-specific evaluation and guardrails.
• Chatbot use without instruction tuning and safety tuning.
• Factual question answering where accuracy is required.

Training Data

The corpus was built from three public Hugging Face datasets with a legal-first mix:

Source	HF dataset	Kept docs	Proxy corpus tokens	Tokenized train tokens	Tokenized val tokens	Tokenized mix
Case law	HFforLegal/case-law	206,684	806,244,775	714,858,496	7,223,296	35.06%
SEC filings	PleIAs/SEC	45,035	1,091,148,638	860,025,856	8,688,640	42.18%
Web/education	HuggingFaceFW/fineweb-edu / sample-10BT	418,405	499,936,147	464,188,416	4,689,920	22.76%
Total		670,124	2,397,329,560	2,039,072,768	20,601,856	100.00%

Data processing included:

• Basic line/document cleaning.
• OCR/noise filtering for case-law data.
• Exact duplicate removal.
• MinHash near-duplicate removal.
• Decontamination against held-out evaluation sources: coastalcph/lex_glue and casehold/casehold.
• Byte-level BPE tokenizer training on the cleaned corpus.
• Packed fixed-length token windows with sequence length 1,024.
• Validation routing of every 100th packed window, giving an approximately 99/1 train/validation split.

Deduplication and decontamination summary:

Source	Exact duplicates removed	Near duplicates removed	Contaminated docs removed
Case law	0	1,606	24,002
SEC filings	1,989	0	175
FineWeb-Edu	62	0	0

Training Procedure

Training ran on Modal using 8x H100 GPUs.

Field	Value
GPUs	8 x H100
Precision	bfloat16 autocast
Optimizer	AdamW
Learning rate	6e-4
Minimum LR	6e-5
Warmup tokens	200,000,000
Weight decay	0.1
Betas	(0.9, 0.95)
Gradient clipping	1.0
Micro batch size	64 windows per GPU
World size	8
Gradient accumulation	1
Global batch tokens	524,288
Steps per full epoch	3,889
Train windows available	1,991,282
Train windows consumed per epoch	1,991,168
Tokens consumed per epoch	2,038,956,032
Checkpoint interval	Every 500 steps
Evaluation interval	Every 1,000 steps and at epoch end
Seed	1337

The model has completed 4 full training epochs over the packed train split.

Evaluation

Perplexity is computed as exp(validation_loss).

The reported validation metrics use 4,096 validation windows, or about 4.19M validation tokens, sampled from the internal packed validation split. The full validation split contains 20,119 windows, or 20,601,856 tokens.

Epoch	Status	Epoch tokens	Total tokens seen	Steps	Validation loss	Validation perplexity
1	Completed	2.039B	2.039B	3,889	3.3730	29.17
2	Completed	2.039B	4.078B	3,889	2.9986	20.06
3	Completed	2.039B	6.117B	3,889	2.8545	17.37
4	Completed	2.039B	8.156B	3,889	2.7765	16.06

The latest completed evaluation is:

Metric	Value
Validation loss	2.7765
Validation perplexity	16.06
Total tokens seen	8,155,824,128

No downstream task benchmark results are reported yet.

Approximate Training Cost

Known pretraining cost by epoch:

Epoch	H100	CPU	Memory	Total
1	~$7.18	Not recorded	Not recorded	~$7.18+
2	~$9.60	Not recorded	Not recorded	~$9.60+
3	$10.18	$0.25	$0.33	$10.76
4	$11.02	$0.27	$0.36	$11.65

Approximate known pretraining total: ~$39.19+. This excludes any unrecorded dataset preparation, storage, network, or earlier CPU-only costs.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "asinha08/slm-125m-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "<|bos|>The plaintiff alleged that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=False))

Because this is a base model, prompts should be treated as text-completion prefixes. The tokenizer contains <|user|>, <|assistant|>, and <|system|> tokens, but the model has not been trained to follow chat instructions.

Limitations and Risks

• The model is small and undertrained relative to modern production LLMs.
• The model can hallucinate, fabricate citations, and produce legally incorrect text.
• The model is not safe for legal advice, compliance review, contract review, court filings, or any high-stakes use without extensive validation.
• The model was not instruction tuned, RLHF tuned, preference tuned, or safety aligned.
• The training corpus is mostly legal/financial English text plus a smaller web-education slice; performance outside that distribution may be poor.
• The model may reproduce biases, sensitive patterns, or memorized fragments from public data.
• Evaluation currently reports only internal validation perplexity, not downstream legal-task accuracy.
• Source datasets have their own licenses and use constraints; users should verify dataset licenses before commercial or redistributed use.

Reproducibility Notes

Main artifacts were stored on the Modal volume slm125mLIVE:

Artifact	Path
Cleaned data	/data/clean
Deduplicated corpus	/data/corpus
Tokenizer	/data/tokenizer
Tokenized train/val windows	/data/tokens
Checkpoints	/data/checkpoints
Final base checkpoint	/data/checkpoints/base

Latest training summary:

Field	Value
Status	completed
Steps	3,889 / 3,889
Epoch base tokens	6,116,868,096
Epoch tokens	2,038,956,032
Total tokens seen	8,155,824,128
World size	8
Gradient accumulation	1
Final checkpoint	/data/checkpoints/ckpt.pt
Final model directory	/data/checkpoints/base

Citation

No formal citation is available. If you use this model, cite the Hugging Face repository:

asinha08/slm-125m-base