Instructions to use DeependraVerma/slm-500m-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DeependraVerma/slm-500m-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DeependraVerma/slm-500m-base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DeependraVerma/slm-500m-base")
model = AutoModelForCausalLM.from_pretrained("DeependraVerma/slm-500m-base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DeependraVerma/slm-500m-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DeependraVerma/slm-500m-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-500m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DeependraVerma/slm-500m-base

SGLang

How to use DeependraVerma/slm-500m-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DeependraVerma/slm-500m-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-500m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DeependraVerma/slm-500m-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DeependraVerma/slm-500m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use DeependraVerma/slm-500m-base with Docker Model Runner:
```
docker model run hf.co/DeependraVerma/slm-500m-base
```

slm-500m-base

A 528.5-million-parameter Llama-style language model trained completely from scratch — random initialization, its own tokenizer (shared with this author's earlier 125M build), its own data pipeline, its own pretraining run. This is the scaled-up successor to DeependraVerma/slm-125m-base, built to test whether more parameters and more (better-balanced) legal/financial data would produce a genuinely stronger model, not just a bigger one.

Repo / full build: github.com/DeependraVerma/legal-slm-125M
Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
Fine-tuned Q&A version: DeependraVerma/legal-slm-500m-sft
Predecessor (125M): DeependraVerma/slm-125m-base

This is a base (completion) model, not an assistant. It continues text; it does not follow instructions or answer questions reliably. For a Q&A-capable model, use legal-slm-500m-sft. Never use this model's output as legal, financial, or factual advice — it invents specifics (names, numbers, citations) like any base language model, and this is a genuine, measured limitation, not a hypothetical one (see "Known limitations" below).

Model description


Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder (12→24 layers, 768→1152 hidden vs. the 125M build), full multi-head attention (no GQA/MQA)
Layers / hidden / heads	24 / 1152 / 18 (head dim 64)
MLP	SwiGLU, intermediate size 4,608
Positional encoding	RoPE, θ = 10,000
Normalization	RMSNorm, ε = 1e-5
Vocabulary	16,384, byte-level BPE — the same tokenizer as the 125M build, reused deliberately rather than retrained
Context length	1,024 tokens
Embeddings	tied input/output
Precision	bf16 compute (autocast) · fp32 master weights and saved checkpoint

Training data

Seven sources, expanded in two stages. The original five (~14.7B unique tokens): US case law, SEC 10-K filings, educational web text, bulk SEC 10-K/10-Q/8-K filings, and bulk US case law. A subsequent continuation run added two new government sources plus more SEC filing types:

Source	HF dataset / origin	License
US case law	`HFforLegal/case-law`	CC BY 4.0
SEC 10-K filings	`PleIAs/SEC`	—
Educational web	`HuggingFaceFW/fineweb-edu`	ODC-BY
SEC 10-K/10-Q/8-K/S-1/20-F filings (bulk)	`TeraflopAI/SEC-EDGAR`	Apache-2.0
US case law (bulk)	`common-pile/caselaw_access_project`	Public domain
Code of Federal Regulations (CFR)	`govinfo.gov` bulk XML API	Public domain (US government work)
Federal Register	`govinfo.gov` bulk XML API	Public domain (US government work)

The mix isn't static. A flat/proportional mix of the original five sources was tried first and measurably diluted the original case-law/SEC/fineweb-edu "identity" of the 125M build down to ~13% of training data, causing a real regression on general text (fineweb-edu perplexity got worse than the 125M model, 22.82 vs 20.73). The fix — adopted from precedent in MiniCPM's and SmolLM2's published training recipes — is a staged (Warmup-Stable-Decay) schedule: broad, balanced per-source sampling weights for most of training, then a decay phase near the end that upweights the curated core sources specifically. The schedule was recomputed for all seven sources when CFR and the Federal Register were added, and the training budget was extended from a single-epoch, 28,054-step run to roughly 200 tokens/parameter (201,620 steps) via a checkpoint-resumed continuation.

Evaluation

Full held-out perplexity, per source (not just one blended number, since the seven sources are genuinely different registers of text):

Source	Perplexity (7-source)	Perplexity (original 5-source)
SEC 10-K/10-Q/8-K/S-1/20-F (bulk)	3.38	3.96
SEC-EDGAR (bulk)	3.83	3.84
Federal Register	4.18	(new source)
CFR	4.95	(new source)
US case law	6.62	7.23
US case law (bulk)	7.63	8.23
Educational web	18.07	17.81
Aggregate (all 7 sources)	4.642	5.072

Both new sources (CFR, Federal Register) score better than case law individually — regulatory/administrative prose is comparatively predictable text, and this is a genuinely encouraging sign the expansion is being learned as intended rather than just diluting the mix. Compared against two current small open-weight general-purpose models (OLMo-2-1B, Gemma-3-270M) on external benchmarks (HellaSwag, ARC-Easy, PIQA, MMLU-legal, CaseHOLD, LegalBench), this checkpoint closes most of the gap to the ~2x-larger OLMo-2-1B on legal-domain-specific tasks specifically (CaseHOLD, LegalBench), while trailing on general commonsense reasoning as expected for a narrow-domain model at this scale — and beats same-parameter-count Gemma-3-270M on every benchmark measured. Full tables, methodology, and raw logs: github.com/DeependraVerma/legal-slm-125M.

Intended uses & limitations

Intended use: text continuation / completion in a legal and financial register; research and education on small-model pretraining, data-mix design, and staged training schedules.

Not intended for: question answering, instruction following, or any use where factual accuracy matters. Like the 125M base model (and any base model at this scale), it will fabricate case names, statute citations, dollar figures, and other specifics that sound right but are not grounded in anything. A 528.5M-parameter model can store, at most, roughly 2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — about 132MB of total compressible fact storage, shared across grammar, style, and every fact it knows. This is a hard, measured ceiling, not a training defect. Do not use its output as legal, financial, or factual advice, or present it as such to end users without a human review step. For a model that at least attempts to answer questions, see legal-slm-500m-sft — and note that model has its own documented hallucination limitations too.

Citation

@misc{verma2026slm500mbase,
  author = {Deependra Verma},
  title  = {slm-500m-base: A 528.5M-Parameter Legal/Financial Language Model Trained From Scratch},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/slm-500m-base},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

Downloads last month: 43

Safetensors

Model size

0.5B params

Tensor type

F32

Model tree for DeependraVerma/slm-500m-base

Finetunes

1 model

Paper for DeependraVerma/slm-500m-base

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Paper • 2404.05405 • Published Apr 8, 2024 • 11

Evaluation results

Perplexity (aggregate, 7-source val set) on legal-slm-500M held-out validation set (case law + SEC filings/EDGAR + CFR + Federal Register + fineweb-edu), full 202.0M-token split
self-reported

4.642