Instructions to use xxparthparekhxx/pluto-124m-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xxparthparekhxx/pluto-124m-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="xxparthparekhxx/pluto-124m-base", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("xxparthparekhxx/pluto-124m-base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xxparthparekhxx/pluto-124m-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xxparthparekhxx/pluto-124m-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/xxparthparekhxx/pluto-124m-base

SGLang

How to use xxparthparekhxx/pluto-124m-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xxparthparekhxx/pluto-124m-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xxparthparekhxx/pluto-124m-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xxparthparekhxx/pluto-124m-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use xxparthparekhxx/pluto-124m-base with Docker Model Runner:
```
docker model run hf.co/xxparthparekhxx/pluto-124m-base
```

Pluto 124M Base

Pluto Base is a 121M-parameter language model. xxparthparekhxx trained it from scratch on 1B tokens of FineWeb-Edu, on one RTX 5060 Laptop GPU (8 GB) in about 10 hours. It continues text and is not tuned for chat; for a chat model, use pluto-124m-chat.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xxparthparekhxx/pluto-124m-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

inputs = tokenizer("The most important thing to know about the ocean is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=60)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True is needed for both the tokenizer and the model. The architecture (gated attention with QK-norm and a logit soft-cap) is not a stock transformers class; its code is modeling_pluto.py in this repo.

Evaluation

Compared with GPT-2 124M

All three models were scored with the same code: lm-evaluation-harness 0.4.13, zero-shot, fp32. GPT-2 is openai-community/gpt2.

Task (metric)	Pluto base	Pluto chat	GPT-2 124M
HellaSwag (acc_norm)	30.2	31.1	31.1
ARC-Easy (acc_norm)	45.2	41.9	39.5
ARC-Challenge (acc_norm)	24.3	26.7	22.7
OpenBookQA (acc_norm)	29.4	27.2	27.2
PIQA (acc_norm)	60.3	59.1	62.5
Winogrande (acc)	52.3	51.7	51.6
LAMBADA (acc)	23.3	16.8	32.6

The standard errors are about ±0.5 points on HellaSwag and LAMBADA, ±1 to 1.5 on ARC, PIQA and Winogrande, and ±2 on OpenBookQA.

Pluto saw 1B training tokens, far fewer than GPT-2, and it is ahead on the science and school-knowledge questions: ARC-Easy, ARC-Challenge and OpenBookQA. That matches its educational pretraining data. It is behind on PIQA, and well behind on LAMBADA, which asks for the last word of passages from novels; FineWeb-Edu contains little fiction. Winogrande is near chance (50%) for all three models.

On the same 4M held-out FineWeb-Edu tokens, the validation loss is 3.203 for Pluto base (perplexity 24.6) and 3.274 for GPT-2 (perplexity 26.4). This text is in-distribution for Pluto, so the comparison favors it.

Training

Data: 1B tokens from FineWeb-Edu sample-10BT, tokenized with GPT-2's BPE.
Steps: 1,907 steps of 524,288 tokens (512 sequences of 1,024), in bf16 with torch.compile.
Optimizer: Muon for the 2-D weight matrices and AdamW for the embeddings and norms. Muon uses the DeepSeek-V4 variant: Newton–Schulz orthogonalization with the update rescaled to RMS 0.18, so both optimizers share one learning rate.
Learning rate: peak 3e-3, weight decay 0.1. 200 warmup steps, then flat, then a linear decay to 10% of the peak over the last 25% of steps (warmup-stable-decay).
Compute: about 10 hours on one RTX 5060 Laptop GPU, at roughly 26.7k tokens/s.

Architecture


Parameters	121.2M (embeddings tied with the output layer)
Layers	12, pre-norm, RMSNorm
Width	768
Attention	grouped-query: 12 query heads, 4 key/value heads, head size 64
	RoPE (θ = 10,000), RMSNorm on queries and keys, sigmoid output gate
Feed-forward	SwiGLU, hidden size 2,048
Output	tanh logit soft-cap at 30
Vocabulary	GPT-2 BPE plus 4 chat tokens (unused here), padded to 50,304
Context	1,024 tokens

Limitations

English only, with a 1,024-token context.
It continues text; it does not follow instructions.
It was trained on educational web pages only, so it knows little about other kinds of text, such as code or dialogue.
Its facts are unreliable.

License

Apache-2.0. The training data, FineWeb-Edu, is ODC-By 1.0.

Downloads last month: -

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for xxparthparekhxx/pluto-124m-base

Finetunes

1 model

xxparthparekhxx
/

pluto-124m-base