syaffers/Atom-350M-NVFP4

Model Overview

Description:

Atom-350M-NVFP4 is the quantized version of TinyModels/Atom-350M, a compact open-source chat assistant fine-tuned from SmolLM2-360M-Instruct. Quantization was performed with llm-compressor using one-shot post-training quantization (PTQ). For more information on the base model, please check here.

This model is ready for commercial/non-commercial use.

Third-Party Community Consideration

This model is a quantized derivative of TinyModels/Atom-350M. See the Atom-350M Model Card for details on the base model's development and intended use.

License/Terms of Use:

Apache 2.0

Deployment Geography:

Global

Use Case:

Developers looking for an extremely compact, pre-quantized model for on-device inference, edge deployment, chatbots, and other resource-constrained AI applications.

Release Date:

Hugging Face 06/03/2026 via https://huggingface.co/syaffers/Atom-350M-NVFP4

Model Architecture:

Architecture Type: Transformers
Network Architecture: Decoder-only causal language model
Number of Model Parameters: ~360M

Input:

Input Type(s): Text
Input Format(s): String
Input Parameters: 1D (One Dimensional)
Other Properties Related to Input: Supports multi-turn conversations formatted via the model's chat template. Maximum context length of 8192 tokens.

Output:

Output Type(s): Text
Output Format: String
Output Parameters: 1D (One Dimensional)

Software Integration:

Supported Runtime Engine(s):

• Hugging Face Transformers
  
• vLLM (tested on v0.21.0)
  

Preferred Operating System(s):

• Linux
  

Model Version(s):

The model is quantized to NVFP4 using llmcompressor.

---

Calibration Dataset:

Link: HuggingFaceH4/ultrachat_200k
Split: train_sft
Samples used: 1024
Data Collection Method: Automated
Labeling Method: Automated
Properties: UltraChat 200k is a large-scale, high-quality dataset of multi-turn conversational exchanges. Samples were formatted using the model's chat template and tokenized with truncation at 8192 tokens.

---

Inference:

Acceleration Engine: Hugging Face Transformers, vLLM
Test Hardware: NVIDIA GPU (CUDA)

---

Post Training Quantization

This model was obtained by quantizing the weights and activations of Atom-350M to the NVFP4 data type using one-shot PTQ via llm-compressor. All Linear layers within transformer blocks are quantized; the lm_head is excluded and kept at full precision to preserve output distribution.

Usage

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "syaffers/Atom-350M-NVFP4",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("syaffers/Atom-350M-NVFP4")

messages = [{"role": "user", "content": "Explain how a bicycle stays upright in simple terms."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=150, temperature=0.7)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

vLLM — Offline Inference

Tested on vLLM v0.21.0 or later.

from vllm import LLM, SamplingParams

llm = LLM(model="syaffers/Atom-350M-NVFP4", quantization="modelopt")

sampling_params = SamplingParams(temperature=0.7, max_tokens=150)

messages = [{"role": "user", "content": "Explain how a bicycle stays upright in simple terms."}]
outputs = llm.chat(messages, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

vLLM — Online Inference (OpenAI-Compatible Server)

Start the server:

vllm serve syaffers/Atom-350M-NVFP4 --quantization modelopt --port 8000

Then query it with the OpenAI client or any HTTP client:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="placeholder")

response = client.chat.completions.create(
    model="syaffers/Atom-350M-NVFP4",
    messages=[{"role": "user", "content": "Explain how a bicycle stays upright in simple terms."}],
    max_tokens=150,
    temperature=0.7,
)
print(response.choices[0].message.content)

Model Limitations:

The base model was fine-tuned on data that may contain biases present in the source corpora. The NVFP4 quantization introduces a small degree of approximation error that may slightly affect output quality on edge cases. This model is not intended for safety-critical applications without additional guardrails.