Text Generation
Transformers
Safetensors
English
Italian
quark
causal-lm
small-language-model
gqa
rope
swiglu
bash
code
custom_code
Instructions to use ThingAI/Quark-72M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ThingAI/Quark-72M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ThingAI/Quark-72M", trust_remote_code=True)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("ThingAI/Quark-72M", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ThingAI/Quark-72M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ThingAI/Quark-72M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ThingAI/Quark-72M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/ThingAI/Quark-72M
- SGLang
How to use ThingAI/Quark-72M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ThingAI/Quark-72M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ThingAI/Quark-72M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ThingAI/Quark-72M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ThingAI/Quark-72M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use ThingAI/Quark-72M with Docker Model Runner:
docker model run hf.co/ThingAI/Quark-72M
Update README.md
Browse files
README.md
CHANGED
|
@@ -51,7 +51,7 @@ This is an **instruction-tuned** checkpoint, fine-tuned via SFT on top of a base
|
|
| 51 |
| **Languages** | English, Italian |
|
| 52 |
| **License** | MIT |
|
| 53 |
| **Finetuned from** | Quark-72M base (pre-trained on math/code/reasoning mix) |
|
| 54 |
-
| **Repository** | [ThingAI/Quark-72M
|
| 55 |
|
| 56 |
Quark-72M-Instruct is part of a broader effort at ThingAI to build small, self-hostable language models that can be trained, fine-tuned, and served entirely on personal infrastructure — without dependency on third-party APIs. It trades raw capability for transparency, inspectability, and low resource requirements.
|
| 57 |
|
|
@@ -59,7 +59,7 @@ Quark-72M-Instruct is part of a broader effort at ThingAI to build small, self-h
|
|
| 59 |
|
| 60 |
## Architecture
|
| 61 |
|
| 62 |
-
Quark-72M
|
| 63 |
|
| 64 |
| Component | Detail |
|
| 65 |
|---|---|
|
|
@@ -86,7 +86,7 @@ Quark-72M-Instruct uses a standard decoder-only Transformer stack with several e
|
|
| 86 |
|
| 87 |
## Intended Use
|
| 88 |
|
| 89 |
-
Quark-72M
|
| 90 |
|
| 91 |
- A **research and educational artifact** for studying small-scale LLM architecture, training, and inference.
|
| 92 |
- A **lightweight conversational base** for hobbyist or self-hosted projects where running a multi-billion-parameter model is impractical.
|
|
@@ -162,7 +162,7 @@ At ~72M parameters, this model runs comfortably on CPU for single-turn interacti
|
|
| 162 |
|
| 163 |
## Prompt Format
|
| 164 |
|
| 165 |
-
Quark-72M
|
| 166 |
|
| 167 |
```
|
| 168 |
<|im_start|>user
|
|
@@ -285,7 +285,7 @@ Because this model relies on custom modeling code rather than a built-in `transf
|
|
| 285 |
|
| 286 |
## Citation
|
| 287 |
|
| 288 |
-
If you use Quark-72M
|
| 289 |
|
| 290 |
```bibtex
|
| 291 |
@misc{quark72m,
|
|
|
|
| 51 |
| **Languages** | English, Italian |
|
| 52 |
| **License** | MIT |
|
| 53 |
| **Finetuned from** | Quark-72M base (pre-trained on math/code/reasoning mix) |
|
| 54 |
+
| **Repository** | [ThingAI/Quark-72M](https://huggingface.co/ThingAI/Quark-72M) |
|
| 55 |
|
| 56 |
Quark-72M-Instruct is part of a broader effort at ThingAI to build small, self-hostable language models that can be trained, fine-tuned, and served entirely on personal infrastructure — without dependency on third-party APIs. It trades raw capability for transparency, inspectability, and low resource requirements.
|
| 57 |
|
|
|
|
| 59 |
|
| 60 |
## Architecture
|
| 61 |
|
| 62 |
+
Quark-72M uses a standard decoder-only Transformer stack with several efficiency-oriented design choices common in modern small LLMs:
|
| 63 |
|
| 64 |
| Component | Detail |
|
| 65 |
|---|---|
|
|
|
|
| 86 |
|
| 87 |
## Intended Use
|
| 88 |
|
| 89 |
+
Quark-72M is intended primarily as:
|
| 90 |
|
| 91 |
- A **research and educational artifact** for studying small-scale LLM architecture, training, and inference.
|
| 92 |
- A **lightweight conversational base** for hobbyist or self-hosted projects where running a multi-billion-parameter model is impractical.
|
|
|
|
| 162 |
|
| 163 |
## Prompt Format
|
| 164 |
|
| 165 |
+
Quark-72M was fine-tuned using a ChatML-style template with `<|im_start|>` / `<|im_end|>` role delimiters:
|
| 166 |
|
| 167 |
```
|
| 168 |
<|im_start|>user
|
|
|
|
| 285 |
|
| 286 |
## Citation
|
| 287 |
|
| 288 |
+
If you use Quark-72M in your work, please consider citing the repository:
|
| 289 |
|
| 290 |
```bibtex
|
| 291 |
@misc{quark72m,
|