Instructions to use roskosmos19/Orca-4B-flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use roskosmos19/Orca-4B-flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="roskosmos19/Orca-4B-flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("roskosmos19/Orca-4B-flash")
model = AutoModelForMultimodalLM.from_pretrained("roskosmos19/Orca-4B-flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use roskosmos19/Orca-4B-flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "roskosmos19/Orca-4B-flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Orca-4B-flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/roskosmos19/Orca-4B-flash

SGLang

How to use roskosmos19/Orca-4B-flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "roskosmos19/Orca-4B-flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Orca-4B-flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "roskosmos19/Orca-4B-flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "roskosmos19/Orca-4B-flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use roskosmos19/Orca-4B-flash with Docker Model Runner:
```
docker model run hf.co/roskosmos19/Orca-4B-flash
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

A newer version of this model is available: roskosmos19/Orca-4B-Instruct

Orca-4B-flash

EN: A 4B System One + agentic model built on Qwen/Qwen3.5-4B. Its default system prompt steers it towards fast, calibrated, typed decisions (Choice / Score / Noul) with confidence estimates, and towards planning and tool use when a task needs more than one step. Because the base is Qwen3.5-4B, it also understands images and video.

DE: Ein 4B-Modell im Stil „System One + Agent“, aufgebaut auf Qwen/Qwen3.5-4B. Sein Standard-Systemprompt lenkt es auf schnelle, kalibrierte, typisierte Entscheidungen (Choice / Score / Noul) mit Konfidenzangabe und auf Planung und Tool-Nutzung, wenn eine Aufgabe mehrere Schritte braucht. Da die Basis Qwen3.5-4B ist, versteht es auch Bilder und Videos.

🛠️ Repository fix (2026-11-13)

EN: Before this update the repo could not be loaded: config.json described a different model (Qwen3 dense, 36 layers) than the weights (Qwen3.5 hybrid linear/full attention with vision encoder), and tokenizer.json, vocab.json and merges.txt were placeholders. Now the config, tokenizer and pre-processor files match the weights; the chat template is the official Qwen3.5 template (tools, thinking, images/video) with the Orca System One prompt as default system message; generation_config.json stops at <|im_end|>. The weights were not changed.

DE: Vor diesem Update ließ sich das Repo nicht laden: config.json beschrieb ein anderes Modell (Qwen3 dense, 36 Layer) als die Gewichte (Qwen3.5 mit hybrider Linear-/Full-Attention und Vision-Encoder), und tokenizer.json, vocab.json und merges.txt waren Platzhalter. Jetzt passen Config, Tokenizer und Preprocessor zu den Gewichten; das Chat-Template ist das offizielle Qwen3.5-Template (Tools, Thinking, Bilder/Video) mit dem Orca-System-One-Prompt als Standard-Systemnachricht; generation_config.json stoppt bei <|im_end|>. Die Gewichte wurden nicht verändert.

Model details / Modelldetails

Property / Eigenschaft	Value / Wert
Architecture / Architektur	`Qwen3_5ForConditionalGeneration` (hybrid Gated DeltaNet linear attention + full attention, 3:1)
Parameters / Parameter	~4.7B incl. vision encoder (text model ~4B)
Layers / Layer	32 (+1 MTP layer)
Context / Kontext	262,144 tokens
Modalities / Modalitäten	text, image, video → text
Vocabulary / Vokabular	248,320
Precision / Präzision	bfloat16
License / Lizenz	Apache 2.0

Requires a recent transformers with Qwen3.5 support (model_type: qwen3_5). / Benötigt eine aktuelle transformers-Version mit Qwen3.5-Unterstützung.

Usage / Nutzung

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "roskosmos19/Orca-4B-flash"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "Ticket: 'App crashes on login since update 2.3.' "
                                 "Choice: bug | feature_request | question. Answer with the label and a confidence 0-1."}
    ]}
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
    enable_thinking=False,   # fast System One mode / schneller System-One-Modus
).to(model.device)

out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

enable_thinking=False: direct answer, best for typed decisions / direkte Antwort, ideal für typisierte Entscheidungen.
Default (thinking on): the model reasons inside <think>…</think> first / Standard: das Modell denkt zuerst in <think>…</think>.
Pass your own system message to replace the default System One prompt / Eigene Systemnachricht ersetzt den Standard-Prompt.

System One prompt (default) / Standard-Systemprompt

You are Orca-4B-flash, a System One + agentic model. Prefer fast, calibrated, typed decisions (Choice / Score / Noul) when the task allows. When deeper reasoning or multi-step agency is required, plan, use tools and reflect. Always report confidence / uncertainty when making decisions. Be concise, precise and reliable for software integration.

Intended use / Einsatzzweck

EN: Decision steps inside applications (routing, scoring, guardrails), agent workflows that need short, structured answers with confidence, quick image/screenshot questions.
DE: Entscheidungsschritte in Anwendungen (Routing, Scoring, Guardrails), Agenten-Workflows mit kurzen, strukturierten Antworten samt Konfidenz, schnelle Fragen zu Bildern/Screenshots.

Limitations / Einschränkungen

EN: Confidence values are produced by the language model and are not formally calibrated; validate them on your own data. Structured output is not guaranteed without constrained decoding.
DE: Konfidenzwerte erzeugt das Sprachmodell selbst; sie sind nicht formal kalibriert, bitte auf eigenen Daten prüfen. Strukturierte Ausgaben sind ohne Constrained Decoding nicht garantiert.