examples/README.md

# Janus-35B examples

Four minimal entry points. Pick the one that matches how you run models.

| File | Backend | When to use |
|---|---|---|
| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
| `transformers_quickstart.py` | Hugging Face Transformers | You want to run the Heretic safetensors (`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`) on GPU, optionally in 4-bit via bitsandbytes. |
| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
| `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf` and answers questions about an image. The only working vision path right now. |

All four apply the same Janus system prompt and sampling defaults
(`temp=0.6, top_p=0.95, top_k=20, repeat_penalty=1.05`) so behavior should
be consistent across backends modulo quantization noise. The three
non-Ollama scripts set them explicitly; `ollama_chat.py` inherits them
from the `Modelfile` / bridge files.

## Setup

### Ollama

Pull straight from HF (gets the bundled Q4_K_M GGUF + this repo's
root-level `template` / `system` / `params` files via HF's Ollama
bridge):

```bash
ollama pull hf.co/FoolDev/Janus-35B-HERETIC              # ~19 GB Q4_K_M (only bundled quant)
pip install requests
MODEL=hf.co/FoolDev/Janus-35B-HERETIC python ollama_chat.py
```

Or build a local `janus` tag from this repo's bundled GGUF without going
through the HF pull:

```bash
cd ..  &&  ./scripts/load_bundle.sh  &&  cd examples
python ollama_chat.py                                   # MODEL defaults to `janus`
```

For a non-bundled quant (e.g. Q3_K_M ~14 GB, Q5_K_M ~24 GB),
`./scripts/build.sh` downloads it from
`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`, patches the `Modelfile`
`FROM` line into a temp copy, and creates the local `janus` tag:

```bash
cd ..  &&  QUANT=Q5_K_M ./scripts/build.sh  &&  cd examples
python ollama_chat.py
```

### Transformers (safetensors)

```bash
pip install --upgrade "transformers>=4.45" accelerate sentencepiece bitsandbytes
python transformers_quickstart.py            # 4-bit, ~20 GB VRAM
python transformers_quickstart.py --no-4bit  # bf16, ~70 GB VRAM
```

### llama-cpp-python (GGUF, no daemon)

```bash
pip install llama-cpp-python  # CPU-only build
python llama_cpp_quickstart.py /path/to/Janus-35B-A3B.Q4_K_M.gguf --gpu-layers 99
```

For GPU offload, rebuild llama-cpp-python with the matching backend — see
the script header for `CMAKE_ARGS` recipes (CUDA, Metal, ROCm/HIP).

### Vision (image input)

```bash
# Pull the projector once (~903 MB) into the repo root:
cd ..  &&  ./scripts/fetch_vision.sh  &&  cd examples

pip install llama-cpp-python pillow
python llama_cpp_vision.py \
  --gguf ../Janus-35B-A3B.Q4_K_M.gguf \
  --mmproj ../Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
  --image /path/to/photo.jpg \
  --prompt "Describe this image."
```

Why not Ollama? Ollama's Go engine has the `qwen35` / `qwen35moe`
arch entries (text inference works in 0.24+), but the C++ llama.cpp
fallback that Ollama switches to when an mmproj is attached still
lacks them. `ollama create` accepts the dual-`FROM` and `ollama show`
reports `vision` capability, but the first inference call fails with
`error loading model architecture: unknown model architecture:
'qwen35moe'`. Tracked in
[ollama/ollama#15898](https://github.com/ollama/ollama/issues/15898).
Until that's fixed, llama.cpp / llama-cpp-python is the working path
for vision.