Instructions to use groxaxo/Code-Writer-V2-Obliterated with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use groxaxo/Code-Writer-V2-Obliterated with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="groxaxo/Code-Writer-V2-Obliterated")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("groxaxo/Code-Writer-V2-Obliterated")
model = AutoModelForMultimodalLM.from_pretrained("groxaxo/Code-Writer-V2-Obliterated")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use groxaxo/Code-Writer-V2-Obliterated with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "groxaxo/Code-Writer-V2-Obliterated"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/groxaxo/Code-Writer-V2-Obliterated

SGLang

How to use groxaxo/Code-Writer-V2-Obliterated with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "groxaxo/Code-Writer-V2-Obliterated" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "groxaxo/Code-Writer-V2-Obliterated" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use groxaxo/Code-Writer-V2-Obliterated with Docker Model Runner:
```
docker model run hf.co/groxaxo/Code-Writer-V2-Obliterated
```

Code-Writer-V2-Obliterated / MANIFEST.md

groxaxo

Upload folder using huggingface_hub

3899a37 verified 17 days ago

preview code

Raw

History Blame Contribute Delete

3.43 kB

	# Qwen3.5-27B Writer-V2-Heretic + coding_mix_8k LoRA — FP8 (servable)

	Built 2026-06-22. This is the corrected, servable FP8 artifact.

	## Provenance
	- Base: `llmfan46/Qwen3.5-27B-Writer-V2-uncensored-heretic` (BF16, model_type qwen3_5, hybrid GDN+full attention, VLM)
	- LoRA: `coding_mix_8k` checkpoint-25 (r16, alpha32), preserved at
	`training/qwen35-writer-heretic-lora-r16/adapters/final_candidates/8k_checkpoint_25_candidate_20260622_072520`
	- Merged BF16: `/home/op/models/qwen/qwen35-writer-v2-heretic-coding-merged-bf16`
	- Quant: llmcompressor 0.12.0, `QuantizationModifier(targets=Linear, scheme=FP8_DYNAMIC)` (compressed-tensors, float-quantized; channel-wise FP8 weights, dynamic per-token activations). Calibration-free.

	## What is quantized
	- FP8 (256 Linears): the standard text LMs only — `q/k/v/o_proj` on the 16 full-attention layers + `gate/up/down_proj` on all 64 layers. All 64-aligned → load via vLLM FP8 Marlin (W8A16) on Ampere.
	- BF16 (kept): `lm_head`, the entire GDN linear-attention subtree (`model...linear_attn.`), and the entire vision tower* (`model.visual.*`).

	### Why (do not regress this)
	The first quant used plain-string `ignore` (`"visual"`, `"vision"`…). llmcompressor matches exact names / `re:` regex, so those matched nothing → vision tower + GDN projections got FP8'd with Marlin-incompatible dims (2152, 48) → vLLM crashed on Ampere: `size_n = 2152 is not divisible by tile_n_size = 64`. Fixed `ignore` to `["lm_head", "re:.linear_attn.", "re:.visual."]`.
	Also: llmcompressor's `save_pretrained` calls `from_accelerate()` which grabs a GPU and OOMs after writing weights but before the tokenizer → run the quant with `CUDA_VISIBLE_DEVICES=""`. Tokenizer/processor/chat_template here were copied from the merged BF16 dir.

	## Serving (validated)
	- Backend launcher: `/home/op/ULTIMATE/NAMEOFMODEL/run_qwen35_27b_writer_heretic_coding_fp8.sh`
	- vLLM 0.19.1 (conda env `vllm`), TP=2 on GPUs 0,1, dtype bfloat16, `--max-model-len 200000`, `--kv-cache-dtype fp8`, util 0.92, text-only, `--disable-custom-all-reduce`, `--reasoning-parser qwen3`. No MTP (this model has no MTP head — layers 0–63 only). No tool-calling.
	- TP must be 2 (or 4): `num_key_value_heads=4` is not divisible by 3, so TP=3 is INVALID; only 3 GPUs available, so TP=2.
	- 200k fits via the hybrid arch (only 16/64 layers carry growing KV) + fp8 KV: KV pool 53,312 tokens, "max concurrency for 200,000 tokens/request: 1.04x" (≈1 full-length request at a time; shorter requests are more concurrent).
	- Proxy family `writer-heretic-coding-fp8` on http://127.0.0.1:12434 (backend :12562), 5 variants: `writer-heretic-coding-fp8-{instruct,general,coding,thinking,rp}` (rp = roleplay; added to the proxy's global presets).
	- Sampling = official Qwen3.5-27B recommended (per-family `preset_overrides`, no presence_penalty): instruct 1.0/0.95, general 0.7/0.8, coding 0.6/0.95 (think), thinking 1.0/0.95 (think), rp 1.0/0.95 — all top_k 20, min_p 0.
	- Smoke: direct (:12562) + proxy (:12434), all modes (non-thinking + thinking + rp) return correct, coherent output. `verify_fp8.py` PASS.

	## Notes
	- `../qwen35-writer-v2-heretic-coding-fp8-dynamic` is the FIRST (broken) quant — kept for forensics; safe to delete to reclaim ~28GB.
	- Vision tower is present in BF16 but served text-only; enabling vision needs an image-path test first.