Instructions to use groxaxo/Code-Writer-V2-Obliterated with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use groxaxo/Code-Writer-V2-Obliterated with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="groxaxo/Code-Writer-V2-Obliterated")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("groxaxo/Code-Writer-V2-Obliterated")
model = AutoModelForMultimodalLM.from_pretrained("groxaxo/Code-Writer-V2-Obliterated")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use groxaxo/Code-Writer-V2-Obliterated with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "groxaxo/Code-Writer-V2-Obliterated"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/groxaxo/Code-Writer-V2-Obliterated

SGLang

How to use groxaxo/Code-Writer-V2-Obliterated with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "groxaxo/Code-Writer-V2-Obliterated" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "groxaxo/Code-Writer-V2-Obliterated" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Code-Writer-V2-Obliterated",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use groxaxo/Code-Writer-V2-Obliterated with Docker Model Runner:
```
docker model run hf.co/groxaxo/Code-Writer-V2-Obliterated
```

Code-Writer-V2-Obliterated / MANIFEST.md

groxaxo

Upload folder using huggingface_hub

3899a37 verified 17 days ago

preview code

Raw

History Blame Contribute Delete

3.43 kB

Qwen3.5-27B Writer-V2-Heretic + coding_mix_8k LoRA — FP8 (servable)

Built 2026-06-22. This is the corrected, servable FP8 artifact.

Provenance

Base: llmfan46/Qwen3.5-27B-Writer-V2-uncensored-heretic (BF16, model_type qwen3_5, hybrid GDN+full attention, VLM)
LoRA: coding_mix_8k checkpoint-25 (r16, alpha32), preserved at training/qwen35-writer-heretic-lora-r16/adapters/final_candidates/8k_checkpoint_25_candidate_20260622_072520
Merged BF16: /home/op/models/qwen/qwen35-writer-v2-heretic-coding-merged-bf16
Quant: llmcompressor 0.12.0, QuantizationModifier(targets=Linear, scheme=FP8_DYNAMIC) (compressed-tensors, float-quantized; channel-wise FP8 weights, dynamic per-token activations). Calibration-free.

What is quantized

FP8 (256 Linears): the standard text LMs only — q/k/v/o_proj on the 16 full-attention layers + gate/up/down_proj on all 64 layers. All 64-aligned → load via vLLM FP8 Marlin (W8A16) on Ampere.
BF16 (kept): lm_head, the entire GDN linear-attention subtree (model...linear_attn.*), and the entire vision tower (model.visual.*).

Why (do not regress this)

The first quant used plain-string ignore ("visual", "vision"…). llmcompressor matches exact names / re: regex, so those matched nothing → vision tower + GDN projections got FP8'd with Marlin-incompatible dims (2152, 48) → vLLM crashed on Ampere: size_n = 2152 is not divisible by tile_n_size = 64. Fixed ignore to ["lm_head", "re:.*linear_attn.*", "re:.*visual.*"]. Also: llmcompressor's save_pretrained calls from_accelerate() which grabs a GPU and OOMs after writing weights but before the tokenizer → run the quant with CUDA_VISIBLE_DEVICES="". Tokenizer/processor/chat_template here were copied from the merged BF16 dir.

Serving (validated)

Backend launcher: /home/op/ULTIMATE/NAMEOFMODEL/run_qwen35_27b_writer_heretic_coding_fp8.sh
vLLM 0.19.1 (conda env vllm), TP=2 on GPUs 0,1, dtype bfloat16, --max-model-len 200000, --kv-cache-dtype fp8, util 0.92, text-only, --disable-custom-all-reduce, --reasoning-parser qwen3. No MTP (this model has no MTP head — layers 0–63 only). No tool-calling.
TP must be 2 (or 4): num_key_value_heads=4 is not divisible by 3, so TP=3 is INVALID; only 3 GPUs available, so TP=2.
200k fits via the hybrid arch (only 16/64 layers carry growing KV) + fp8 KV: KV pool 53,312 tokens, "max concurrency for 200,000 tokens/request: 1.04x" (≈1 full-length request at a time; shorter requests are more concurrent).
Proxy family writer-heretic-coding-fp8 on http://127.0.0.1:12434 (backend :12562), 5 variants: writer-heretic-coding-fp8-{instruct,general,coding,thinking,rp} (rp = roleplay; added to the proxy's global presets).
Sampling = official Qwen3.5-27B recommended (per-family preset_overrides, no presence_penalty): instruct 1.0/0.95, general 0.7/0.8, coding 0.6/0.95 (think), thinking 1.0/0.95 (think), rp 1.0/0.95 — all top_k 20, min_p 0.
Smoke: direct (:12562) + proxy (:12434), all modes (non-thinking + thinking + rp) return correct, coherent output. verify_fp8.py PASS.

Notes

../qwen35-writer-v2-heretic-coding-fp8-dynamic is the FIRST (broken) quant — kept for forensics; safe to delete to reclaim ~28GB.
Vision tower is present in BF16 but served text-only; enabling vision needs an image-path test first.