base README.md

README.md

@@ -0,0 +1,178 @@
+---
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+- INT4
+- quantized
+- llm-compressor
+- compressed-tensors
+- red hat
+base_model:
+- Qwen/Qwen3-Coder-Next
+---
+
+
+# Qwen3-Coder-Next.w4a16
+
+## Model Overview
+- **Model Architecture:** Qwen3NextForCausalLM
+  - **Input:** Text
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Weight quantization:** INT4
+  - **Activation quantization:** FP16
+- **Release Date:** 
+- **Version:** 1.0
+- **Model Developers:**: Red Hat
+
+Quantized version of [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next).
+
+### Model Optimizations
+
+This model was obtained by quantizing the weights and activations of [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) to INT4 data type.
+This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.
+Only the weights and activations of the linear operators within transformers blocks of the language model are quantized. 
+
+## Deployment
+
+### Use with vLLM
+
+1. Initialize vLLM server:
+```
+vllm serve inference-optimization/Qwen3-Coder-Next.w4a16 --port 8000 --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+ 
+```
+
+2. Send requests to the server:
+
+```python
+# Your tool implementation
+def square_the_number(num: float) -> dict:
+    return num ** 2
+
+# Define Tools
+tools=[
+    {
+        "type":"function",
+        "function":{
+            "name": "square_the_number",
+            "description": "output the square of the number.",
+            "parameters": {
+                "type": "object",
+                "required": ["input_num"],
+                "properties": {
+                    'input_num': {
+                        'type': 'number', 
+                        'description': 'input_num is a number that will be squared'
+                        }
+                },
+            }
+        }
+    }
+]
+
+from openai import OpenAI
+# Define LLM
+client = OpenAI(
+    # Use a custom endpoint compatible with OpenAI API
+    base_url='http://localhost:8000/v1',  # api_base
+    api_key="EMPTY"
+)
+ 
+messages = [{'role': 'user', 'content': 'square the number 1024'}]
+
+completion = client.chat.completions.create(
+    messages=messages,
+    model="RedHatAI/Qwen3-Coder-Next.w4a16",
+    max_tokens=65536,
+    tools=tools,
+)
+
+print(completion.choices[0])
+```
+
+
+## Creation
+
+This model was quantized using the [llm-compressor](https://github.com/vllm-project/llm-compressor) library as shown below.
+
+<details>
+  <summary>Creation details</summary>
+
+```python
+from datasets import load_dataset
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from llmcompressor import oneshot
+from llmcompressor.modifiers.quantization import GPTQModifier
+
+MODEL_ID = "Qwen/Qwen3-Coder-Next"
+
+# Load model.
+model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+
+NUM_CALIBRATION_SAMPLES=512
+MAX_SEQUENCE_LENGTH=2048
+
+# Load dataset.
+ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]")
+ds = ds.shuffle(seed=42)
+
+# Preprocess the data into the format the model is trained with.
+def preprocess(example):
+    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False, )}
+
+ds = ds.map(preprocess)
+
+# Tokenize the data (be careful with bos tokens - we need add_special_tokens=False since the chat_template already added it).
+def tokenize(sample):
+    return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
+ds = ds.map(tokenize, remove_columns=ds.column_names)
+
+# Configure the quantization algorithm to run.
+recipe = GPTQModifier(targets="Linear", scheme="W4A16", weight_observer="mse", ignore= ['re:.*lm_head', 're:.*mlp.gate$', 're:.*mlp.shared_expert_gate$', 're:.*linear_attn.*'])
+
+# Apply quantization.
+oneshot(
+    model=model, dataset=ds,
+    recipe=recipe,
+    max_seq_length=MAX_SEQUENCE_LENGTH,
+    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
+)
+
+# Save to disk compressed.
+SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-G128"
+model.save_pretrained(SAVE_DIR, save_compressed=True)
+tokenizer.save_pretrained(SAVE_DIR)
+```
+</details>
+
+
+## Evaluation
+
+
+The model was evaluated on the OpenLLM leaderboard task, using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
+[vLLM](https://docs.vllm.ai/en/stable/) was used for all evaluations.
+
+<details>
+  <summary>Evaluation details</summary>
+
+  **Coding Benchmarks **
+  
+  **SWE-Bench**
+  ```
+  python -m swebench.harness.run_evaluation \
+    --dataset_name princeton-nlp/SWE-bench_Lite \
+    --predictions_path preds.json \
+    --run_id validate-preds
+  ```
+
+</details>
+
+
+## Accuracy
+
+| Category | Metric | Qwen3-Coder-Next | Qwen3-Coder-Next.w4a16 | Recovery (%) |
+|----------|--------|-------------|-------------------|--------------|
+| SWE-Bench | Lite | 49.33 | 44 | 89.2 |