Qwen2.5-Coder-7B-Instruct-OpenVINO-INT4

This repository contains an optimized OpenVINO™ IR version of Alibaba Cloud's Qwen2.5-Coder-7B-Instruct, quantized to INT4 precision using NNCF. This model is a state-of-the-art coding assistant, specifically optimized for high-performance code generation and technical reasoning on local Windows workstations.

---

🐍 Python Inference (Optimum-Intel)

To run this coding assistant locally using the optimum-intel library:

from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer

model_id = "CelesteImperia/Qwen2.5-Coder-7B-Instruct-OpenVINO-INT4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)

prompt = "Write a high-performance C# thread pool manager for a factory automation system."
messages = [
    {"role": "user", "content": prompt},
]
input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

outputs = model.generate(input_ids, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

💻 For C# / .NET Users (LLamaSharp Implementation)

This model is the ideal backend for a local IDE plugin or internal developer tool. Use LLamaSharp for native integration.

using LLama.Common;
using LLama;

// 1. Initialize the OpenVINO Model for Qwen2.5-Coder
var parameters = new ModelParams("path/to/openvino_model.xml")
{
    ContextSize = 16384, // Larger context for complex code files
    GpuLayerCount = 0 
};

// 2. Load Weights and Create Context
using var weights = LLamaWeights.LoadFromFile(parameters);
using var context = weights.CreateContext(parameters);
var executor = new StatelessExecutor(weights, parameters);

// 3. Coding Task Execution
var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.User, "Explain how to optimize this C# LINQ query for high-throughput sensor data.");

foreach (var text in executor.InferAsync(chatHistory, new InferenceParams { MaxTokens = 1024 }))
{
    Console.Write(text);
}

---

🏗️ Technical Details

• Optimization Tool: NNCF (Neural Network Compression Framework)
• Quantization: INT4 Asymmetric (Group Size: 128)
• Workstation Validation: Dual-GPU (RTX 3090 + RTX A4000)
• Infrastructure: S: NVMe Scratch / K: 12TB Warehouse

---

☕ Support the Forge

Maintaining a dual-GPU AI workstation and hosting high-bandwidth models requires significant resources. If our open-source tools power your projects, consider supporting our development:

Platform	Support Link
Global & India	Support via Razorpay

Scan to support via UPI (India Only):

---

📜 License

This model is released under the Apache 2.0 License.

---

Connect with the architect: Abhishek Jaiswal on LinkedIn