|
|
--- |
|
|
license: mit |
|
|
base_model: microsoft/NextCoder-14B |
|
|
tags: |
|
|
- code |
|
|
- fp8 |
|
|
- quantized |
|
|
- nextcoder |
|
|
- microsoft |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# NextCoder-14B-FP8 |
|
|
|
|
|
**High-quality FP8 quantization of Microsoft's NextCoder-14B, optimized for production inference** |
|
|
|
|
|
This is an FP8 (E4M3) quantized version of [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) using compressed_tensors format. Quantized by [TevunahAi](https://huggingface.co/TevunahAi) on enterprise-grade hardware with 2048 calibration samples. |
|
|
|
|
|
## 🎯 Recommended Usage: vLLM |
|
|
|
|
|
For optimal performance with **full FP8 benefits** (2x memory savings + faster inference), use **vLLM** or **TensorRT-LLM**: |
|
|
|
|
|
### Quick Start with vLLM |
|
|
|
|
|
```bash |
|
|
pip install vllm |
|
|
``` |
|
|
|
|
|
**Python API:** |
|
|
|
|
|
```python |
|
|
from vllm import LLM, SamplingParams |
|
|
from transformers import AutoTokenizer |
|
|
|
|
|
# vLLM auto-detects FP8 from model config |
|
|
llm = LLM(model="TevunahAi/NextCoder-14B-FP8", dtype="auto") |
|
|
|
|
|
# Prepare prompt with chat template |
|
|
tokenizer = AutoTokenizer.from_pretrained("TevunahAi/NextCoder-14B-FP8") |
|
|
messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}] |
|
|
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
|
|
|
|
# Generate |
|
|
outputs = llm.generate(prompt, SamplingParams(temperature=0.7, max_tokens=512)) |
|
|
print(outputs[0].outputs[0].text) |
|
|
``` |
|
|
|
|
|
**OpenAI-Compatible API Server:** |
|
|
|
|
|
```bash |
|
|
vllm serve TevunahAi/NextCoder-14B-FP8 \ |
|
|
--dtype auto \ |
|
|
--max-model-len 4096 |
|
|
``` |
|
|
|
|
|
Then use with OpenAI client: |
|
|
|
|
|
```python |
|
|
from openai import OpenAI |
|
|
|
|
|
client = OpenAI( |
|
|
base_url="http://localhost:8000/v1", |
|
|
api_key="token-abc123", # dummy key |
|
|
) |
|
|
|
|
|
response = client.chat.completions.create( |
|
|
model="TevunahAi/NextCoder-14B-FP8", |
|
|
messages=[ |
|
|
{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"} |
|
|
], |
|
|
temperature=0.7, |
|
|
max_tokens=512, |
|
|
) |
|
|
|
|
|
print(response.choices[0].message.content) |
|
|
``` |
|
|
|
|
|
### vLLM Benefits |
|
|
|
|
|
- ✅ **Weights, activations, and KV cache in FP8** |
|
|
- ✅ **~14GB VRAM** (50% reduction vs BF16) |
|
|
- ✅ **Native FP8 tensor core acceleration** on Ada/Hopper GPUs |
|
|
- ✅ **Faster inference** with optimized CUDA kernels |
|
|
- ✅ **Single GPU deployment** on RTX 5000 Ada, RTX 4090, or H100 |
|
|
|
|
|
## ⚙️ Alternative: Transformers (Not Recommended) |
|
|
|
|
|
This model can be loaded with `transformers`, but **will decompress FP8 → BF16 during inference**, requiring ~28GB+ VRAM. For 14B models, **vLLM is strongly recommended** for practical single-GPU deployment. |
|
|
|
|
|
<details> |
|
|
<summary>Transformers Example (Click to expand)</summary> |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
import torch |
|
|
|
|
|
# Loads FP8 weights but decompresses to BF16 during compute |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
"TevunahAi/NextCoder-14B-FP8", |
|
|
device_map="auto", |
|
|
torch_dtype="auto", |
|
|
low_cpu_mem_usage=True, |
|
|
) |
|
|
tokenizer = AutoTokenizer.from_pretrained("TevunahAi/NextCoder-14B-FP8") |
|
|
|
|
|
# Generate code |
|
|
messages = [{"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}] |
|
|
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
|
|
inputs = tokenizer([text], return_tensors="pt").to(model.device) |
|
|
|
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=512, |
|
|
temperature=0.7, |
|
|
do_sample=True |
|
|
) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
**Requirements:** |
|
|
```bash |
|
|
pip install torch>=2.1.0 transformers>=4.40.0 accelerate compressed-tensors |
|
|
``` |
|
|
|
|
|
**System Requirements:** |
|
|
- **~28GB+ VRAM** (decompressed to BF16) - requires multi-GPU or high-end single GPU |
|
|
- CUDA 11.8 or newer |
|
|
- PyTorch 2.1+ with CUDA support |
|
|
|
|
|
**⚠️ Warning:** Most consumer GPUs will struggle with transformers inference at this size. Use vLLM for practical deployment. |
|
|
|
|
|
</details> |
|
|
|
|
|
## 📊 Quantization Details |
|
|
|
|
|
| Property | Value | |
|
|
|----------|-------| |
|
|
| **Base Model** | [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) | |
|
|
| **Quantization Method** | FP8 E4M3 weight-only | |
|
|
| **Framework** | llm-compressor + compressed_tensors | |
|
|
| **Storage Size** | ~14GB (sharded safetensors) | |
|
|
| **VRAM (vLLM)** | ~14GB | |
|
|
| **VRAM (Transformers)** | ~28GB+ (decompressed to BF16) | |
|
|
| **Target Hardware** | NVIDIA Ada (RTX 4000/5000) or Hopper (H100/GH200) | |
|
|
| **Quantization Date** | November 22, 2025 | |
|
|
|
|
|
### Quantization Infrastructure |
|
|
|
|
|
Professional hardware ensures consistent, high-quality quantization: |
|
|
|
|
|
- **CPUs:** Dual Intel Xeon Max 9480 (112 cores / 224 threads, 128GB HBM2e) |
|
|
- **GPU:** NVIDIA RTX 5000 Ada Generation (32GB VRAM, native FP8 support) |
|
|
- **Memory:** 256GB DDR5 + 128GB HBM2e = 384GB total system memory |
|
|
- **Software Stack:** Ubuntu 25.10 | Python 3.12 | PyTorch 2.8 | CUDA 13.0 | llm-compressor |
|
|
|
|
|
## 🔧 Why FP8? |
|
|
|
|
|
### With vLLM/TensorRT-LLM: |
|
|
- ✅ **50% memory reduction** vs BF16 (weights + activations + KV cache) |
|
|
- ✅ **Faster inference** via native FP8 tensor cores |
|
|
- ✅ **Single GPU deployment** on 24GB+ cards |
|
|
- ✅ **Better throughput** with optimized kernels |
|
|
- ✅ **Minimal quality loss** (sub-1% perplexity increase) |
|
|
|
|
|
### With Transformers: |
|
|
- ✅ **Smaller download size** (~14GB vs ~28GB BF16) |
|
|
- ✅ **Compatible** with standard transformers workflow |
|
|
- ⚠️ **Decompresses to BF16** during inference (no runtime memory benefit) |
|
|
- ❌ **Requires 28GB+ VRAM** - impractical for most setups |
|
|
|
|
|
**For 14B models, vLLM is essential for practical deployment.** |
|
|
|
|
|
## 💾 Model Files |
|
|
|
|
|
This model is sharded into multiple safetensors files (all required for inference). The compressed format enables efficient storage and faster downloads. |
|
|
|
|
|
## 🚀 Performance vs 7B |
|
|
|
|
|
The 14B model offers significant improvements over 7B: |
|
|
|
|
|
- ✅ **Superior code quality** and more accurate completions |
|
|
- ✅ **Enhanced understanding** of complex programming concepts |
|
|
- ✅ **Better reasoning** for difficult coding tasks |
|
|
- ✅ **Improved context handling** for larger codebases |
|
|
- ⚠️ **Trade-off:** 2x VRAM requirement (14GB vs 7GB with vLLM) |
|
|
|
|
|
**With vLLM**, the 14B model fits comfortably on a single RTX 4090 (24GB) or RTX 5000 Ada (32GB). |
|
|
|
|
|
|
|
|
|
|
|
## 📚 Original Model |
|
|
|
|
|
This quantization is based on [microsoft/NextCoder-14B](https://huggingface.co/microsoft/NextCoder-14B) by Microsoft. |
|
|
|
|
|
For comprehensive information about: |
|
|
- Model architecture and training methodology |
|
|
- Capabilities, use cases, and limitations |
|
|
- Evaluation benchmarks and results |
|
|
- Ethical considerations and responsible AI guidelines |
|
|
|
|
|
Please refer to the [original model card](https://huggingface.co/microsoft/NextCoder-14B). |
|
|
|
|
|
## 🔧 Hardware Requirements |
|
|
|
|
|
### Minimum (vLLM): |
|
|
- **GPU:** NVIDIA RTX 4090 (24GB) or RTX 5000 Ada (32GB) |
|
|
- **VRAM:** 16GB minimum, 24GB+ recommended |
|
|
- **CUDA:** 11.8 or newer |
|
|
|
|
|
### Recommended (vLLM): |
|
|
- **GPU:** NVIDIA RTX 5000 Ada (32GB) / H100 (80GB) |
|
|
- **VRAM:** 24GB+ |
|
|
- **CUDA:** 12.0+ |
|
|
|
|
|
### Transformers: |
|
|
- **GPU:** Multi-GPU setup or A100 (40GB+) |
|
|
- **VRAM:** 28GB+ (single GPU) or distributed across multiple GPUs |
|
|
- **Not recommended** for practical deployment |
|
|
|
|
|
## 📖 Additional Resources |
|
|
|
|
|
- **vLLM Documentation:** [docs.vllm.ai](https://docs.vllm.ai/) |
|
|
- **TensorRT-LLM:** [github.com/NVIDIA/TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) |
|
|
- **TevunahAi Models:** [huggingface.co/TevunahAi](https://huggingface.co/TevunahAi) |
|
|
- **llm-compressor:** [github.com/vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor) |
|
|
|
|
|
## 📄 License |
|
|
|
|
|
This model inherits the **MIT License** from the original NextCoder-14B model. |
|
|
|
|
|
## 🙏 Acknowledgments |
|
|
|
|
|
- **Original Model:** Microsoft NextCoder team |
|
|
- **Quantization Framework:** Neural Magic's llm-compressor |
|
|
- **Quantized by:** [TevunahAi](https://huggingface.co/TevunahAi) |
|
|
|
|
|
## 📝 Citation |
|
|
|
|
|
If you use this model, please cite the original NextCoder work: |
|
|
|
|
|
```bibtex |
|
|
@misc{nextcoder2024, |
|
|
title={NextCoder: Next-Generation Code LLM}, |
|
|
author={Microsoft}, |
|
|
year={2024}, |
|
|
url={https://huggingface.co/microsoft/NextCoder-14B} |
|
|
} |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
<div align="center"> |
|
|
|
|
|
**Professional AI Model Quantization by TevunahAi** |
|
|
|
|
|
*Enterprise-grade quantization on specialized hardware* |
|
|
|
|
|
[View all models](https://huggingface.co/TevunahAi) | [Contact for custom quantization](https://huggingface.co/TevunahAi) |
|
|
|
|
|
</div> |
