metadata
license: mit
base_model: microsoft/NextCoder-7B
tags:
- code
- fp8
- quantized
- nextcoder
- microsoft
library_name: transformers
pipeline_tag: text-generation
NextCoder-7B-FP8
This is an FP8 quantized version of microsoft/NextCoder-7B.
Model Description
FP8 (8-bit floating point) quantization of NextCoder-7B for efficient inference on NVIDIA Ada Lovelace and newer GPUs.
Quantization Details
| Property | Value |
|---|---|
| Original Model | microsoft/NextCoder-7B |
| Quantization Method | FP8 (E4M3) via llm-compressor |
| Target Hardware | NVIDIA Ada Lovelace (RTX 40xx, RTX 5000 Ada, etc.) |
| Quantization Date | 2025-11-22 |
| Quantization Time | 47.0 minutes |
| Hardware Used | NVIDIA RTX 5000 Ada Generation (31.5 GB) |
Usage
Loading the Model
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load model with FP8 quantization
model = AutoModelForCausalLM.from_pretrained(
"TevunahAi/NextCoder-7B-FP8",
torch_dtype=torch.float8_e4m3fn, # FP8 dtype
device_map="auto",
low_cpu_mem_usage=True,
)
tokenizer = AutoTokenizer.from_pretrained("TevunahAi/NextCoder-7B-FP8")
# Generate code
messages = [{"role": "user", "content": "Write a function to calculate fibonacci numbers"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.7,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Requirements
pip install torch>=2.1.0 # FP8 support requires PyTorch 2.1+
pip install transformers>=4.40.0
pip install accelerate
Note: FP8 inference requires:
- PyTorch 2.1 or newer with CUDA support
- NVIDIA GPU with FP8 support (Ada Lovelace or newer: RTX 40xx series, RTX 5000 Ada, H100, etc.)
- CUDA 11.8 or newer
## Benefits of FP8
- **~50% memory reduction** compared to FP16/BF16
- **Faster inference** on Ada Lovelace GPUs with native FP8 support
- **Minimal quality loss** compared to INT8 or INT4 quantization
## Original Model
This quantization is based on microsoft/NextCoder-7B by Microsoft.
Please refer to the [original model card](https://huggingface.co/microsoft/NextCoder-7B) for:
- Training details
- Intended use cases
- Limitations and biases
- License information
## License
This model inherits the MIT license from the original NextCoder model.