Instructions to use TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8")
model = AutoModelForCausalLM.from_pretrained("TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8

SGLang

How to use TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 with Docker Model Runner:
```
docker model run hf.co/TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8
```

granite-8b-code-instruct-4k-2048-Calibration-FP8

Premium FP8 quantization with 2,048 code-optimized calibration samples

This is a premium FP8 quantized version of ibm-granite/granite-8b-code-instruct-4k featuring rigorous code-optimized multi-dataset calibration for production-grade reliability. Quantized by TevunahAi on enterprise-grade hardware.

🎯 Recommended Usage: vLLM

For optimal performance with full FP8 benefits and code-optimized quality, use vLLM or TensorRT-LLM:

Quick Start with vLLM

pip install vllm

Python API:

from vllm import LLM, SamplingParams

# vLLM auto-detects FP8 from model config
llm = LLM(model="TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8", dtype="auto")

# Generate code
prompt = "Write a Python function to calculate fibonacci numbers:"
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate([prompt], sampling_params)
for output in outputs:
    print(output.outputs[0].text)

OpenAI-Compatible API Server:

vllm serve TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8 \
    --dtype auto \
    --max-model-len 4096

Then use with OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",  # dummy key
)

response = client.chat.completions.create(
    model="TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8",
    messages=[
        {"role": "user", "content": "Write a Python function to calculate fibonacci numbers"}
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)

vLLM Benefits

✅ Weights, activations, and KV cache in FP8
✅ ~8GB VRAM (50% reduction vs BF16)
✅ Native FP8 tensor core acceleration on Ada/Hopper GPUs
✅ Runs on consumer GPUs (RTX 4070, RTX 3080+)
✅ Premium 2048-sample code-optimized calibration
✅ Production-grade code quality

⚙️ Alternative: Transformers

This model can also be loaded with transformers. Note: Transformers will decompress FP8 → BF16 during inference. However, at 8B parameters, this is manageable (~16GB VRAM).

Transformers Example (Click to expand)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Loads FP8 weights but decompresses to BF16 during compute
model = AutoModelForCausalLM.from_pretrained(
    "TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8",
    device_map="auto",
    torch_dtype="auto",
    low_cpu_mem_usage=True,
)
tokenizer = AutoTokenizer.from_pretrained("TevunahAi/granite-8b-code-instruct-4k-2048-Calibration-FP8")

# Generate
prompt = "Write a Python function to calculate fibonacci numbers:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements:

pip install torch>=2.1.0 transformers>=4.40.0 accelerate compressed-tensors

System Requirements:

~16GB VRAM (decompressed to BF16)
CUDA 11.8 or newer
PyTorch 2.1+ with CUDA support

📊 Model Details

Property	Value
Base Model	ibm-granite/granite-8b-code-instruct-4k
Architecture	Dense (8B parameters)
Context Length	4K tokens
Quantization Method	FP8 E4M3 weight-only
Framework	llm-compressor + compressed_tensors
Calibration Samples	2,048 (4-8x industry standard)
Calibration Type	Code-optimized (4 datasets)
Storage Size	~8GB
VRAM (vLLM)	~8GB
VRAM (Transformers)	~16GB (decompressed to BF16)
Target Hardware	NVIDIA RTX 3080, RTX 4070, RTX 5000 Ada
Quantization Time	55.8 minutes

🏆 Premium Code-Optimized Calibration

This model was quantized using TevunahAi's premium code-focused calibration process:

Calibration Details

Total Samples: 2,048 (4-8x industry standard)
Datasets Used: 4 code-focused sources
Coverage: Comprehensive across coding tasks

Dataset	Samples	Purpose
HuggingFaceH4/CodeAlpaca_20K	512	Code instruction pairs
garage-bAInd/Open-Platypus	512	STEM/reasoning (includes code)
teknium/OpenHermes-2.5	512	Diverse instructions
theblackcat102/evol-codealpaca-v1	512	Evolved code examples

Why Code-Optimized Calibration?

Most FP8 quantizations use generic chat data for calibration. TevunahAi uses 2,048 samples from 4 code-focused datasets, ensuring:

✅ Superior code generation quality
✅ Better handling of programming syntax
✅ Optimized for multiple languages
✅ Accurate completion of complex code
✅ Production-grade reliability for coding tasks

For code models, generic calibration isn't enough. TevunahAi uses code-specific data.

🔧 Why FP8?

With vLLM/TensorRT-LLM:

✅ 50% memory reduction vs BF16 (weights + activations + KV cache)
✅ Faster inference via native FP8 tensor cores
✅ Better throughput with optimized kernels
✅ Minimal quality loss with premium code-optimized calibration
✅ Accessible on consumer GPUs (RTX 3080+, RTX 4070+)

With Transformers:

✅ Smaller download size (~8GB vs ~16GB BF16)
✅ Compatible with standard transformers workflow
⚠️ Decompresses to BF16 during inference (no runtime memory benefit)

For production inference, use vLLM to realize the full FP8 benefits.

💾 Model Files

This model is stored as safetensors files (all required for inference). The compressed format enables efficient storage and faster downloads.

🔬 IBM Granite Code Models

Granite Code models are specifically trained for enterprise code generation. This 8B parameter version offers:

Strong code generation across 100+ programming languages
Optimized for enterprise coding tasks
4K context window
Excellent efficiency for fast iteration
Apache 2.0 license for commercial use

📈 IBM Granite Code Family

TevunahAi provides premium FP8 quantizations for the IBM Granite Code family:

Model	Parameters	Context	Quantization Time	VRAM Usage
granite-8b-code-instruct-4k-2048-Calibration-FP8 (this)	8B	4K	55.8 min	~8GB
granite-20b-code-instruct-8k-2048-Calibration-FP8	20B	8K	124.8 min	~20GB
granite-34b-code-instruct-8k-2048-Calibration-FP8	34B	8K	230.9 min	~34GB

All models calibrated with identical premium 2048-sample code-focused datasets.

⚖️ Comparison: Standard vs Premium Calibration

TevunahAi offers two quantization tiers for this model:

Version	Calibration	Samples	Datasets	Quant Time	Use Case
Standard FP8	Basic	512	1 generic	~23 min	Quick deployment
Premium FP8 (this)	Code-optimized	2,048	4 code-focused	56 min	Production-grade

When to Choose Premium:

✅ Production deployments
✅ Quality-critical applications
✅ API services at scale
✅ Benchmarking and evaluation
✅ Enterprise code generation

When Standard is Fine:

✅ Quick testing
✅ Development/prototyping
✅ Resource-constrained environments
✅ Non-critical applications

🔬 Quantization Infrastructure

Professional hardware for premium calibration:

CPUs: Dual Intel Xeon Max 9480 (224 threads, 128GB HBM2e @ 2000 GB/s)
Memory: 256GB DDR5-4800 (16 DIMMs, 8-channel per socket, ~614 GB/s)
Total Memory Bandwidth: ~2,614 GB/s aggregate
GPU: NVIDIA RTX 5000 Ada Generation (32GB VRAM, native FP8 support)
Software: Ubuntu 25.10 | Python 3.12 | PyTorch 2.8 | CUDA 13.0 | llm-compressor

Why This Matters:

56 minutes of rigorous quantization and validation
Code-specific calibration requires specialized datasets
Professional infrastructure enables quality impossible on consumer setups

📚 Original Model

This quantization is based on ibm-granite/granite-8b-code-instruct-4k by IBM.

For comprehensive information about:

Model architecture and training methodology
Supported programming languages
Evaluation benchmarks and results
Ethical considerations

Please refer to the original model card.

🔧 Hardware Requirements

Minimum (vLLM):

GPU: NVIDIA RTX 3080 (10GB) or better
VRAM: 8GB minimum, 10GB+ recommended
CUDA: 11.8 or newer

Recommended (vLLM):

GPU: NVIDIA RTX 4070 / 4090 / RTX 5000 Ada
VRAM: 12GB+
CUDA: 12.0+

Transformers:

GPU: Any CUDA-capable GPU
VRAM: 16GB+
Works but not optimal for performance

📖 Additional Resources

vLLM Documentation: docs.vllm.ai
TensorRT-LLM: github.com/NVIDIA/TensorRT-LLM
TevunahAi Models: huggingface.co/TevunahAi
llm-compressor: github.com/vllm-project/llm-compressor
IBM Granite: github.com/ibm-granite

📄 License

This model inherits the Apache 2.0 License from the original Granite model.

🙏 Acknowledgments

Original Model: IBM Granite team
Quantization Framework: Neural Magic's llm-compressor
Quantized by: TevunahAi

📝 Citation

If you use this model, please cite the original Granite work:

@misc{granite2024,
  title={Granite Code Models},
  author={IBM Research},
  year={2024},
  url={https://huggingface.co/ibm-granite/granite-8b-code-instruct-4k}
}

🌟 Why TevunahAi Premium Calibration FP8?

Task-Optimized Calibration

TevunahAi doesn't use one-size-fits-all calibration:

Model Type	Calibration Focus	Example Datasets
Code Models	Code-specific	CodeAlpaca, evol-codealpaca
General Models	Diverse instructions	UltraChat, SlimOrca
MoE Models	Balanced distribution	Multi-task datasets

The right calibration for the right model.

The Difference is in the Details

Aspect	Standard FP8	TevunahAi Premium FP8
Calibration Samples	128-512	2,048
Datasets	Single generic	4 code-focused
Calibration Time	~23 min	56 min
Edge Case Handling	Adequate	Superior
Code Quality	Good	Excellent
Production Ready	Maybe	Absolutely
Infrastructure	Consumer/Prosumer	Enterprise-grade