Update README.md

README.md

@@ -1,3 +1,92 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+base_model: ibm-granite/granite-34b-code-instruct-8k
+tags:
+  - fp8
+  - quantized
+  - code
+  - granite
+  - ibm
+  - llmcompressor
+  - vllm
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# granite-34b-code-instruct-8k-FP8
+
+This is an FP8 quantized version of [granite-34b-code-instruct-8k](https://huggingface.co/ibm-granite/granite-34b-code-instruct-8k) for efficient inference.
+
+## Model Description
+
+- **Base Model:** [granite-34b-code-instruct-8k](https://huggingface.co/ibm-granite/granite-34b-code-instruct-8k)
+- **Quantization:** FP8 (E4M3 format)
+- **Quantization Method:** llmcompressor oneshot with FP8 scheme
+- **Calibration Dataset:** open_platypus (512 samples)
+- **Quantization Time:** 31.0 minutes
+
+## Usage
+
+### With Transformers
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model = AutoModelForCausalLM.from_pretrained(
+    "TevunahAi/granite-34b-code-instruct-8k-FP8",
+    torch_dtype=torch.float8_e4m3fn,  # FP8 dtype
+    device_map="auto",
+    low_cpu_mem_usage=True,
+)
+
+tokenizer = AutoTokenizer.from_pretrained("TevunahAi/granite-34b-code-instruct-8k-FP8")
+
+# Generate
+prompt = "Write a Python function to calculate fibonacci numbers:"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=256)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### With vLLM (Recommended for production)
+```python
+from vllm import LLM, SamplingParams
+
+llm = LLM(model="TevunahAi/granite-34b-code-instruct-8k-FP8")
+sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
+
+prompts = ["Write a Python function to calculate fibonacci numbers:"]
+outputs = llm.generate(prompts, sampling_params)
+```
+
+## Quantization Details
+
+- **Target Layers:** All Linear layers except lm_head
+- **Precision:** FP8 (E4M3 format)
+- **Hardware Requirements:** NVIDIA Ada Lovelace or Hopper (native FP8) or Ampere with emulation
+
+### Quantization Infrastructure
+
+Quantized on professional hardware to ensure quality and reliability:
+- **CPUs:** Dual Intel Xeon Max 9480 (224 threads, 128GB HBM2e)
+- **GPU:** NVIDIA RTX 5000 Ada Generation (32GB VRAM) with native FP8 support
+- **Memory:** 256GB DDR5 + 128GB HBM2e = 384GB total
+- **Software:** Ubuntu 25.10 | Python 3.12 | PyTorch 2.8 | CUDA 13 | llm-compressor
+
+### Performance Notes
+
+This 34B model demonstrates optimal HBM2e utilization:
+- Full CPU/HBM2e processing path for maximum efficiency
+- Superior per-parameter performance (0.91 min/B)
+- Counterintuitively faster than smaller 20B model due to pure HBM2e workflow
+- Ideal size for our hardware architecture
+
+## License
+
+Apache 2.0 (same as original model)
+
+## Credits
+
+- Original model by [IBM Granite](https://huggingface.co/ibm-granite)
+- Quantized by [TevunahAi](https://huggingface.co/TevunahAi)
+- Quantization powered by [llm-compressor](https://github.com/vllm-project/llm-compressor)