Add model card

README.md

@@ -0,0 +1,79 @@
+---
+tags:
+- fp8
+- moe
+- vllm
+- llm-compressor
+- compressed-tensors
+library_name: transformers
+license: apache-2.0
+base_model: thinkingmachines/Inkling
+pipeline_tag: image-text-to-text
+---
+
+# Inkling-FP8-dynamic
+
+## Model Overview
+- **Model Architecture:** InklingForConditionalGeneration
+  - **Input:** Text / Image / Audio
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Weight quantization:** FP8
+  - **Activation quantization:** FP8
+- **Release Date:** 2026-07-15
+- **Version:** 1.0
+- **Model Developers:** RedHatAI
+
+This model is a quantized version of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling), a 975B total / 41B active parameter multimodal Mixture-of-Experts model that accepts text, image, and audio inputs and generates text outputs.
+
+### Model Optimizations
+
+This model was obtained by quantizing the weights and activations of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling) to FP8 data type using dynamic per-token quantization, ready for inference with vLLM.
+This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
+
+Weights are quantized statically using per-channel FP8 scaling, and activations are quantized dynamically at inference time using per-token scaling. Only the weights and activations of the linear (attention and MoE expert) layers within the language backbone are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor). The vision encoder, audio encoder, token embedding and unembedding layers, normalization layers, biases, MoE routing/gating logic, shared experts, and the model's early dense MLP layers are kept in their original precision.
+
+## Deployment
+
+### Use with vLLM
+
+This model can be deployed using [vLLM](https://docs.vllm.ai/en/latest/).
+
+1. Start the vLLM server:
+```
+vllm serve RedHatAI/Inkling-FP8-dynamic \
+  --tensor-parallel-size 8 \
+  --max-model-len 131072 \
+  --gpu-memory-utilization 0.90 \
+  --limit-mm-per-prompt '{"image": 4, "audio": 1}'
+```
+
+> **Tip:** For text-only workloads, pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip the vision/audio encoder memory allocation and free up GPU memory for a longer context window.
+
+2. Send requests to the server:
+
+```python
+from openai import OpenAI
+
+openai_api_key = "EMPTY"
+openai_api_base = "http://<your-server-host>:8000/v1"
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+model = "RedHatAI/Inkling-FP8-dynamic"
+
+messages = [
+    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
+]
+
+outputs = client.chat.completions.create(
+    model=model,
+    messages=messages,
+)
+
+generated_text = outputs.choices[0].message.content
+print(generated_text)
+```