Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,184 @@
+---
+license: other
+base_model: TheDrummer/Fallen-Command-A-111B-v1.1
+base_model_relation: quantized
+language:
+- en
+library_name: transformers
+tags:
+- nvfp4
+- fp4
+- modelopt
+- vllm
+- cohere2
+- command-a
+- dgx-spark
+- gb10
+- roleplay
+pipeline_tag: text-generation
+---
+
+# Fallen-Command-A-111B-v1.1 — NVFP4
+
+NVFP4 (4-bit floating-point, `group_size=16`) quantization of [TheDrummer/Fallen-Command-A-111B-v1.1](https://huggingface.co/TheDrummer/Fallen-Command-A-111B-v1.1) — a roleplay/creative finetune of Cohere's **Command-A** (Cohere2 architecture, 111B). Produced with a custom **3-node heterogeneous distributed pipeline** on a personal **2× NVIDIA DGX Spark + RTX 3090** setup. Stored as modelopt NVFP4 weights, served via vLLM's modelopt path.
+
+Cohere2 has a few architectural quirks — tied embeddings, `layer_norm_eps`, hybrid local/global attention — that needed pipeline-side handling; see [Cohere2-specific handling](#cohere2-specific-handling) below.
+
+---
+
+## Serving mode on Blackwell (GB10)
+
+On DGX Spark / GB10 with vLLM, this model serves as **weight-only FP4**: the 4-bit NVFP4 weights are dequantized for each matmul; activations stay BF16. vLLM 0.20.x has no FP4-activation GEMM kernel for Blackwell (sm_120/121), so the NVFP4 path is weight-only regardless of the `input_activations` field in `config.json` — on this stack a W4A4-config and a W4A16-config produce bit-identical output. This is the standard, and currently highest-quality, NVFP4 serving mode on Spark. On an FP4-activation-capable stack (TensorRT-LLM, or a future vLLM with a Blackwell FP4 GEMM) the same weights could run as true W4A4.
+
+A practical consequence: because serving is weight-only, the **calibration dataset does not affect the served output** — the per-tensor weight scales are determined by the weights alone. The calibration pass below is part of the standard modelopt flow but is effectively output-invariant for this serving mode.
+
+---
+
+## Quick facts
+
+| | |
+|---|---|
+| **Base model** | [TheDrummer/Fallen-Command-A-111B-v1.1](https://huggingface.co/TheDrummer/Fallen-Command-A-111B-v1.1) (Cohere Command-A finetune) |
+| **Architecture** | Cohere2ForCausalLM — 64 layers, hidden_size 12288, intermediate 36864, 96 attn heads, 8 KV heads, head_dim 128 |
+| **Notable arch features** | Parallel attention/MLP block, hybrid local/global attention (sliding_window 4096, pattern 4), tied input/output embeddings, RoPE θ=50000, 256K max context |
+| **Original size** | ~207 GB (BF16) |
+| **Quantized size** | ~69 GB (14 shards, see Files tab) |
+| **Quant format** | NVFP4 via [nvidia-modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.43.0, `group_size=16` |
+| **lm_head** | Kept BF16 (unquantized), listed in `quantization_config.ignore` |
+| **Quantized modules** | 448 Linear layers (64 × 7: q/k/v/o + gate/up/down) |
+| **KV cache** | Configurable at serve time (FP8 recommended) |
+| **Calibration** | 256-sample pass (~25.7 min); see note above on output-invariance |
+| **Conversion date** | 2026-05-22 |
+
+---
+
+## The hardware: 2× DGX Spark + 1× RTX 3090
+
+The cluster used to produce this artifact:
+
+| Node | GPU | Memory | Role |
+|---|---|---|---|
+| DX10-01 (GB10 Spark) | NVIDIA GB10 (sm_121) | 128 GB UMA | shard0: layers 0–29 + embed_tokens |
+| DX10-02 (GB10 Spark) | NVIDIA GB10 (sm_121) | 128 GB UMA | shard1: layers 30–59 |
+| eGPU host (Proxmox VM) | NVIDIA RTX 3090 (sm_86) | 24 GB VRAM | shard2: layers 60–63 + final norm + lm_head |
+
+A 30/30/4-layer split keeps each Spark well inside its 128 GB UMA budget while the 3090 handles the tail 4 layers plus the norm and lm_head. Ray RPC carries cross-node hidden states transparently; the Ampere 3090 has no native FP4 hardware but only handles BF16 calibration math, so the architecture mismatch is irrelevant until inference time — the exported NVFP4 file is identical to what an all-Blackwell cluster would produce.
+
+The pipeline is open-source at **[github.com/KaletoAI/distrib-nvfp4](https://github.com/KaletoAI/distrib-nvfp4)** (Apache 2.0): N-way layer splits via `--shard-layers a,b,c`, memory-sorted node placement so the smallest-VRAM node gets the smallest shard, and disk-checkpointed phases for resumable runs.
+
+---
+
+## Cohere2-specific handling
+
+Command-A's Cohere2 architecture needed three fixes beyond the standard modelopt NVFP4 flow:
+
+1. **Tied embeddings.** Cohere2 sets `use_embedding_sharing=true` — the output projection reuses `model.embed_tokens.weight` and the checkpoint has no separate `lm_head.weight`. The head-bearing shard reconstructs `lm_head` from `embed_tokens` so it can be exported (BF16) into the merged model.
+2. **Norm epsilon name.** Cohere2 names the layernorm epsilon `layer_norm_eps` (Llama/Mistral use `rms_norm_eps`); the per-layer export template reads the correct attribute with a fallback.
+3. **Generation config.** Cohere2's `generation_config` sets `cache_implementation=hybrid`, which the 1-layer export template (built with `use_cache=False`) rejects. It is dropped during per-layer export and the real `generation_config` is restored in the merged model.
+
+Calibration health-check on the run that produced this artifact — clean, no zero or NaN amax statistics:
+
+- shard0 (layers 0–29 + embed): **good=210, zero=0, nan=0**
+- shard1 (layers 30–59): **good=210, zero=0, nan=0**
+- shard2 (layers 60–63 + norm + lm_head): **good=28, zero=0, nan=0**
+
+(`NVFP4_DEFAULT_CFG` inserts 7 weight quantizers per layer.)
+
+After merge, `config.json` is patched to keep `lm_head` in `quantization_config.ignore`, set `input_activations.dynamic: true`, and inject `input_scale=1.0` for every weight quantizer (modelopt 0.43 omits these keys, and vLLM's loader otherwise registers an uninitialized parameter and decodes garbage).
+
+---
+
+## Verification
+
+Loaded and smoke-tested on a single DGX Spark (GB10) with vLLM `0.20.2rc1.dev53` — `FlashInferCutlassNvFp4LinearKernel` for the NVFP4 GEMM, FlashInfer attention backend:
+
+- Model weights occupy **~62.6 GiB**; on a 128 GB UMA Spark at `gpu-memory-utilization 0.90` this leaves a ~43 GiB KV-cache pool (≈175K tokens at 4K context).
+- All test generations are coherent and accurate — e.g. *"The capital of France is"* → *"Paris. The area of France is 212,935 square miles…"*; *"17 + 25"* → *"42."*
+
+The weight-scale layout was also verified directly: every `down_proj.weight_scale` is `[12288, 2304]` (2304 = intermediate 36864 / `group_size` 16), and there are no stray `_quantizer` / `_double_scale` keys — the checkpoint loads with stock vLLM.
+
+A formal throughput benchmark has not been run yet.
+
+---
+
+## Usage
+
+### vLLM (serve)
+
+Verified on GB10 with vLLM `0.20.2rc1`:
+
+```bash
+vllm serve /path/to/Fallen-Command-111B-NVFP4 \
+  --served-model-name Fallen-Command-111B-NVFP4 \
+  --attention-backend flashinfer \
+  --dtype auto \
+  --kv-cache-dtype fp8 \
+  --max-model-len 32768 \
+  --max-num-seqs 4 \
+  --gpu-memory-utilization 0.90 \
+  --enable-chunked-prefill \
+  --enable-prefix-caching \
+  --port 9007
+```
+
+vLLM auto-detects the modelopt NVFP4 quantization from `config.json` — no explicit `--quantization` flag is needed. `--gpu-memory-utilization 0.90` leaves enough KV-cache pool for 32K context at `max-num-seqs 4` on a 128 GB Spark; drop to 0.85 if you don't need the longer context.
+
+### llama-swap entry
+
+```yaml
+"Fallen-Command-111B-NVFP4":
+  proxy: "http://127.0.0.1:9007"
+  ttl: 0
+  checkEndpoint: "/health"
+  cmd: >-
+    /home/<user>/vllm-env/bin/python3 -m vllm.entrypoints.openai.api_server
+    --model /home/<user>/models/Fallen-Command-111B-NVFP4
+    --served-model-name Fallen-Command-111B-NVFP4
+    --attention-backend flashinfer
+    --dtype auto
+    --kv-cache-dtype fp8
+    --max-model-len 32768
+    --max-num-seqs 4
+    --gpu-memory-utilization 0.90
+    --enable-chunked-prefill
+    --enable-prefix-caching
+    --port 9007
+    --host 127.0.0.1
+```
+
+### Prompt format
+
+Use the **Cohere / Command chat template** (it ships in `tokenizer_config.json`, so `apply_chat_template` and vLLM's OpenAI server handle it automatically). See [TheDrummer's original card](https://huggingface.co/TheDrummer/Fallen-Command-A-111B-v1.1) for finetune-specific usage notes.
+
+---
+
+## Files in this repository
+
+- `model-NNNNN-of-00014.safetensors` — 14 shards, NVFP4-packed weights + scales (~69 GB total)
+- `model.safetensors.index.json` — weight map (1859 keys: 448 quantized linears × 3 scale/weight keys + injected `input_scale` keys + 64 layernorms + embed + lm_head)
+- `config.json` — Cohere2 config with `quantization_config.ignore=["lm_head"]` and `input_activations.dynamic: true`
+- `hf_quant_config.json`, `generation_config.json` — auxiliary modelopt + generation configs
+- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` — Command-A tokenizer, untouched from upstream
+
+---
+
+## Acknowledgments
+
+- **[TheDrummer](https://huggingface.co/TheDrummer)** for the Fallen-Command-A-111B finetune
+- **[Cohere / Cohere Labs](https://huggingface.co/CohereLabs)** for the Command-A base model and the Cohere2 architecture
+- **NVIDIA** for the DGX Spark / GB10 platform, the NVFP4 format, and [modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
+- **vLLM project** for modelopt NVFP4 inference support
+
+---
+
+## License
+
+This NVFP4 quantization inherits the license of the base model TheDrummer/Fallen-Command-A-111B-v1.1, which is derived from Cohere's **Command-A** — released under **CC-BY-NC 4.0** with Cohere's Acceptable Use Policy. **For research, evaluation, and personal non-commercial use only.**
+
+- Pipeline code (Apache 2.0): https://github.com/KaletoAI/distrib-nvfp4
+
+---
+
+## Status
+
+Single-author release. Feedback welcome — on the model artifact (vLLM behaviour, sampling, RP quality) and on the pipeline that built it.