Add paper link, library metadata, and code link

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +25 -5
README.md CHANGED
@@ -1,22 +1,28 @@
1
  ---
2
- license: apache-2.0
3
  datasets:
4
  - zerofata/Roleplay-Anime-Characters
5
  language:
6
  - en
7
- base_model:
8
- - LatitudeGames/Muse-12B
9
  pipeline_tag: text-generation
 
10
  tags:
11
  - nvfp4
12
  - text adventure
13
  - roleplay
14
  ---
 
15
  ![image/jpeg](muse.jpg)
16
 
17
  # Muse-12B-NVFP4-FP8
18
 
19
- Quantized weights of the [Muse-12B](https://huggingface.co/LatitudeGames/Muse-12B) model for use with nVidia Blackwell GPUs, in a hybrid format using NVFP4 with [Four Over Six](https://arxiv.org/abs/2512.02010) adaptive block scaling for the MLP layers and `FP8_DYNAMIC` for the self-attention layers. More information about the hybrid format [here](https://huggingface.co/DataSnake/Mistral-Nemo-Instruct-2407-NVFP4-FP8), but the short version is that FP8 attention has minimal impact on speed and VRAM usage while making a marked difference in output quality, especially at longer context lengths.
 
 
 
 
 
20
 
21
  ## Inference
22
  Tested on a RTX 5060 Ti 16GB with [Aphrodite Engine](https://github.com/aphrodite-engine/aphrodite-engine) and [vLLM](https://github.com/vllm-project/vllm). It requires compressed-tensors 0.14.0 or later, so you'll have to update the version in your venv if you use Aphrodite Engine 0.10.0 or an older version of vLLM. On my system, Aphrodite Engine 0.10.0 was able to run the checkpoint with a 32k context window with the `--single-user-mode` flag, while vLLM 0.20.0 and Aphrodite Engine 0.20.0, which don't have that flag, were able to do the same with `--max-num-seqs 1 --cudagraph-capture-sizes 2` flags, though with the caveat that each crashed with OOM errors the first time they ran the model but ran fine from the second time onwards.
@@ -73,4 +79,18 @@ As such, I would recommend using that format for inference.
73
  ## Credits
74
  Muse-12B was made by [Latitude Games](https://huggingface.co/LatitudeGames) with help from [Gryphe Padar](https://huggingface.co/Gryphe)
75
 
76
- Four Over Six was discovered by Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, and Song Han
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ base_model: LatitudeGames/Muse-12B
3
  datasets:
4
  - zerofata/Roleplay-Anime-Characters
5
  language:
6
  - en
7
+ license: apache-2.0
 
8
  pipeline_tag: text-generation
9
+ library_name: transformers
10
  tags:
11
  - nvfp4
12
  - text adventure
13
  - roleplay
14
  ---
15
+
16
  ![image/jpeg](muse.jpg)
17
 
18
  # Muse-12B-NVFP4-FP8
19
 
20
+ Quantized weights of the [Muse-12B](https://huggingface.co/LatitudeGames/Muse-12B) model for use with nVidia Blackwell GPUs, in a hybrid format using NVFP4 with [Four Over Six](https://huggingface.co/papers/2512.02010) adaptive block scaling for the MLP layers and `FP8_DYNAMIC` for the self-attention layers.
21
+
22
+ - **Paper:** [Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling](https://huggingface.co/papers/2512.02010)
23
+ - **Official Code:** [GitHub - mit-han-lab/fouroversix](https://github.com/mit-han-lab/fouroversix)
24
+
25
+ More information about the hybrid format [here](https://huggingface.co/DataSnake/Mistral-Nemo-Instruct-2407-NVFP4-FP8), but the short version is that FP8 attention has minimal impact on speed and VRAM usage while making a marked difference in output quality, especially at longer context lengths.
26
 
27
  ## Inference
28
  Tested on a RTX 5060 Ti 16GB with [Aphrodite Engine](https://github.com/aphrodite-engine/aphrodite-engine) and [vLLM](https://github.com/vllm-project/vllm). It requires compressed-tensors 0.14.0 or later, so you'll have to update the version in your venv if you use Aphrodite Engine 0.10.0 or an older version of vLLM. On my system, Aphrodite Engine 0.10.0 was able to run the checkpoint with a 32k context window with the `--single-user-mode` flag, while vLLM 0.20.0 and Aphrodite Engine 0.20.0, which don't have that flag, were able to do the same with `--max-num-seqs 1 --cudagraph-capture-sizes 2` flags, though with the caveat that each crashed with OOM errors the first time they ran the model but ran fine from the second time onwards.
 
79
  ## Credits
80
  Muse-12B was made by [Latitude Games](https://huggingface.co/LatitudeGames) with help from [Gryphe Padar](https://huggingface.co/Gryphe)
81
 
82
+ Four Over Six was discovered by Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, and Song Han
83
+
84
+ ## Citation
85
+
86
+ ```bibtex
87
+ @misc{cook2025sixaccuratenvfp4quantization,
88
+ title={Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling},
89
+ author={Jack Cook and Junxian Guo and Guangxuan Xiao and Yujun Lin and Song Han},
90
+ year={2025},
91
+ eprint={2512.02010},
92
+ archivePrefix={arXiv},
93
+ primaryClass={cs.CL},
94
+ url={https://arxiv.org/abs/2512.02010},
95
+ }
96
+ ```