Update README.md
Browse files
README.md
CHANGED
|
@@ -11,5 +11,6 @@ them from NVIDIA model. The resulting model here seems to work fine for chat, wi
|
|
| 11 |
I have included extras directory in model files with the following tools:
|
| 12 |
- Script used to delete experts from nvidia model based on structure of saricles model, should be reusable for other REAP transfer between quants
|
| 13 |
- Modified chat template that allows turning off reasoning through the same mechanism as Qwen 3.5, enable_thinking: false in kwargs
|
|
|
|
| 14 |
- cpp tool to aggressively clear memory and swap out inactive processes before running model in order to max out context length, I get ~120K tokens with FP8 kv cache
|
| 15 |
-
- Example script to run vLLM with optimized parameters for speed and memory efficiency, for example limited CUDA graph captures.
|
|
|
|
| 11 |
I have included extras directory in model files with the following tools:
|
| 12 |
- Script used to delete experts from nvidia model based on structure of saricles model, should be reusable for other REAP transfer between quants
|
| 13 |
- Modified chat template that allows turning off reasoning through the same mechanism as Qwen 3.5, enable_thinking: false in kwargs
|
| 14 |
+
- Modified reasoning parser to support new template
|
| 15 |
- cpp tool to aggressively clear memory and swap out inactive processes before running model in order to max out context length, I get ~120K tokens with FP8 kv cache
|
| 16 |
+
- Example script to run vLLM with optimized parameters for speed and memory efficiency, for example limited CUDA graph captures, as well as custom chat template/reasoning parser.
|