initial commit

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+imatrix-*.dat filter=lfs diff=lfs merge=lfs -text
+*.gguf filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,87 @@
+---
+quantized_by: ubergarm
+pipeline_tag: text-generation
+base_model: stepfun-ai/Step-3.5-Flash
+base_model_relation: quantized
+license: apache-2.0
+tags:
+- imatrix
+- conversational
+- ik_llama.cpp
+- step3p5
+---
+
+## WIP
+- [ ] compute imatrix against bf16 and upload
+- [ ] cook some quants
+- [ ] test perplexity
+
+## `ik_llama.cpp` imatrix Quantizations of stepfun-ai/Step-3.5-Flash
+*NOTE* `ik_llama.cpp` can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.
+
+Some of ik's new quants are supported with [Nexesenex/croco.cpp](https://github.com/Nexesenex/croco.cpp) fork of KoboldCPP with Windows builds for CUDA 12.9. Also check for [Windows builds by Thireus here.](https://github.com/Thireus/ik_llama.cpp/releases) which have been CUDA 12.8.
+
+These quants provide best in class perplexity for the given memory footprint.
+
+## Big Thanks
+Shout out to Wendell and the **Level1Techs** crew, the community [Forums](https://forum.level1techs.com/t/deepseek-deep-dive-r1-at-home/225826), [YouTube Channel](https://www.youtube.com/@Level1Techs)!  **BIG thanks** for providing **BIG hardware** expertise and access to run these experiments and make these great quants available to the community!!!
+
+Also thanks to all the folks in the quanting and inferencing community on [BeaverAI Club Discord](https://huggingface.co/BeaverAI) and on [r/LocalLLaMA](https://www.reddit.com/r/LocalLLaMA/) for tips and tricks helping each other run, test, and benchmark all the fun new models! Thanks to huggingface for hosting all these big quants!
+
+Finally, I *really* appreciate the support from [aifoundry.org](https://aifoundry.org) so check out their open source RISC-V based solutions!
+
+## Quant Collection
+Perplexity computed against *wiki.test.raw*. (lower is "better")
+
+![Perplexity Chart](images/perplexity.png "Chart showing Perplexity vs Model Size.")
+(not sure why the y-axis doesn't look log scale on this one)
+
+These two are just a test quants for baseline perplexity comparison:
+* `BF16` TODO
+  - TODO
+* `Q8_0` TODO
+  - TODO
+
+*NOTE*: The first split file is much smaller on purpose to only contain metadata, its fine!
+
+## IQ5_K TODO
+TODO
+
+<details>
+
+<summary>👈 Secret Recipe</summary>
+
+```bash
+echo TODO
+```
+
+</details>
+
+## More Quants TODO
+
+## Quick Start
+
+```bash
+# Clone and checkout
+$ git clone https://github.com/ikawrakow/ik_llama.cpp
+$ cd ik_llama.cpp
+
+# Build for hybrid CPU+CUDA
+$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
+$ cmake --build build --config Release -j $(nproc)
+
+## https://github.com/ikawrakow/ik_llama.cpp/pull/1236
+## https://github.com/ikawrakow/ik_llama.cpp/pull/1231
+## https://github.com/ikawrakow/ik_llama.cpp/pull/1239
+## https://github.com/ikawrakow/ik_llama.cpp/pull/1240
+echo TODO
+```
+
+For tool use you can always bring your own template with `--jinja --chat-template-file myTemplate.jinja` and might need `--special` etc.
+
+## References
+* [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)
+* [Getting Started Guide (already out of date lol)](https://github.com/ikawrakow/ik_llama.cpp/discussions/258)
+* [ubergarm-imatrix-calibration-corpus-v02.txt](https://gist.github.com/ubergarm/edfeb3ff9c6ec8b49e88cdf627b0711a?permalink_comment_id=5682584#gistcomment-5682584)
+* [mainline llama.cpp PR19283](https://github.com/ggml-org/llama.cpp/pull/19283) converted with `pull/19283/head:pr/step3.5-flash@402fc2e4e` plus casting `step35.attention.sliding_window_pattern` to `[INT32]` as for some reason it defaults to `[BOOL]` for me.
+* [ik_llama.cpp PR1231](https://github.com/ikawrakow/ik_llama.cpp/pull/1231) imatrix & quantized with `ik/step35_compat@9a0b5e80`