tiny-granite-switch-4.1-8b-for-testing-GGUF

This is a modified, damaged copy of ibm-granite/granite-switch-4.1-8b, made only to test software. Its output is gibberish. Do not use it for anything else.

GGUF of ggml-org-dev/tiny-granite-switch-4.1-8b-for-testing, converted with convert_hf_to_gguf.py of llama.cpp (F16, then quantized to Q4_0 with llama-quantize, embeddings included). Used by the llama.cpp server tests of the granite-switch decision type and of text generation with embedded adapters.

The vocab of the GGUF is cut to keep it tiny: 100355 -> 3446 tokens. It keeps the special and control tokens (including the adapter control tokens <|guardian-core|> and <|decision|>), all single characters, the tokens needed by the decision labels and the results of the first 2000 BPE merges; the token ids (including adapters.token_ids_activate / token_ids_substitute) are remapped. Any text still tokenizes, but into more tokens than with the original tokenizer.

llama-server -hf ggml-org-dev/tiny-granite-switch-4.1-8b-for-testing-GGUF:Q4_0

License

Derived from granite-switch-4.1-8b by IBM, under the Apache License 2.0.

Downloads last month
28
GGUF
Model size
1.76M params
Architecture
graniteswitch
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ggml-org-dev/tiny-granite-switch-4.1-8b-for-testing-GGUF