Support Us Through

image

GGUF Version

GGUF with Quants! Allowing you to run models using KoboldCPP and other AI Environments!

Quantizations:

Quant Type Benefits Cons
Q4_K_M โœ… Smallest size (fastest inference) โŒ Lowest accuracy compared to other quants
โœ… Requires the least VRAM/RAM โŒ May struggle with complex reasoning
โœ… Ideal for edge devices & low-resource setups โŒ Can produce slightly degraded text quality
Q5_K_M โœ… Better accuracy than Q4, while still compact โŒ Slightly larger model size than Q4
โœ… Good balance between speed and precision โŒ Needs a bit more VRAM than Q4
โœ… Works well on mid-range GPUs โŒ Still not as accurate as higher-bit models
Q8_0 โœ… Highest accuracy (closest to full model) โŒ Requires significantly more VRAM/RAM
โœ… Best for complex reasoning & detailed outputs โŒ Slower inference compared to Q4 & Q5
โœ… Suitable for high-end GPUs & serious workloads โŒ Larger file size (takes more storage)

Model Details:

Read the Model details on huggingface Model Detail Here!

Downloads last month
133
GGUF
Model size
1B params
Architecture
llama
Hardware compatibility
Log In to add your hardware

2-bit

4-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for N-Bot-Int/OpenElla-StoryWriter-TypeB-GGUF

Collection including N-Bot-Int/OpenElla-StoryWriter-TypeB-GGUF