v2ray

/

Mixtral-8x22B-v0.1

like 142

uwo

@v2ray

How much VRAM does it require ? Any GPT-Q versions available?

@RageshAntony It's a 140B model, so it would require around 280GB VRAM to load, and a few more GBs for the context and KV cache.
And afaik there's currently no GPTQ version available, but someone would do it soon I think.

@v2ray 280GB ??

Even H100 has 80GB only? How to load this then ?

@RageshAntony You use multiple GPUs, or use a quantized version(Which doesn't exist yet).