ewin-reg

/

WeMM-Embedding-2B-Quantized

Like 5

Yah

Great! Many thanks for your quick reply and effort!

Ur welcome

@WC-ECL How was your experience wiith using it?

@ewin-reg I used the smaller 1.5GB model in my application, 256 dims still performed pretty well despite the bulky torch env.
However, so far I only tested in Windows by CUDA, not yet in macOS. Do you know if it can work well with MPS for good acceleration too?

Hi @WC-ECL , glad the smaller checkpoint held up well.

I checked modeling_wemm_embedding.py. The INT4 W4A8 layers just unpack uint8 bits and call F.linear, nothing CUDA-specific, so those should run fine on MPS.

The FP8 modules are the real risk: the vocab embedding table and the 6 full-attention layers store weights as torch.float8_e4m3fn and dequantize by indexing into that tensor then casting to float32. MPS float8 support varies by PyTorch version. That's where this breaks if it breaks.

Test it first:

import torch
x = torch.zeros(4, 4, dtype=torch.float8_e4m3fn, device="mps")
print(x.to(torch.float32))

If that throws, you've found the wall. If it works, nothing else in the model is CUDA-locked, so device="mps" should carry through fine. Post your PyTorch version once you know, it'll help anyone else hitting this.

@ewin-reg , Many thanks for your technical advices. Currently, I'm still developing Windows version for my app. If I encounter related technical issues on macOS version, would consult with you again.

@WC-ECL , hope it goes well! You should check out our more light weight quantized model that's from jina embeddings small omni retrieval!
https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized

@ewin-reg , Thanks for your new suggestion. It is a pity! If I knew it earlier, probably I would have selected it directly in my app. But maybe it would be a good alternative if my app need to handle audio embedding too.