Howto run nvidia/NV-Embed-v2 on GPU

#26

by hozbey - opened Dec 4, 2024

Dec 4, 2024

Hello,
I am using "nvidia/NV-Embed-v2" with Open-WebUI for RAG(Retrieval-augmented generation). But the Model is running on CPU.
Would you please comment that how could I run "nvidia/NV-Embed-v2" on GPU resources?

Best Regards

Huseyin A. Ozbey

skfrost19

Dec 6, 2024

load like this

model_embed_nvidia = AutoModel.from_pretrained(
        f"{MOUNT_DIR}/NV-Embed-v2", device_map={"": "cuda:0"}, trust_remote_code=True
    )

yansanity

Dec 28, 2024

load like this

model_embed_nvidia = AutoModel.from_pretrained(
        f"{MOUNT_DIR}/NV-Embed-v2", device_map={"": "cuda:0"}, trust_remote_code=True
    )

This will only use the first GPU. Try this

model = AutoModel.from_pretrained('nvidia/NV-Embed-v2', trust_remote_code=True, device_map="auto")

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment