Instructions to use city96/t5-v1_1-xxl-encoder-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use city96/t5-v1_1-xxl-encoder-gguf with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="city96/t5-v1_1-xxl-encoder-gguf", filename="t5-v1_1-xxl-encoder-Q3_K_L.gguf", )
output = llm( "Once upon a time,", max_tokens=512, echo=True ) print(output)
- Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use city96/t5-v1_1-xxl-encoder-gguf with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Use Docker
docker model run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- LM Studio
- Jan
- Ollama
How to use city96/t5-v1_1-xxl-encoder-gguf with Ollama:
ollama run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- Unsloth Studio new
How to use city96/t5-v1_1-xxl-encoder-gguf with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for city96/t5-v1_1-xxl-encoder-gguf to start chatting
- Docker Model Runner
How to use city96/t5-v1_1-xxl-encoder-gguf with Docker Model Runner:
docker model run hf.co/city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
- Lemonade
How to use city96/t5-v1_1-xxl-encoder-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull city96/t5-v1_1-xxl-encoder-gguf:Q4_K_M
Run and chat with the model
lemonade run user.t5-v1_1-xxl-encoder-gguf-Q4_K_M
List all available models
lemonade list
How do I load t5-v1_1-xxl-encoder-gguf?
same problem
Yes, just merged it a few minutes ago so you'll want to update. Should be this one:
and clip-vit-large-patch14.bin is?
I thought flux is using clip_l.safetensors from here? (just 200-MBs)
https://huggingface.co/lllyasviel/flux_text_encoders/tree/main
I can't use clip_l.safetensors with this DualCLIPLoader (GGUF),
so where can I get this clip-vit-large-patch14.bin?
here?
https://huggingface.co/openai/clip-vit-large-patch14/tree/main
I can't find the exact file.
EDIT: sorry it seems ComfyUI problem, portable version have problem, but my manual-install version works fine with clip_l.safetensors..
Thanks it works! Painfully slow on my Mac but no swap triggerred now and it finally gets there!
Does it work with Forge UI?
I can't use clip_l.safetensors with this DualCLIPLoader (GGUF) - same problem... I am still using portable version but I hope you have a solution for this. in the meantime, thank you for your efforts
I too face the same issue. On portable version and clip_l.safetensors isn't working.
There were recent updates to both the node and comfy, it may be worth updating both and retrying.
My bad.
Remember to set DualCLIPLoader (GGUF) type to flux.
Alreay set DualCLIPLoader (GGUF) type to flux... but same here.. :(
And different error today :
E:\ComfyUI_TEST\ComfyUI\custom_nodes\ComfyUI-GGUF\dequant.py:7: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data = torch.tensor(tensor.data)
E:\ComfyUI_TEST\python_embeded\Lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
hi i have downloaded the t5 xxl encoder gguf file and pasted in the clip folder but its not showing in the dual clip loader or any node that

