duyntnet/codegeex4-all-9b-imatrix-GGUF

Quantizations of https://huggingface.co/THUDM/codegeex4-all-9b

Inference Clients/UIs

• llama.cpp
• JanAI
• KoboldCPP
• text-generation-webui
• ollama
• GPT4All

---

From original readme

Get Started

Use 4.39.0<=transformers<=4.40.2 to quickly launch codegeex4-all-9b：

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained("THUDM/codegeex4-all-9b", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "THUDM/codegeex4-all-9b",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).to(device).eval()
inputs = tokenizer.apply_chat_template([{"role": "user", "content": "write a quick sort"}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ).to(device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_length=256)
    outputs = outputs[:, inputs['input_ids'].shape[1]:]
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

If you want to build the chat prompt manually, please make sure it follows the following format:

f"<|system|>\n{system_prompt}\n<|user|>\n{prompt}\n<|assistant|>\n"

Default system_prompt:

你是一位智能编程助手，你叫CodeGeeX。你会为用户回答关于编程、代码、计算机方面的任何问题，并提供格式规范、可以执行、准确安全的代码，并在必要时提供详细的解释。

The English version:

You are an intelligent programming assistant named CodeGeeX. You will answer any questions users have about programming, coding, and computers, and provide code that is formatted correctly.

For infilling ability, please use (without system prompt):

f"<|user|>\n<|code_suffix|>{suffix}<|code_prefix|>{prefix}<|code_middle|><|assistant|>\n"

Additional infos (like file path, programming language, mode) can be added. Example:

<|user|>
###PATH:src/example.py
###LANGUAGE:Python
###MODE:BLOCK
<|code_suffix|>{suffix}<|code_prefix|>{prefix}<|code_middle|><|assistant|>