Instructions to use mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mandanya/Qwen2.5-Coder-0.5B-LCQ-v2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mandanya/Qwen2.5-Coder-0.5B-LCQ-v2")
model = AutoModelForCausalLM.from_pretrained("mandanya/Qwen2.5-Coder-0.5B-LCQ-v2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mandanya/Qwen2.5-Coder-0.5B-LCQ-v2

SGLang

How to use mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mandanya/Qwen2.5-Coder-0.5B-LCQ-v2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 with Docker Model Runner:
```
docker model run hf.co/mandanya/Qwen2.5-Coder-0.5B-LCQ-v2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model for building advanced queries in Logseq

Logseq use clojure script over datalog to interact with notes. LCQ - Logseq Clojure Query.

Description of the approach

About 100 copies were collected manually and about 4500 more were created on their basis using the Qwen2.5-Coder-7B-Instruct model. The test part of the dataset (about 100 synthetic copies) are run through the model with a system prompt describing the specifics of the queries and validated by the codestral-mamba model.

SYSTEM_PROMPT = """
You should create advanced query for logseq. 
Advanced query should be written in clojure script over datalog and should starts and ends with `#+BEGIN_QUERY` and `#+END_QUERY` respectively.
You should respond only with query, without any additional information.

query may consists of:
    - :title - title of the query (required)
    - :query - query itself, usually contains :find, :where, ... (required)
    - :result-transform - transform function for the result (optional)
    - :group-by-page? (true or false, optional)
    - :collapsed? (true or false, usually false, optional)

example of respond:
#+BEGIN_QUERY
...
#+END_QUERY
"""

Results

model	overal	zero_shot	1_shot	3_shot	5_shot
Qwen2.5-Coder-0.5B-LCQ-v2	0.3333	0.3333	nan	nan	nan
Qwen2.5-Coder-0.5B-LCQ-v1	0.2963	0.2963	nan	nan	nan
Qwen2.5-Coder-7B-Instruct-AWQ	0.0586	0.0247	0.0494	0.0988	0.0617
gpt-4o	0.0401	0.0123	0.0741	0.037	0.037
gpt-4o-mini	0.034	0.0123	0.0247	0.0617	0.037
Qwen2.5-Coder-3B-Instruct	0.0278	0	0.0123	0.0617	0.037
Qwen2.5-Coder-1.5B-Instruct	0.0123	0	0	0.0123	0.037
Qwen2.5-Coder-0.5B-Instruct	0.0031	0	0	0.0123	0

How to use

I prefer to run model with sglang:

python3.11 -m venv .venv --prompt llm-inf

source .venv/bin/activate

pip install "sglang[all]"
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/

python3.11 -m sglang.launch_server \
--model-path mandanya/Qwen2.5-Coder-0.5B-LCQ-v2 \
--port 23335 \
--host 0.0.0.0 \
--mem-fraction-static 0.5 \
--served-model-name "Qwen2.5-Coder-0.5B-LCQ-v2"