mudasir13cs/E-commerce-query-rewriter-gguf

Query Rewriter - GGUF Format

This repository contains GGUF format models for efficient inference with llama.cpp.

Note: This is the GGUF (quantized) version of the model. For the full HuggingFace format model, see the merged model repository.

Model Information

• Base Model: unsloth/Qwen2.5-3B-Instruct
• Task: Context-aware query rewriting for e-commerce
• Fine-tuning Method: LoRA (Low-Rank Adaptation) with Unsloth
• LoRA Config: R=64, Alpha=128, Dropout=0.05
• Training Examples: 3140 train, 392 validation, 394 test
• Training Epochs: 5
• Effective Batch Size: 16
• Learning Rate: 0.0002
• Max Sequence Length: 2048
• Optimizer: adamw_torch_fused
• Precision: bf16
• Trained On: NVIDIA RTX 4090 (24GB VRAM)

Fine-Tuning Details

Training Configuration

• Framework: Unsloth (optimized for fast training)
• LoRA Rank: 64
• LoRA Alpha: 128 (2x rank for optimal scaling)
• LoRA Dropout: 0.05
• Batch Size: 8 per device
• Gradient Accumulation: 2 steps
• Learning Rate Schedule: cosine
• Weight Decay: 0.01
• Warmup Ratio: 0.1

Dataset

The model was fine-tuned on a custom e-commerce dataset containing:

• Pronoun resolution (30%)
• Ellipsis expansion (20%)
• Ordinal references (15%)
• Product name references (15%)
• Price/category queries (10%)
• Navigation commands (5%)
• Query refinements (5%)

Total: ~10,000 examples from real e-commerce product data (Flipkart, Amazon, Okayhai).

Available Formats

This repository contains multiple quantization levels:

• f16: Full precision (largest, best quality) - ~6GB
• q4_k_m: 4-bit quantization (smallest, recommended for most use cases) - ~2GB
• q5_k_m: 5-bit quantization (balanced quality/size) - ~2.5GB
• q8_0: 8-bit quantization (high quality, larger size) - ~3.5GB

Usage with Different Backends

1. llama.cpp (Recommended for GGUF)

Installation

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j

Basic Usage

# Using llama-cli
./llama.cpp/build/bin/llama-cli \
    -m path/to/query-rewriter-q4_k_m.gguf \
    -p "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# Using llama-server (for API access)
./llama.cpp/build/bin/llama-server \
    -m path/to/query-rewriter-q4_k_m.gguf \
    --port 8080

Python Example

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="path/to/query-rewriter-q4_k_m.gguf",
    n_ctx=2048,  # Context window
    n_threads=4  # Number of CPU threads
)

# Prepare prompt (ChatML format)
prompt = '<|im_start|>system\nRewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow me details of iPhone 15 Pro from Smartphones search<|im_end|>'

# Generate
response = llm(
    prompt,
    max_tokens=128,
    temperature=0.7,
    stop=["<|im_end|>", "<|im_start|>"]
)

print(response['choices'][0]['text'])

2. Ollama

Import GGUF Model to Ollama

Option 1: Using Modelfile (Recommended)

1. Create a Modelfile:

FROM ./query-rewriter-q4_k_m.gguf

TEMPLATE '<|im_start|>system\nRewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow me details of iPhone 15 Pro from Smartphones search<|im_end|>'

PARAMETER temperature 0.7
PARAMETER num_predict 128
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

2. Import the model:

ollama create query-rewriter -f Modelfile

Option 2: Direct Import

# Import GGUF file directly
ollama import query-rewriter-q4_k_m.gguf

Usage with Ollama

# Command line
ollama run query-rewriter "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# With context
ollama run query-rewriter "Context: Previous search: Smartphones\nQuery: show me that one"

Python API

import requests

# Generate
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "query-rewriter",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "stream": False,
        "options": {
            "temperature": 0.7,
            "num_predict": 128,
            "stop": ["<|im_end|>", "<|im_start|>"]
        }
    }
)

print(response.json()["response"])

Chat API

import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "query-rewriter",
        "messages": [
            {"role": "system", "content": "Rewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information."},
            {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
        ],
        "stream": False
    }
)

print(response.json()["message"]["content"])

3. vLLM (For Merged Models)

Note: vLLM works best with HuggingFace format models. Use the merged model instead of GGUF.

Installation

pip install vllm

Usage

from vllm import LLM, SamplingParams

# Load merged model (not GGUF)
llm = LLM(
    model="USERNAME/E-commerce-query-rewriter",
    trust_remote_code=True,
    max_model_len=2048
)

# Prepare prompt
prompt = '<|im_start|>system\nRewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow me details of iPhone 15 Pro from Smartphones search<|im_end|>'

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=128,
    stop=["<|im_end|>", "<|im_start|>"]
)

# Generate
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

vLLM Server

# Start server
python -m vllm.entrypoints.openai.api_server \
    --model USERNAME/E-commerce-query-rewriter \
    --trust-remote-code \
    --port 8000

# Use OpenAI-compatible API
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "query-rewriter",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "max_tokens": 128,
        "temperature": 0.7
    }'

4. Text Generation Inference (TGI)

Note: TGI works with HuggingFace format models. Use the merged model.

Installation

# Using Docker (recommended)
docker pull ghcr.io/huggingface/text-generation-inference:latest

Usage

docker run --gpus all \
    -p 8080:80 \
    -v /path/to/model:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id USERNAME/E-commerce-query-rewriter \
    --trust-remote-code

Python Client

from text_generation import Client

client = Client("http://localhost:8080")

response = client.generate(
    prompt='<|im_start|>system\nRewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow me details of iPhone 15 Pro from Smartphones search<|im_end|>',
    max_new_tokens=128,
    temperature=0.7,
    stop_sequences=["<|im_end|>", "<|im_start|>"]
)

print(response.generated_text)

5. Transformers (For Merged Models)

Note: Use the merged HuggingFace model for Transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load merged model
model = AutoModelForCausalLM.from_pretrained(
    "USERNAME/E-commerce-query-rewriter",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "USERNAME/E-commerce-query-rewriter",
    trust_remote_code=True
)

# Prepare input
messages = [
    {"role": "system", "content": "Rewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information."},
    {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
]

# Apply chat template
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)

Prompt Format

The model uses ChatML format (Qwen2.5's native format):

<|im_start|>system
Rewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.<|im_end|>
<|im_start|>user
Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one<|im_end|>
<|im_start|>assistant
show me details of iPhone 15 Pro from Smartphones search<|im_end|>

Input Structure

System Message:

Rewrite the ambiguous query using the provided context to make it clear and searchable. Resolve pronouns, ellipsis, ordinals, and other ambiguous references using the context information.

User Message (Context + Query):

Context:
Previous search: [category]
State: [SEARCH_RESULTS|PRODUCT_DETAIL|INITIAL]
Last command: [show_list|show_item|go_back|close]
Products (N): [product1, product2, ...]
Product count: N

Query: [user query]

Expected Output:

show me details of iPhone 15 Pro from Smartphones search

Example Prompts

Example 1: Pronoun Resolution

Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one

Example 2: Ellipsis Expansion

Context:
Previous search: Laptops
State: SEARCH_RESULTS
Last command: show_list
Products (5): MacBook Pro, Dell XPS, HP Spectre, Lenovo ThinkPad, ASUS ZenBook
Product count: 5

Query: under 50000

Example 3: Ordinal Reference

Context:
Previous search: Headphones
State: SEARCH_RESULTS
Last command: show_list
Products (4): Sony WH-1000XM5, Bose QuietComfort, AirPods Max, Sennheiser Momentum
Product count: 4

Query: show me the second one

Quantization Comparison

Format	Size	Quality	Use Case
f16	~6GB	Best	Maximum quality, sufficient VRAM
q8_0	~3.5GB	Excellent	High quality, moderate VRAM
q5_k_m	~2.5GB	Very Good	Balanced quality/size
q4_k_m	~2GB	Good	Smallest size, limited VRAM

Performance

• Inference Speed: Optimized for CPU and GPU (CUDA)
• Memory Usage: Significantly lower than original models
• Quality: Minimal quality loss with quantization
• Test Accuracy: 97.72% (385/394) on test set

Download

Download the desired quantization level:

# Using huggingface-cli
huggingface-cli download USERNAME/E-commerce-query-rewriter-gguf \
    query-rewriter-q4_k_m.gguf \
    --local-dir ./models

# Or download all formats
huggingface-cli download USERNAME/E-commerce-query-rewriter-gguf \
    --local-dir ./models

Related Models

• Merged Model (HuggingFace Format): USERNAME/E-commerce-query-rewriter

  • Full model in HuggingFace format
  • Can be used with Transformers, Unsloth, or other HF-compatible libraries
  • Suitable for further fine-tuning or inference

• Original LoRA Adapter: See merged model repository for LoRA adapter details

Backend Comparison

Backend	Format	Best For	Pros	Cons
llama.cpp	GGUF	CPU/GPU inference, edge devices	Fast, low memory, cross-platform	Limited to GGUF format
Ollama	GGUF	Local development, easy deployment	Simple API, auto-manages models	Requires model import
vLLM	HF	High-throughput serving	Very fast, batching support	Requires HF format, more memory
TGI	HF	Production serving	Optimized serving, Docker support	Requires HF format
Transformers	HF	Research, fine-tuning	Full flexibility, easy integration	Slower inference, more memory

Requirements

For GGUF Models (llama.cpp, Ollama)

• llama.cpp (for C/C++ usage)
• llama-cpp-python (for Python usage)
• Ollama (optional, for Ollama backend)
• CUDA (optional, for GPU acceleration)

For Merged Models (vLLM, TGI, Transformers)

• vLLM (for high-throughput serving)
• Text Generation Inference (for production serving)
• Transformers (for research/fine-tuning)
• CUDA (recommended for GPU acceleration)

Citation

If you use this model, please cite:

@software{ecommerce_agent_models,
  title = {E-commerce Agent Models - Query Rewriter},
  author = {Syed Mudasir},
  year = {2025},
  url = {https://huggingface.co/USERNAME/E-commerce-query-rewriter-gguf}
}

License

Apache 2.0

Notes

• GGUF models are optimized for inference, not training
• Use q4_k_m for most production deployments
• f16 format is recommended for maximum quality if VRAM allows
• For training or further fine-tuning, use the merged HuggingFace model
• The model was trained on English e-commerce data and performs best on similar queries