Instructions to use krogoldAI/QueryRefiner-0.5B-v0.1-SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use krogoldAI/QueryRefiner-0.5B-v0.1-SFT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="krogoldAI/QueryRefiner-0.5B-v0.1-SFT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("krogoldAI/QueryRefiner-0.5B-v0.1-SFT")
model = AutoModelForCausalLM.from_pretrained("krogoldAI/QueryRefiner-0.5B-v0.1-SFT", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use krogoldAI/QueryRefiner-0.5B-v0.1-SFT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "krogoldAI/QueryRefiner-0.5B-v0.1-SFT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "krogoldAI/QueryRefiner-0.5B-v0.1-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/krogoldAI/QueryRefiner-0.5B-v0.1-SFT

SGLang

How to use krogoldAI/QueryRefiner-0.5B-v0.1-SFT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "krogoldAI/QueryRefiner-0.5B-v0.1-SFT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "krogoldAI/QueryRefiner-0.5B-v0.1-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "krogoldAI/QueryRefiner-0.5B-v0.1-SFT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "krogoldAI/QueryRefiner-0.5B-v0.1-SFT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use krogoldAI/QueryRefiner-0.5B-v0.1-SFT with Docker Model Runner:
```
docker model run hf.co/krogoldAI/QueryRefiner-0.5B-v0.1-SFT
```

krogoldAI commited on Oct 13, 2025

Commit

794d193

verified ·

1 Parent(s): 1d68c82

Update README.md

Browse files

Files changed (1) hide show

README.md +1 -9

README.md CHANGED Viewed

@@ -164,8 +164,6 @@ The model underwent full fine-tuning (not parameter-efficient methods like LoRA)
 The training code can be found [here](Code/Fine-tuning.py).
-<!-- This training configuration was chosen to balance thorough adaptation to the structured output task while preserving the base model's language understanding capabilities. The full fine-tuning approach, rather than parameter-efficient alternatives, ensures the model fully internalizes the strict XML formatting requirements and the nuanced query analysis patterns present in the training data. -->
 ## Model Capabilities
 The model employs a systematic, ontology-inspired analysis framework that decomposes queries into structured XML representations. This analytical approach (which identifies domains, intents, concepts, relations, and ambiguities) aims to enhance the quality of the rephrased query output. For every query, it provides domain classification and intent detection with confidence scores that sum to `1.0`, properly handling both unambiguous cases (single candidate with confidence `1.0`) and ambiguous cases (multiple candidates with distributed confidence). The model can optionally extract explicit and implicit concepts, identify relations between entities using subject-predicate-object triples, and normalize ambiguous terms when disambiguation would improve retrieval.
@@ -319,7 +317,7 @@ messages = [
 input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
 inputs = tokenizer(input_text, return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=512)
 analysis = tokenizer.decode(outputs[0], skip_special_tokens=True)
 print(analysis)
@@ -327,8 +325,6 @@ print(analysis)
 ## Evaluation and Performance
-<!-- The model was trained on data that passed stringent quality thresholds, with all training examples achieving perfect structural conformance and semantic quality scores of 4 or higher across six evaluation dimensions: domain accuracy, intent accuracy, ambiguity assessment, rephrasing quality, intent preservation, and guideline adherence. This ensures the model learned from consistently high-quality demonstrations of the desired behavior. -->
 ### Structural Validation
 To assess the model's ability to produce correctly formatted outputs, we evaluated structural conformance across 1,000 examples from the test split of [krogoldAI/rag-ambiguous-queries](https://huggingface.co/datasets/krogoldAI/rag-ambiguous-queries), comparing both QueryRefiner-0.5B-v0.1 models against their base model [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct). The evaluation measured adherence to the required XML schema, including tag presence, well-formedness, element ordering, and confidence score formatting.
@@ -345,8 +341,6 @@ To assess the model's ability to produce correctly formatted outputs, we evaluat
 Here, *tag structure* verifies that all required XML tags are present, *XML validity* ensures the output is well-formed and parseable, *order* confirms that required tags appear in the correct sequence, and *confidence* validates that confidence values are properly formatted and sum to `1.0`.
-<!-- The results demonstrate that fine-tuning substantially improved structural reliability, with QueryRefiner achieving near-perfect conformance (99.6%) compared to the base model's inability to produce any fully compliant outputs. -->
 ### Semantic Validation
 Beyond structural correctness, we evaluated the semantic quality of the model's outputs using an LLM-as-a-judge protocol with GPT5 on 1,000 examples from the test split of [krogoldAI/rag-ambiguous-queries](https://huggingface.co/datasets/krogoldAI/rag-ambiguous-queries). Each output was assessed across six dimensions aligned with the model's core objectives.
@@ -365,8 +359,6 @@ Beyond structural correctness, we evaluated the semantic quality of the model's
 All values are reported as mean ± standard deviation (%), computed over test examples. The base model produced too few valid XML samples for meaningful semantic evaluation.
-<!-- The model demonstrates strong performance across all semantic dimensions, with particularly high scores in domain classification, intent detection, and ambiguity assessment. The relatively higher variance in rephrasing quality and intent preservation reflects the inherent complexity and subjectivity of these tasks, where optimal outputs may vary based on retrieval context and domain-specific requirements. -->
 <details>
   <summary><i>Unroll to see the system prompt used for the "judge" LLM</i></summary>

 The training code can be found [here](Code/Fine-tuning.py).
 ## Model Capabilities
 The model employs a systematic, ontology-inspired analysis framework that decomposes queries into structured XML representations. This analytical approach (which identifies domains, intents, concepts, relations, and ambiguities) aims to enhance the quality of the rephrased query output. For every query, it provides domain classification and intent detection with confidence scores that sum to `1.0`, properly handling both unambiguous cases (single candidate with confidence `1.0`) and ambiguous cases (multiple candidates with distributed confidence). The model can optionally extract explicit and implicit concepts, identify relations between entities using subject-predicate-object triples, and normalize ambiguous terms when disambiguation would improve retrieval.
 input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
 inputs = tokenizer(input_text, return_tensors="pt")
+outputs = model.generate(**inputs, temperature=0.7, max_new_tokens=512)
 analysis = tokenizer.decode(outputs[0], skip_special_tokens=True)
 print(analysis)
 ## Evaluation and Performance
 ### Structural Validation
 To assess the model's ability to produce correctly formatted outputs, we evaluated structural conformance across 1,000 examples from the test split of [krogoldAI/rag-ambiguous-queries](https://huggingface.co/datasets/krogoldAI/rag-ambiguous-queries), comparing both QueryRefiner-0.5B-v0.1 models against their base model [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct). The evaluation measured adherence to the required XML schema, including tag presence, well-formedness, element ordering, and confidence score formatting.
 Here, *tag structure* verifies that all required XML tags are present, *XML validity* ensures the output is well-formed and parseable, *order* confirms that required tags appear in the correct sequence, and *confidence* validates that confidence values are properly formatted and sum to `1.0`.
 ### Semantic Validation
 Beyond structural correctness, we evaluated the semantic quality of the model's outputs using an LLM-as-a-judge protocol with GPT5 on 1,000 examples from the test split of [krogoldAI/rag-ambiguous-queries](https://huggingface.co/datasets/krogoldAI/rag-ambiguous-queries). Each output was assessed across six dimensions aligned with the model's core objectives.
 All values are reported as mean ± standard deviation (%), computed over test examples. The base model produced too few valid XML samples for meaningful semantic evaluation.
 <details>
   <summary><i>Unroll to see the system prompt used for the "judge" LLM</i></summary>