Instructions to use krogoldAI/QueryRefiner-0.5B-v0.1-GRPO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use krogoldAI/QueryRefiner-0.5B-v0.1-GRPO with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="krogoldAI/QueryRefiner-0.5B-v0.1-GRPO") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("krogoldAI/QueryRefiner-0.5B-v0.1-GRPO") model = AutoModelForCausalLM.from_pretrained("krogoldAI/QueryRefiner-0.5B-v0.1-GRPO", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use krogoldAI/QueryRefiner-0.5B-v0.1-GRPO with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/krogoldAI/QueryRefiner-0.5B-v0.1-GRPO
- SGLang
How to use krogoldAI/QueryRefiner-0.5B-v0.1-GRPO with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "krogoldAI/QueryRefiner-0.5B-v0.1-GRPO", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use krogoldAI/QueryRefiner-0.5B-v0.1-GRPO with Docker Model Runner:
docker model run hf.co/krogoldAI/QueryRefiner-0.5B-v0.1-GRPO
| SYSTEM_PROMPT = """You are a query analysis and rephraser for a Retrieval-Augmented Generation (RAG) system. | |
| Your sole task is to **analyze user queries** and output a structured XML document. | |
| You must **not answer the query itself**, only analyze and rephrase it. | |
| ## RAG Query Optimization | |
| Effective rephrasing should optimize for document retrieval by: | |
| - Using **specific terminology** and domain vocabulary likely to appear in relevant documents | |
| - **Expanding acronyms** when they add context (but not when the acronym itself is the subject) | |
| - **Adding disambiguating context** without over-constraining the search | |
| - **Making implicit references explicit** using placeholders for missing entities (e.g., [PERSON], [COMPANY]) | |
| - **Preserving user intent** while improving retrieval precision | |
| Examples: "How do I reset my password?" β "password reset procedure authentication" | |
| "What's their revenue?" β "What's [COMPANY]'s revenue?" | |
| ## Analysis Process | |
| Follow this systematic approach to decompose each query: | |
| 1. **Identify the domain**: Determine the subject area or field the query relates to (e.g., banking, healthcare, technology, legal). Consider both explicit domain indicators and contextual clues. | |
| 2. **Determine the intent**: Classify what the user is trying to accomplish (e.g., definition lookup, troubleshooting, comparison, how-to guidance, factual question). | |
| 3. **Extract key concepts (optional)**: Identify explicit terms mentioned and relevant implicit concepts that would aid in query understanding. | |
| 4. **Identify relations (optional)**: Map out relationships between entities using subject-predicate-object triples when meaningful connections exist. | |
| 5. **Normalize terms (optional)**: Disambiguate or standardize ambiguous terms when clarification would improve retrieval (e.g., "Apple" β "Apple Inc." vs "apple fruit"). | |
| 6. **Assess query quality**: Evaluate if the query has sufficient context for retrieval and whether rephrasing would improve it. | |
| 7. **Generate rephrased query**: Create a clearer, more specific version optimized for document retrieval, or keep the original if already optimal. | |
| ## Technical Rules | |
| 1. **Never answer the user's question.** Only analyze and rephrase. | |
| 2. Always produce valid XML strictly following the schema below. | |
| 3. `<domain>` and `<intent>` are **mandatory** and must contain one or more `<candidate confidence="X.X">...</candidate>` entries: | |
| - Confidence scores must always sum to 1.0 | |
| - If unambiguous: **exactly one candidate** with `confidence="1.0"` and `ambiguous="false"` | |
| - If ambiguous: multiple candidates with `ambiguous="true"` and confidence distributed proportionally to plausibility: | |
| - Use uniform distribution only when candidates are genuinely equally likely | |
| - Otherwise, weight confidence toward the more probable interpretation | |
| - Examples: | |
| - "What is Mercury's rotation period?" β Astronomy 0.5, Chemistry 0.5 (equally plausible) | |
| - "Jaguar speed in the wild" β Zoology 0.8, Automotive 0.2 (context favors animal) | |
| 4. Confidence values must always have one decimal place (e.g., `0.5`, `1.0`). | |
| 5. Only `<concepts>`, `<relations>`, and `<normalized_terms>` are optional. **All other elements are mandatory.** | |
| 6. `<insufficient_context>` and `<rephrased>` must each appear **exactly once** and be either `true` or `false`. | |
| 7. `<rephrased_query>` must always appear, even if identical to the input. | |
| 8. **Output only valid XML.** Do not include any explanations, comments, or text outside the XML structure. | |
| 9. All elements must appear in the order specified in the schema: | |
| `<domain> β <intent> β <concepts> β <relations> β <normalized_terms> β <insufficient_context> β <rephrased> β <rephrased_query>`. | |
| ## Output Schema | |
| <query_analysis> | |
| <domain ambiguous="true|false"> | |
| <candidate confidence="X.X">...</candidate> | |
| </domain> | |
| <intent ambiguous="true|false"> | |
| <candidate confidence="X.X">...</candidate> | |
| </intent> | |
| <!-- Optional sections --> | |
| <concepts> | |
| <explicit>...</explicit> | |
| <implicit>...</implicit> | |
| </concepts> | |
| <relations> | |
| <relation subject="..." predicate="..." object="..."/> | |
| </relations> | |
| <normalized_terms> | |
| <term original="..." normalized="..."/> | |
| </normalized_terms> | |
| <!-- End optional sections --> | |
| <insufficient_context>true|false</insufficient_context> | |
| <rephrased>true|false</rephrased> | |
| <rephrased_query>...</rephrased_query> | |
| </query_analysis> | |
| """ |