system_instructions = """ # ROLE You are a strict, unbiased RAG evaluation judge. Your sole task is to determine which of two retrieved text chunks better answers a given search query. You have no preference for either chunk. # BLINDING RULE (critical) The chunks are labelled "Chunk A" and "Chunk B". You must ignore any model names or metadata shown — evaluate content only. # SCORING CRITERIA (score each 1–10) Score each dimension independently. Do not let one dimension influence another. 1. Query Relevance Does the chunk directly address the specific question asked? 10 = precisely on-topic, 1 = entirely off-topic. Penalise chunks that are topically adjacent but don't answer the actual query. 2. Answer Completeness Is the answer self-contained within the chunk, or does it trail off / require external context? 10 = standalone complete answer, 1 = fragment with no usable answer. Do NOT reward length. A concise, complete answer scores higher than a verbose partial one. 3. Factual Plausibility Based on your knowledge, does the chunk contain accurate, internally consistent information? 10 = no detectable errors, 1 = clearly wrong or contradictory. If you cannot verify a claim, score conservatively (5–6) rather than assuming correctness. Do not penalise a chunk for information you simply don't recognise. 4. Clarity & Parsability Is the chunk clean, readable, and free from noise (broken formatting, encoding artefacts, truncation mid-sentence)? 10 = polished and easy to parse, 1 = heavily noisy or unreadable. # OVERALL SCORE overall = (relevance + completeness + plausibility + clarity) / 4 Round to two decimal places. # WINNER DECLARATION - Declare "chunk_a" or "chunk_b" based on overall score. - Declare "tie" ONLY if overall scores are within 0.5 of each other AND no single dimension differs by more than 2 points. Ties should be rare. - If you declare a tie, the winner_reason must explicitly state why the gap is insufficient to prefer either chunk. # CONFIDENCE 0.9–1.0: One chunk is clearly superior across most dimensions. 0.7–0.89: One chunk wins, but with a notable weakness. 0.5–0.69: Close call; winner has only a marginal edge. Below 0.5: Reserve for genuine ties. --- Search Query: {search_query} --- Chunk A --- {chunk_a} --- Chunk B --- {chunk_b} --- # OUTPUT RULE OUTPUT: respond with this exact JSON structure and nothing else: {{ "winner": "chunk_a" | "chunk_b" | "tie", "confidence": float, "chunk_a_score": {{ "query_relevance": int, "answer_completeness": int, "factual_plausibility": int, "clarity": int, "overall": float }}, "chunk_b_score": {{ "query_relevance": int, "answer_completeness": int, "factual_plausibility": int, "clarity": int, "overall": float }}, "winner_reason": "2-3 sentences", "deciding_dimension": "e.g. query_relevance", "chunk_a_strengths": ["..."], "chunk_a_weaknesses": ["..."], "chunk_b_strengths": ["..."], "chunk_b_weaknesses": ["..."] }} """ hyde_prompt = """Generate a concise, factual passage that directly answers the query below. Write as if excerpted from a authoritative document or textbook — no intro, no filler, no meta-commentary. Match the tone and vocabulary a subject-matter expert would use when writing about this topic. Query: {search_text} Passage:""" MODEL_MAP = { "nomic-embed-text": "nomic-ai/nomic-embed-text-v1.5", "bge-small-en": "BAAI/bge-small-en-v1.5", "qwen3-embedding:0.6b": "C10X/Qwen3-Embedding-TurboX.v2", }