aelgendy commited on
Commit
6aaa61d
·
1 Parent(s): c1c14bb

Strictly ground scripture quotes in indexed retrieved context and format Quran citations with canonical URLs

Browse files
Files changed (3) hide show
  1. app/prompts.py +66 -61
  2. app/search.py +12 -3
  3. app/state.py +84 -35
app/prompts.py CHANGED
@@ -13,43 +13,45 @@ from app.config import cfg
13
  PERSONA = (
14
  "You are Sheikh QModel, a meticulous Islamic scholar with expertise "
15
  "in Quran, Tafsir (Quranic exegesis), Hadith sciences, and Arabic. "
16
- "You respond with scholarly rigor and modern clarity. "
17
- "IMPORTANT: You answer EXCLUSIVELY from the ISLAMIC CONTEXT block provided. "
18
- "You NEVER add facts, verses, hadiths, or explanations from your training knowledge "
19
- "that are not present in the context. When the context lacks information, you say so explicitly."
 
 
20
  )
21
 
22
  TASK_INSTRUCTIONS: Dict[str, str] = {
23
  "tafsir": (
24
  "The user asks about a Quranic verse — by partial text, topic, or meaning. Steps:\n"
25
- "1. Identify the matching verse(s) from the RETRIEVED RESULTS.\n"
26
- "2. Quote the Arabic verse text EXACTLY from the results.\n"
27
- "3. Provide the full reference using ONLY the [REF] metadata from the results:\n"
28
- " Surah name (Arabic & English), Surah number, and Ayah number.\n"
29
- " CRITICAL: You MUST copy the Surah name AND Ayah number from the [REF] line.\n"
30
- " NEVER guess or recall a reference from memory — use ONLY what appears in the results.\n"
31
- "4. Provide the English translation EXACTLY as given in the results.\n"
32
- "5. If the user searched by partial text, confirm the full verse found.\n"
33
- "6. Provide Tafsir: if a 'Tafsir:' field appears in the results for this verse, "
34
- "quote and build upon it as your primary source. "
35
- "If no Tafsir field is present, explain based on the verse text and translation alone.\n"
36
- "7. If related verses appear in the results, draw connections.\n"
37
- "8. Answer the user's specific question directly.\n"
38
- "9. Do NOT reference verses that are not in the results.\n"
39
- "10. If you cannot find a matching verse in the results, say so clearly."
 
 
 
40
  ),
41
  "hadith": (
42
  "The user asks about a Hadith — by partial text, topic, or meaning. Steps:\n"
43
- "1. Find the best matching Hadith from the RETRIEVED RESULTS.\n"
44
- "2. Quote the Hadith text EXACTLY — both Arabic and English from the results.\n"
45
- " IMPORTANT: Highlight the chain of narrators (Isnad) using bold text for narrator markers "
46
- "(e.g., **حَدَّثَنَا**, **عَنْ**, **Narrated**, **on the authority of**).\n"
47
- "3. State the full reference: collection name, book/chapter, hadith number.\n"
48
- "4. State the grade/authenticity (Sahih, Hasan, Da'if) if available in the results.\n"
49
- "5. If the user searched by partial text, present the complete hadith found.\n"
50
- "6. Explain the meaning, context, and scholarly implications.\n"
51
- "7. Note any related Hadiths from the results.\n"
52
- "CRITICAL: If the Hadith is NOT in the results, say so clearly — do NOT fabricate."
53
  ),
54
  "auth": (
55
  "The user asks about Hadith authenticity or grade. YOU MUST:\n"
@@ -89,23 +91,21 @@ TASK_INSTRUCTIONS: Dict[str, str] = {
89
  ),
90
  "general": (
91
  "The user may be asking a general Islamic question OR pasting text to look up. Steps:\n"
92
- "1. Check ALL retrieved results from EVERY source (Quran AND Hadith).\n"
93
- "2. For EACH result that matches the user's text or question, state WHERE it appears:\n"
94
- " • For Quran: Surah name (Arabic & English), Surah number, and Ayah number from the [REF] line.\n"
95
- " • For Hadith: collection name, book/chapter, hadith number, and grade.\n"
96
- "3. If the same text appears in MULTIPLE places, list ALL of them explicitly.\n"
97
- " Example: ‘هذا النص ذُكِر في سورة البقرة (آية ٢٥٥) وآل عمران (آية ٢)’\n"
98
- "4. Quote the Arabic text and English translation EXACTLY from the results.\n"
99
- "5. Provide brief context or explanation of the text.\n"
100
- "6. Answer the user's specific question if one was asked.\n"
101
- "CRITICAL: Do NOT give a generic answer. Always mention the exact sources from the results."
102
  ),
103
  "fatwa": (
104
  "The user asks for a juristic ruling (Fatwa). Steps:\n"
105
  "1. Identify the relevant Fatwa ruling(s) from the RETRIEVED RESULTS.\n"
106
  "2. State the Scholar or Fatwa Council name (e.g. الإمام ابن باز, مركز الفتوى - إسلام ويب, الشيخ عبد الرزاق البدر) PROMINENTLY at the start.\n"
107
  "3. Provide the exact Fatwa title and summary of the Question and Answer from the results.\n"
108
- "4. Provide the exact Source URL from the [FATWA_REF] line if available.\n"
109
  "5. If juristic conditions or exceptions apply, explain them based ONLY on the retrieved text.\n"
110
  "6. If NO 'Fatwa Ruling' item appears in the RETRIEVED RESULTS, say clearly that no specific fatwa was found, "
111
  "present only the relevant Quran/Hadith evidence that was retrieved, and advise consulting a qualified scholar. "
@@ -115,32 +115,37 @@ TASK_INSTRUCTIONS: Dict[str, str] = {
115
  }
116
 
117
  FORMAT_RULES = """\
118
- For each Quranic verse or Hadith you cite as supporting evidence, use this exact format:
119
-
120
- ┌─────────────────────────────────────────────┐
121
- │ ❝ {Arabic text} ❞
122
- │ 📝 Translation: {English translation}
123
- │ 📖 Source: {exact citation from context}
124
- └─────────────────────────────────────────────┘
125
-
126
- ABSOLUTE RULES:
127
- • Use ONLY content from the Islamic Context block. Zero outside knowledge.
128
- • Copy Arabic text and translations VERBATIM from context. Never paraphrase.
129
- • REFERENCE RULE (CRITICAL): For Quran verses, ALWAYS copy the Surah name and Ayah number
130
- from the [REF] line in the context. NEVER recall or guess references from memory.
131
- Wrong references are worse than no references.
132
- • If a specific Hadith/verse is NOT in context → respond with:
 
 
 
 
 
 
 
133
  "هذا الحديث/الآية غير موجود في قاعدة البيانات." (Arabic)
134
  or "This Hadith/verse is not in the available dataset." (English)
135
- • Never invent or guess content.
136
- • Never attribute a verse to a Surah unless the [REF] metadata explicitly says so.
137
 
138
  LANGUAGE RULE (CRITICAL — MUST FOLLOW):
139
  • You MUST answer in the SAME language as the user's question.
140
- • Arabic question → answer ENTIRELY in Arabic (العربية الفصحى). No English except inside evidence boxes.
141
- • English question → answer ENTIRELY in English. No Arabic except inside evidence boxes.
142
- • Mixed question → answer primarily in Arabic with English transliterations where helpful.
143
- • The evidence boxes always show both Arabic text and English translation regardless of language.
144
 
145
  • DIRECT ANSWER RULE: Provide the final scholarly answer directly without outputting internal thoughts, scratchpads, or <think> tags.
146
 
 
13
  PERSONA = (
14
  "You are Sheikh QModel, a meticulous Islamic scholar with expertise "
15
  "in Quran, Tafsir (Quranic exegesis), Hadith sciences, and Arabic. "
16
+ "You respond with scholarly rigor and modern clarity.\n"
17
+ "CRITICAL MANDATORY INSTRUCTION: You answer EXCLUSIVELY and STRICTLY from the ISLAMIC CONTEXT block provided.\n"
18
+ "You must NEVER EVER generate, recall, recite, or hallucinate Quran verses, Hadith texts, or Fatwas "
19
+ "from your own internal weights or training memory.\n"
20
+ "Every single piece of scripture, Hadith, or Fatwa cited MUST be retrieved and copied VERBATIM "
21
+ "from the indexed data in the context block. When information is not in the context, you state so explicitly."
22
  )
23
 
24
  TASK_INSTRUCTIONS: Dict[str, str] = {
25
  "tafsir": (
26
  "The user asks about a Quranic verse — by partial text, topic, or meaning. Steps:\n"
27
+ "1. Identify the matching verse from the RETRIEVED RESULTS in the ISLAMIC CONTEXT.\n"
28
+ "2. If the user's question is in Arabic (or searches by verse text), format your response EXACTLY as follows:\n"
29
+ " هذه الآية الكريمة هي الآية رقم {verse_number} من سورة {surah_name_ar} في القرآن الكريم: [[1]({surah_url})]\n"
30
+ " ﴿{Arabic verse text copied VERBATIM from context}﴾ [[1]({tafsir_url})]\n"
31
+ " [ثم أضف شرحاً وتفسيراً للآية مع بيان معانيها المستفادة بناءً على التفسير المرفق ومعنى الآية]\n\n"
32
+ " Where {surah_url} and {tafsir_url} are taken directly from the [REF] line for this verse:\n"
33
+ " Surah URL: https://surahquran.com/aya-{verse_number}-sora-{surah_number}.html\n"
34
+ " Tafsir URL: https://quran.ksu.edu.sa/tafseer/qortobi/sura{surah_number}-aya{verse_number}.html\n\n"
35
+ "3. If the user's question is in English, format your response as follows:\n"
36
+ " This holy verse is Ayah {verse_number} of Surah {surah_name_ar} ({surah_name_en}) in the Holy Quran: [[1]({surah_url})]\n"
37
+ " ﴿{Arabic verse text copied VERBATIM from context}﴾ [[1]({tafsir_url})]\n"
38
+ " \"{English translation copied VERBATIM from context}\"\n"
39
+ " [Then add Tafsir and scholarly explanation of the verse meanings]\n\n"
40
+ "4. ABSOLUTE ZERO-HALLUCINATION RULE: NEVER recite, recall, or generate a verse from memory.\n"
41
+ " Every letter and word of the Arabic verse MUST be copied VERBATIM from the context results.\n"
42
+ "5. Provide Tafsir / Explanation: explain the verse's meanings, context, and guidance using the Tafsir provided in the context.\n"
43
+ " If no Tafsir field is present in the context, explain based on the authentic verse text and its meaning.\n"
44
+ "6. If you cannot find a matching verse in the results, say clearly: 'لم أجد هذه الآية في قاعدة البيانات المتاحة.'"
45
  ),
46
  "hadith": (
47
  "The user asks about a Hadith — by partial text, topic, or meaning. Steps:\n"
48
+ "1. Find the best matching Hadith from the RETRIEVED RESULTS in the ISLAMIC CONTEXT.\n"
49
+ "2. Quote the Hadith text EXACTLY — both Arabic and English VERBATIM from the results.\n"
50
+ " IMPORTANT: Highlight narrator markers (e.g., **حَدَّثَنَا**, **عَنْ**, **Narrated**, **on the authority of**) in bold.\n"
51
+ "3. State the full reference: collection name, book/chapter, hadith number, and grade from the results.\n"
52
+ "4. If the user searched by partial text, present the complete Hadith found in the results.\n"
53
+ "5. Explain the meaning, context, and scholarly implications.\n"
54
+ "CRITICAL: If the Hadith is NOT in the results, say so clearly — do NOT generate or fabricate from memory."
 
 
 
55
  ),
56
  "auth": (
57
  "The user asks about Hadith authenticity or grade. YOU MUST:\n"
 
91
  ),
92
  "general": (
93
  "The user may be asking a general Islamic question OR pasting text to look up. Steps:\n"
94
+ "1. Check ALL retrieved results from EVERY source (Quran, Hadith, Fatwa) in the ISLAMIC CONTEXT.\n"
95
+ "2. For Quranic verses found, format as:\n"
96
+ " هذه الآية الكريمة هي الآية رقم {verse_number} من سورة {surah_name_ar} في القرآن الكريم: [[1]({surah_url})]\n"
97
+ " ﴿{Arabic text copied VERBATIM from context}﴾ [[1]({tafsir_url})]\n"
98
+ "3. For Hadith found: state collection name, hadith number, chapter, and grade, with narrator chain highlighted.\n"
99
+ "4. For Fatwa found: state scholar/council name, title, and link [[1]({source_url})].\n"
100
+ "5. NEVER generate or guess verses, hadiths, or fatwas from memory.\n"
101
+ "6. Provide context or explanation based strictly on the retrieved results."
 
 
102
  ),
103
  "fatwa": (
104
  "The user asks for a juristic ruling (Fatwa). Steps:\n"
105
  "1. Identify the relevant Fatwa ruling(s) from the RETRIEVED RESULTS.\n"
106
  "2. State the Scholar or Fatwa Council name (e.g. الإمام ابن باز, مركز الفتوى - إسلام ويب, الشيخ عبد الرزاق البدر) PROMINENTLY at the start.\n"
107
  "3. Provide the exact Fatwa title and summary of the Question and Answer from the results.\n"
108
+ "4. Provide the exact Source URL from the [FATWA_REF] line as a Markdown link: [[1]({source_url})].\n"
109
  "5. If juristic conditions or exceptions apply, explain them based ONLY on the retrieved text.\n"
110
  "6. If NO 'Fatwa Ruling' item appears in the RETRIEVED RESULTS, say clearly that no specific fatwa was found, "
111
  "present only the relevant Quran/Hadith evidence that was retrieved, and advise consulting a qualified scholar. "
 
115
  }
116
 
117
  FORMAT_RULES = """\
118
+ CITATION FORMAT RULES:
119
+ 1. For Quranic verses:
120
+ - Arabic response:
121
+ هذه الآية الكريمة هي الآية رقم {verse_number} من سورة {surah_name_ar} في القرآن الكريم: [[1]({surah_url})]
122
+ ﴿{Arabic text copied VERBATIM from context}﴾ [[1]({tafsir_url})]
123
+ - English response:
124
+ This holy verse is Ayah {verse_number} of Surah {surah_name_ar} ({surah_name_en}) in the Holy Quran: [[1]({surah_url})]
125
+ ﴿{Arabic text copied VERBATIM from context}﴾ [[1]({tafsir_url})]
126
+ "{English translation copied VERBATIM from context}"
127
+
128
+ 2. For Hadiths:
129
+ - Highlight narrator markers in bold (e.g. **عَنْ**, **حَدَّثَنَا**, **Narrated**).
130
+ - State the full reference (Collection, Book, Number, Grade).
131
+
132
+ 3. For Fatwas:
133
+ - State the scholar or council name, fatwa title, answer summary, and source link [[1]({source_url})].
134
+
135
+ ABSOLUTE ZERO-HALLUCINATION RULES (CRITICAL):
136
+ • NEVER generate, recite, or recall Quran verses, Hadiths, or Fatwas from your own model weights or training memory.
137
+ • Every single word of sacred scripture or fatwa MUST come strictly and verbatim from the ISLAMIC CONTEXT block.
138
+ • Under NO circumstances should you quote or invent a verse, hadith, or fatwa that is not present in the context.
139
+ • If the queried verse or hadith is not found in the context, state clearly:
140
  "هذا الحديث/الآية غير موجود في قاعدة البيانات." (Arabic)
141
  or "This Hadith/verse is not in the available dataset." (English)
142
+ • Do NOT substitute another verse or hadith from memory.
143
+ • Copy references, Surah names, Ayah numbers, and URLs EXACTLY from the [REF] / [FATWA_REF] metadata.
144
 
145
  LANGUAGE RULE (CRITICAL — MUST FOLLOW):
146
  • You MUST answer in the SAME language as the user's question.
147
+ • Arabic question → answer ENTIRELY in Arabic (العربية الفصحى).
148
+ • English question → answer ENTIRELY in English.
 
 
149
 
150
  • DIRECT ANSWER RULE: Provide the final scholarly answer directly without outputting internal thoughts, scratchpads, or <think> tags.
151
 
app/search.py CHANGED
@@ -1313,17 +1313,26 @@ def build_context(results: list) -> str:
1313
 
1314
  # Add explicit structured metadata for Quran verses
1315
  if doc_type == "quran":
 
 
 
 
1316
  block += (
1317
  f"\n [REF] Surah: {r.get('surah_name_ar', '')} ({r.get('surah_name_en', '')}) "
1318
- f"| Surah Number: {r.get('surah_number', '')} "
1319
- f"| Ayah: {r.get('verse_number', '')}"
 
 
1320
  )
1321
  # Neighboring verses context if present
1322
  if r.get("context_ayat"):
1323
  for ctx in r["context_ayat"]:
1324
  pos = "Preceding" if ctx["position"] == "previous" else "Following"
1325
  block += f"\n [{pos} Ayah {ctx['ayah']}]: {ctx['arabic']}"
1326
- if r.get("tafsir_en"):
 
 
 
1327
  max_chars = getattr(cfg, "TAFSIR_MAX_CHARS", 1200)
1328
  block += f"\n Tafsir : {r['tafsir_en'][:max_chars]}"
1329
  elif doc_type == "fatwa":
 
1313
 
1314
  # Add explicit structured metadata for Quran verses
1315
  if doc_type == "quran":
1316
+ sn = r.get("surah_number", "")
1317
+ vn = r.get("verse_number", "")
1318
+ surah_url = f"https://surahquran.com/aya-{vn}-sora-{sn}.html" if sn and vn else ""
1319
+ tafsir_url = f"https://quran.ksu.edu.sa/tafseer/qortobi/sura{sn}-aya{vn}.html" if sn and vn else ""
1320
  block += (
1321
  f"\n [REF] Surah: {r.get('surah_name_ar', '')} ({r.get('surah_name_en', '')}) "
1322
+ f"| Surah Number: {sn} "
1323
+ f"| Ayah: {vn}\n"
1324
+ f" Surah URL : {surah_url}\n"
1325
+ f" Tafsir URL: {tafsir_url}"
1326
  )
1327
  # Neighboring verses context if present
1328
  if r.get("context_ayat"):
1329
  for ctx in r["context_ayat"]:
1330
  pos = "Preceding" if ctx["position"] == "previous" else "Following"
1331
  block += f"\n [{pos} Ayah {ctx['ayah']}]: {ctx['arabic']}"
1332
+ if r.get("tafsir_ar"):
1333
+ max_chars = getattr(cfg, "TAFSIR_MAX_CHARS", 1200)
1334
+ block += f"\n Tafsir : {r['tafsir_ar'][:max_chars]}"
1335
+ elif r.get("tafsir_en"):
1336
  max_chars = getattr(cfg, "TAFSIR_MAX_CHARS", 1200)
1337
  block += f"\n Tafsir : {r['tafsir_en'][:max_chars]}"
1338
  elif doc_type == "fatwa":
app/state.py CHANGED
@@ -109,12 +109,24 @@ def _results_fallback(results: list, lang: str) -> str:
109
  surah_en = r.get("surah_name_en", "")
110
  surah_num = r.get("surah_number", "")
111
  verse_num = r.get("verse_number", "")
112
- lines.append("")
113
- lines.append("┌─────────────────────────────────────────────┐")
114
- lines.append(f"│ ❝ {ar_text} ❞")
115
- lines.append(f"│ 📝 Translation: {en_text}")
116
- lines.append(f"│ 📖 Source: سورة {surah_ar} ({surah_en}) | رقم السورة: {surah_num} | الآية: {verse_num}")
117
- lines.append("└─────────────────────────────────────────────┘")
 
 
 
 
 
 
 
 
 
 
 
 
118
 
119
  # --- Hadith results ---
120
  if hadith_hits:
@@ -179,6 +191,10 @@ _SURAH_REF_AR = re.compile(
179
  r"(?:سورة|سوره)\s+([\u0600-\u06FF\u0750-\u077F\s]+?)[\s,،]*"
180
  r"(?:الآية|آية|الايه|ايه)?\s*(\d+)",
181
  )
 
 
 
 
182
  _SURAH_REF_EN = re.compile(
183
  r"(?:surah|sura)\s+([A-Za-z\-' ]+?)[\s,]*"
184
  r"(?:ayah|verse|ayat)?\s*(\d+)",
@@ -266,6 +282,29 @@ def _verify_references(answer: str, results: list) -> str:
266
 
267
  answer = _SURAH_REF_AR.sub(_check_ar_ref, answer)
268
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
269
  # Check English surah name references
270
  def _check_en_ref(m: re.Match) -> str:
271
  raw_name = m.group(1).strip()
@@ -289,6 +328,27 @@ def _verify_references(answer: str, results: list) -> str:
289
 
290
  answer = _SURAH_REF_EN.sub(_check_en_ref, answer)
291
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
292
  return answer
293
 
294
 
@@ -303,7 +363,7 @@ def _verify_citations(answer: str, results: list) -> str:
303
 
304
  source_texts_raw = []
305
  for r in results:
306
- for field in ("arabic", "english", "text", "tafsir", "tafsir_en", "explanation"):
307
  val = r.get(field, "")
308
  if val:
309
  source_texts_raw.append(re.sub(r"\s+", " ", str(val).strip()))
@@ -335,7 +395,7 @@ def _verify_citations(answer: str, results: list) -> str:
335
 
336
  q_words = set(quoted_norm.split())
337
 
338
- # 1. Check in retrieved search results
339
  for src_raw, src_norm, s_words in source_entries:
340
  # Exact or normalized substring match
341
  if _contains(quoted_raw, src_raw) or _contains(quoted_norm, src_norm):
@@ -344,34 +404,23 @@ def _verify_citations(answer: str, results: list) -> str:
344
  if q_words and len(q_words & s_words) / len(q_words) >= 0.4:
345
  return m.group(0)
346
 
347
- # 2. Check full Quran (pre-normalized at startup) so authentic verses aren't flagged
348
- for verse_norm, v_words in state.quran_norm_texts:
349
- if _contains(quoted_norm, verse_norm):
350
- return m.group(0)
351
- if q_words and len(q_words & v_words) / len(q_words) >= 0.5:
352
- return m.group(0)
 
 
 
 
 
 
 
 
 
353
 
354
- # 3. Check full dataset via BM25 (query tokenized like the index was built)
355
- if state.dataset and state.bm25_index:
356
- try:
357
- terms = tokenize_bilingual(quoted_raw)[:20]
358
- top_hits = state.bm25_index.search(terms, top_k=5) if terms else []
359
- for doc_id, _score in top_hits:
360
- if doc_id < len(state.dataset):
361
- doc = state.dataset[doc_id]
362
- doc_ar = normalize_arabic(doc.get("arabic", "") or doc.get("text", ""))
363
- if doc_ar:
364
- if _contains(quoted_norm, doc_ar):
365
- return m.group(0)
366
- d_words = set(doc_ar.split())
367
- if q_words and len(q_words & d_words) / len(q_words) >= 0.5:
368
- return m.group(0)
369
- except Exception as exc:
370
- logger.debug("BM25 citation lookup failed: %s", exc)
371
-
372
- # Quote not found in any source or dataset — flag it
373
- logger.warning("Hallucination detected: quoted text not in sources: %.80s...", quoted_norm)
374
- return "❝ ⚠️ [تم حذف نص غير موثق — النص غير موجود في قاعدة البيانات] ❞"
375
 
376
  return _QUOTE_RE.sub(_check_quote, answer)
377
 
 
109
  surah_en = r.get("surah_name_en", "")
110
  surah_num = r.get("surah_number", "")
111
  verse_num = r.get("verse_number", "")
112
+ surah_url = f"https://surahquran.com/aya-{verse_num}-sora-{surah_num}.html" if surah_num and verse_num else ""
113
+ tafsir_url = f"https://quran.ksu.edu.sa/tafseer/qortobi/sura{surah_num}-aya{verse_num}.html" if surah_num and verse_num else ""
114
+ surah_link = f"[[1]({surah_url})]" if surah_url else ""
115
+ tafsir_link = f"[[1]({tafsir_url})]" if tafsir_url else ""
116
+
117
+ if lang == "arabic":
118
+ lines.append(f"\nهذه الآية الكريمة هي الآية رقم {verse_num} من سورة {surah_ar} في القرآن الكريم: {surah_link}")
119
+ lines.append(f"﴿{ar_text}﴾ {tafsir_link}")
120
+ if r.get("tafsir_ar"):
121
+ lines.append(f"\nالتفسير:\n{r['tafsir_ar']}")
122
+ elif en_text:
123
+ lines.append(f"\nالترجمة الإنجليزية:\n{en_text}")
124
+ else:
125
+ lines.append(f"\nThis holy verse is Ayah {verse_num} of Surah {surah_en} ({surah_ar}) in the Holy Quran: {surah_link}")
126
+ lines.append(f"﴿{ar_text}﴾ {tafsir_link}")
127
+ lines.append(f'"{en_text}"')
128
+ if r.get("tafsir_en"):
129
+ lines.append(f"\nTafsir:\n{r['tafsir_en']}")
130
 
131
  # --- Hadith results ---
132
  if hadith_hits:
 
191
  r"(?:سورة|سوره)\s+([\u0600-\u06FF\u0750-\u077F\s]+?)[\s,،]*"
192
  r"(?:الآية|آية|الايه|ايه)?\s*(\d+)",
193
  )
194
+ _AYAH_FIRST_REF_AR = re.compile(
195
+ r"(?:الآية|آية|الايه|ايه)\s*(?:رقم\s*)?(\d+)\s+من\s+(?:سورة|سوره)\s+([\u0600-\u06FF\u0750-\u077F\s]+?)(?=[\s,،\.\:\;]|$)",
196
+ re.I,
197
+ )
198
  _SURAH_REF_EN = re.compile(
199
  r"(?:surah|sura)\s+([A-Za-z\-' ]+?)[\s,]*"
200
  r"(?:ayah|verse|ayat)?\s*(\d+)",
 
282
 
283
  answer = _SURAH_REF_AR.sub(_check_ar_ref, answer)
284
 
285
+ # Check Arabic ayah-first references like "الآية رقم 101 من سورة الأعراف"
286
+ def _check_ayah_first_ref(m: re.Match) -> str:
287
+ verse_num = int(m.group(1))
288
+ raw_name = m.group(2).strip()
289
+ name_norm = re.sub(r"^(ال)", "", raw_name).strip().lower()
290
+
291
+ matched_sn = name_map.get(name_norm)
292
+ if matched_sn and (matched_sn, verse_num) in valid_refs:
293
+ return m.group(0)
294
+
295
+ for (sn, vn) in valid_refs:
296
+ for r in results:
297
+ if r.get("type") == "quran" and int(r.get("surah_number", 0)) == sn and int(r.get("verse_number", 0)) == vn:
298
+ correct_name = r.get("surah_name_ar", "")
299
+ logger.warning(
300
+ "Hallucinated reference: آية %d من سورة %s -> correcting to الآية رقم %d من سورة %s",
301
+ verse_num, raw_name, vn, correct_name,
302
+ )
303
+ return f"الآية رقم {vn} من سورة {correct_name}"
304
+ return m.group(0)
305
+
306
+ answer = _AYAH_FIRST_REF_AR.sub(_check_ayah_first_ref, answer)
307
+
308
  # Check English surah name references
309
  def _check_en_ref(m: re.Match) -> str:
310
  raw_name = m.group(1).strip()
 
328
 
329
  answer = _SURAH_REF_EN.sub(_check_en_ref, answer)
330
 
331
+ # Check and correct Quranic URLs to match authoritative retrieved reference
332
+ if len(valid_refs) == 1:
333
+ correct_sn, correct_vn = next(iter(valid_refs))
334
+
335
+ def _check_surah_url(m: re.Match) -> str:
336
+ vn, sn = int(m.group(1)), int(m.group(2))
337
+ if (sn, vn) not in valid_refs:
338
+ logger.warning("Correcting hallucinated surah URL %d:%d -> %d:%d", sn, vn, correct_sn, correct_vn)
339
+ return f"https://surahquran.com/aya-{correct_vn}-sora-{correct_sn}.html"
340
+ return m.group(0)
341
+
342
+ def _check_ksu_url(m: re.Match) -> str:
343
+ sn, vn = int(m.group(1)), int(m.group(2))
344
+ if (sn, vn) not in valid_refs:
345
+ logger.warning("Correcting hallucinated KSU URL %d:%d -> %d:%d", sn, vn, correct_sn, correct_vn)
346
+ return f"https://quran.ksu.edu.sa/tafseer/qortobi/sura{correct_sn}-aya{correct_vn}.html"
347
+ return m.group(0)
348
+
349
+ answer = re.sub(r"https://surahquran\.com/aya-(\d+)-sora-(\d+)\.html", _check_surah_url, answer)
350
+ answer = re.sub(r"https://quran\.ksu\.edu\.sa/tafseer/qortobi/sura(\d+)-aya(\d+)\.html", _check_ksu_url, answer)
351
+
352
  return answer
353
 
354
 
 
363
 
364
  source_texts_raw = []
365
  for r in results:
366
+ for field in ("arabic", "english", "text", "tafsir", "tafsir_ar", "tafsir_en", "explanation"):
367
  val = r.get(field, "")
368
  if val:
369
  source_texts_raw.append(re.sub(r"\s+", " ", str(val).strip()))
 
395
 
396
  q_words = set(quoted_norm.split())
397
 
398
+ # 1. Check in retrieved search results (strictly grounded in retrieved indexed context)
399
  for src_raw, src_norm, s_words in source_entries:
400
  # Exact or normalized substring match
401
  if _contains(quoted_raw, src_raw) or _contains(quoted_norm, src_norm):
 
404
  if q_words and len(q_words & s_words) / len(q_words) >= 0.4:
405
  return m.group(0)
406
 
407
+ # Quote not found in retrieved results — LLM generated/recalled scripture from model weights
408
+ logger.warning("Hallucination detected: quoted text not in retrieved results: %.80s...", quoted_norm)
409
+
410
+ # Auto-correct to authentic indexed Quran verse if top retrieved source is a Quran verse
411
+ quran_sources = [r for r in results if r.get("type") == "quran" and r.get("arabic")]
412
+ if quran_sources and (len(quran_sources) == 1 or quran_sources[0].get("_score", 0) >= 2.0):
413
+ correct_ar = quran_sources[0]["arabic"]
414
+ logger.info("Auto-correcting hallucinated scripture quote to indexed Quran verse %s", quran_sources[0].get("id"))
415
+ return f"﴿{correct_ar}﴾"
416
+
417
+ hadith_sources = [r for r in results if r.get("type") == "hadith" and r.get("arabic")]
418
+ if hadith_sources and len(hadith_sources) == 1:
419
+ correct_ar = hadith_sources[0]["arabic"]
420
+ logger.info("Auto-correcting hallucinated quote to indexed Hadith %s", hadith_sources[0].get("id"))
421
+ return f"❝ {correct_ar} ❞"
422
 
423
+ return "❝ ⚠️ [تم حذف نص غير موثق — النص غير موجود في البيانات المسترجعة] ❞"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
424
 
425
  return _QUOTE_RE.sub(_check_quote, answer)
426