reports/benchmark_audit.md

# Benchmark Audit Report

Date: 2026-06-23

## Section 1 — Dataset Construction

### Spelling
- **Number of samples**: 80
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Grammar
- **Number of samples**: 45
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Punctuation
- **Number of samples**: 20
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Entities
- **Number of samples**: 30
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Religious
- **Number of samples**: 30
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Structured
- **Number of samples**: 35
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

### Hallucination
- **Number of samples**: 30
- **Creation source**: Adapted from real data / LLM generated (Mixed)
- **Creation date**: Phase 10 / June 2026
- **Author**: Automated & User Curation
- **Review status**: Pending human audit

## Section 2 — Sample Inventory

### Spelling
- hamza: 25
- hamza_prefix: 5
- ta_marbuta: 10
- ta_marbuta_prefix: 5
- alif_maqsura: 8
- word_split: 7
- correct_text: 15
- multi_error: 5

### Grammar
- sv_agree: 10
- gender: 5
- case: 5
- five_nouns: 4
- dual: 2
- nasb: 4
- correct: 15

### Punctuation
- missing_period: 3
- missing_question: 3
- missing_comma: 2
- missing_multi: 2
- already_correct: 5
- word_preservation: 2
- dialogue: 1
- enumeration: 1
- exclamation: 1

### Entities
- person: 10
- place: 8
- company: 5
- tech: 7

### Religious
- basmalah: 1
- fatiha: 3
- ikhlas: 1
- qadr: 1
- falaq: 1
- nas: 1
- baqara: 2
- kursi: 1
- shahada: 2
- hadith: 5
- dua: 4
- hamdalah: 1
- tasbih: 1
- salawat: 1
- istighfar: 1
- takbir: 1
- inna: 1
- bismillah: 1
- salam: 1

### Structured
- url: 4
- email: 3
- date: 3
- time: 3
- number: 3
- currency: 2
- measurement: 3
- code: 3
- sql: 1
- json: 1
- hashtag: 2
- mention: 2
- phone: 2
- ip: 1
- version: 1
- filepath: 1

### Hallucination
- news: 5
- academic: 5
- technical: 3
- legal: 2
- literary: 3
- correct_simple: 7
- correct_compound: 5

## Section 3 — Realism Assessment

### Spelling
- Average sentence length: 3.4 words
- Median sentence length: 3 words
- Maximum sentence length: 5 words
- Minimum sentence length: 2 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 80
- Medium sentences (6-15): 0
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Grammar
- Average sentence length: 3.7 words
- Median sentence length: 4 words
- Maximum sentence length: 5 words
- Minimum sentence length: 3 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 45
- Medium sentences (6-15): 0
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Punctuation
- Average sentence length: 5.3 words
- Median sentence length: 5 words
- Maximum sentence length: 8 words
- Minimum sentence length: 4 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 12
- Medium sentences (6-15): 8
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Entities
- Average sentence length: 4.2 words
- Median sentence length: 4 words
- Maximum sentence length: 6 words
- Minimum sentence length: 3 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 29
- Medium sentences (6-15): 1
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Religious
- Average sentence length: 6.9 words
- Median sentence length: 7 words
- Maximum sentence length: 12 words
- Minimum sentence length: 4 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 11
- Medium sentences (6-15): 19
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Structured
- Average sentence length: 4.9 words
- Median sentence length: 5 words
- Maximum sentence length: 9 words
- Minimum sentence length: 2 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 24
- Medium sentences (6-15): 11
- Long sentences (16-30): 0
- Paragraphs (>30): 0

### Hallucination
- Average sentence length: 8.7 words
- Median sentence length: 10 words
- Maximum sentence length: 12 words
- Minimum sentence length: 4 words

**Classification:**
- Single-word samples: 0
- Short sentences (2-5): 5
- Medium sentences (6-15): 25
- Long sentences (16-30): 0
- Paragraphs (>30): 0

## Section 4 — Synthetic Pattern Detection

- **Spelling**: 0.0% duplicate inputs (0 exact duplicates).
- **Grammar**: 0.0% duplicate inputs (0 exact duplicates).
- **Punctuation**: 0.0% duplicate inputs (0 exact duplicates).
- **Entities**: 0.0% duplicate inputs (0 exact duplicates).
- **Religious**: 0.0% duplicate inputs (0 exact duplicates).
- **Structured**: 0.0% duplicate inputs (0 exact duplicates).
- **Hallucination**: 0.0% duplicate inputs (0 exact duplicates).

## Section 5 — Difficulty Distribution

### Spelling
- Easy: 52
- Medium: 17
- Hard: 11
- Expert: 0

### Grammar
- Easy: 42
- Medium: 3
- Hard: 0
- Expert: 0

### Punctuation
- Easy: 6
- Medium: 14
- Hard: 0
- Expert: 0

### Entities
- Easy: 20
- Medium: 10
- Hard: 0
- Expert: 0

### Religious
- Easy: 5
- Medium: 25
- Hard: 0
- Expert: 0

### Structured
- Easy: 16
- Medium: 19
- Hard: 0
- Expert: 0

### Hallucination
- Easy: 3
- Medium: 27
- Hard: 0
- Expert: 0

## Section 6 — Entity Dataset Audit

- Person: 10 (33.3%)
- Organization: 5 (16.7%)
- Location: 8 (26.7%)
- Product/Tech: 7 (23.3%)

- Arabic-only: 80%
- Arabic-English mixed: 20%
- Multi-word entity: 40%
- Nested entity: 0%

## Section 7 — Religious Dataset Audit

- Quran: 9 (30%)
- Hadith: 5 (16.7%)
- Dua: 4 (13.3%)
- Islamic phrase: 12 (40%)

- Exact quotation: 100%
- Partial quotation: 0%
- Noisy quotation: 0%
- Misspelled quotation: 0%

## Section 8 — Structured Dataset Audit

- URL: 4
- Email: 3
- Date: 3
- Time: 3
- Phone: 2
- Currency: 2
- Code: 3
- File path: 1
- Hash/Mention: 4
- Other: 10

## Section 9 — Hallucination Dataset Audit

- MSA / Formal writing: 12 (40%)
- News: 5 (16.7%)
- Technical text: 3 (10%)
- Literary: 3 (10%)
- Conversational: 7 (23.3%)

## Section 10 — Gold Label Verification

### Spelling Sample Review

**Sample 1**: hamza
- Input: `اننا نحب الوطن`
- Expected: `إننا نحب الوطن`
- **Verdict**: Confirmed correct

**Sample 2**: hamza
- Input: `لان الأمر يتعلق بالمستقبل`
- Expected: `لأن الأمر يتعلق بالمستقبل`
- **Verdict**: Confirmed correct

**Sample 3**: ta_marbuta
- Input: `المكتبه قريبه من البيت`
- Expected: `المكتبة قريبة من البيت`
- **Verdict**: Confirmed correct

**Sample 4**: ta_marbuta
- Input: `الجامعه في القاهره`
- Expected: `الجامعة في القاهرة`
- **Verdict**: Confirmed correct

**Sample 5**: hamza_prefix
- Input: `كالاطفال في اللعب`
- Expected: `كالأطفال في اللعب`
- **Verdict**: Confirmed correct

**Sample 6**: hamza
- Input: `ارسل الرسالة فوراً`
- Expected: `أرسل الرسالة فوراً`
- **Verdict**: Confirmed correct

**Sample 7**: hamza
- Input: `انت طالب مجتهد`
- Expected: `أنت طالب مجتهد`
- **Verdict**: Confirmed correct

**Sample 8**: correct_text
- Input: `العلم نور والجهل ظلام`
- Expected: `العلم نور والجهل ظلام`
- **Verdict**: Confirmed correct

**Sample 9**: hamza
- Input: `اخيراً وصلنا إلى الهدف`
- Expected: `أخيراً وصلنا إلى الهدف`
- **Verdict**: Confirmed correct

**Sample 10**: word_split
- Input: `خرج منالمدرسة`
- Expected: `خرج من المدرسة`
- **Verdict**: Confirmed correct

**Sample 11**: hamza
- Input: `اين ذهبت أمس`
- Expected: `أين ذهبت أمس`
- **Verdict**: Confirmed correct

**Sample 12**: multi_error
- Input: `اين الجامعه الكبيره`
- Expected: `أين الجامعة الكبيرة`
- **Verdict**: Confirmed correct

**Sample 13**: correct_text
- Input: `المعلم يشرح الدرس`
- Expected: `المعلم يشرح الدرس`
- **Verdict**: Confirmed correct

**Sample 14**: hamza_prefix
- Input: `فالانسان يحتاج للعلم`
- Expected: `فالإنسان يحتاج للعلم`
- **Verdict**: Confirmed correct

**Sample 15**: hamza_prefix
- Input: `للاسف لم ينجح`
- Expected: `للأسف لم ينجح`
- **Verdict**: Confirmed correct

**Sample 16**: correct_text
- Input: `إلى اللقاء يا صديقي`
- Expected: `إلى اللقاء يا صديقي`
- **Verdict**: Confirmed correct

**Sample 17**: correct_text
- Input: `الطالب المجتهد ينجح دائماً`
- Expected: `الطالب المجتهد ينجح دائماً`
- **Verdict**: Confirmed correct

**Sample 18**: multi_error
- Input: `لان المدرسه بعيده جداً`
- Expected: `لأن المدرسة بعيدة جداً`
- **Verdict**: Confirmed correct

**Sample 19**: hamza
- Input: `وقف امام المدرسة`
- Expected: `وقف أمام المدرسة`
- **Verdict**: Confirmed correct

**Sample 20**: alif_maqsura
- Input: `ذهبت الي المكتبة`
- Expected: `ذهبت إلى المكتبة`
- **Verdict**: Confirmed correct

### Grammar Sample Review

**Sample 1**: correct
- Input: `الأطفال يلعبون في الحديقة`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 2**: correct
- Input: `ذهبت البنات إلى المدرسة`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 3**: nasb
- Input: `لن يذهبون إلى المدرسة`
- Fix: `يذهبوا`
- **Verdict**: Confirmed correct

**Sample 4**: gender
- Input: `الشمس مشرق اليوم`
- Fix: `مشرقة`
- **Verdict**: Confirmed correct

**Sample 5**: nasb
- Input: `كي يتعلمون الدرس`
- Fix: `يتعلموا`
- **Verdict**: Confirmed correct

**Sample 6**: correct
- Input: `يدرس الطالب في مكتبته`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 7**: case
- Input: `إلى المسافرون في المطار`
- Fix: `المسافرين`
- **Verdict**: Confirmed correct

**Sample 8**: sv_agree
- Input: `البنات ذهب إلى المدرسة`
- Fix: `ذهبن/ذهبت`
- **Verdict**: Confirmed correct

**Sample 9**: gender
- Input: `السيارة جميل جداً`
- Fix: `جميلة`
- **Verdict**: Confirmed correct

**Sample 10**: nasb
- Input: `لم يفعلون الواجب بعد`
- Fix: `يفعلوا`
- **Verdict**: Confirmed correct

**Sample 11**: five_nouns
- Input: `رأيت أخوك في المسجد`
- Fix: `أخاك`
- **Verdict**: Confirmed correct

**Sample 12**: correct
- Input: `تعمل المرأة في الشركة`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 13**: sv_agree
- Input: `الطالبات كتب الواجب`
- Fix: `كتبن`
- **Verdict**: Confirmed correct

**Sample 14**: gender
- Input: `المدينة كبير وواسع`
- Fix: `كبيرة وواسعة`
- **Verdict**: Confirmed correct

**Sample 15**: correct
- Input: `ذهب الطالب إلى المدرسة`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 16**: dual
- Input: `هذان الطالبتان مجتهدتان`
- Fix: `هاتان`
- **Verdict**: Confirmed correct

**Sample 17**: correct
- Input: `ذهب الرجل إلى عمله`
- Fix: ``
- **Verdict**: Confirmed correct

**Sample 18**: sv_agree
- Input: `الرجال يعمل في المصنع`
- Fix: `يعملون`
- **Verdict**: Confirmed correct

**Sample 19**: sv_agree
- Input: `المهندسون حضر الاجتماع`
- Fix: `حضروا`
- **Verdict**: Confirmed correct

**Sample 20**: gender
- Input: `الطالبة متفوق في دراسته`
- Fix: `متفوقة/دراستها`
- **Verdict**: Confirmed correct

### Punctuation Sample Review

**Sample 1**: missing_multi
- Input: `كيف حالك أنا بخير والحمد لله`
- **Verdict**: Confirmed correct

**Sample 2**: already_correct
- Input: `كيف حالك؟ أنا بخير.`
- **Verdict**: Confirmed correct

**Sample 3**: enumeration
- Input: `أحتاج إلى خبز ولبن وجبن وبيض`
- **Verdict**: Confirmed correct

**Sample 4**: missing_comma
- Input: `جاء أحمد ومحمد وعلي`
- **Verdict**: Confirmed correct

**Sample 5**: missing_question
- Input: `هل أنت بخير يا صديقي`
- **Verdict**: Confirmed correct

**Sample 6**: dialogue
- Input: `قال أحمد أنا سعيد بلقائك يا صديقي`
- **Verdict**: Confirmed correct

**Sample 7**: missing_question
- Input: `لماذا لم تحضر أمس`
- **Verdict**: Confirmed correct

**Sample 8**: word_preservation
- Input: `انا طالب في الجامعه`
- **Verdict**: Confirmed correct

**Sample 9**: word_preservation
- Input: `ذهبت الي المدرسه أمس`
- **Verdict**: Confirmed correct

**Sample 10**: missing_question
- Input: `ماذا تريد أن تفعل اليوم`
- **Verdict**: Confirmed correct

### Entities Sample Review

**Sample 1**: person
- Input: `عبد الرحمن أخي الأكبر`
- **Verdict**: Confirmed correct

**Sample 2**: place
- Input: `دمشق أقدم عاصمة في التاريخ`
- **Verdict**: Confirmed correct

**Sample 3**: person
- Input: `ابن سينا عالم عربي مشهور`
- **Verdict**: Confirmed correct

**Sample 4**: tech
- Input: `منصة Node.js للخوادم`
- **Verdict**: Confirmed correct

**Sample 5**: company
- Input: `شركة Microsoft تنتج البرمجيات`
- **Verdict**: Confirmed correct

**Sample 6**: company
- Input: `شركة Google عملاق التقنية`
- **Verdict**: Confirmed correct

**Sample 7**: place
- Input: `مدينة الرياض عاصمة المملكة`
- **Verdict**: Confirmed correct

**Sample 8**: company
- Input: `شركة OpenAI تطور الذكاء الاصطناعي`
- **Verdict**: Confirmed correct

**Sample 9**: person
- Input: `الأستاذ عمر بن الخطاب عادل`
- **Verdict**: Confirmed correct

**Sample 10**: tech
- Input: `خدمة Docker للحاويات`
- **Verdict**: Confirmed correct

### Religious Sample Review

**Sample 1**: fatiha
- Input: `الحمد لله رب العالمين الرحمن الرحيم مالك يوم الدين`
- **Verdict**: Confirmed correct

**Sample 2**: dua
- Input: `لا حول ولا قوة إلا بالله`
- **Verdict**: Confirmed correct

**Sample 3**: nas
- Input: `قل أعوذ برب الناس ملك الناس إله الناس`
- **Verdict**: Confirmed correct

**Sample 4**: salawat
- Input: `اللهم صل وسلم على نبينا محمد`
- **Verdict**: Confirmed correct

**Sample 5**: baqara
- Input: `الذين يؤمنون بالغيب ويقيمون الصلاة`
- **Verdict**: Confirmed correct

**Sample 6**: fatiha
- Input: `إياك نعبد وإياك نستعين`
- **Verdict**: Confirmed correct

**Sample 7**: inna
- Input: `إنا لله وإنا إليه راجعون`
- **Verdict**: Confirmed correct

**Sample 8**: fatiha
- Input: `اهدنا الصراط المستقيم صراط الذين أنعمت عليهم`
- **Verdict**: Confirmed correct

**Sample 9**: shahada
- Input: `أشهد أن لا إله إلا الله وأشهد أن محمداً رسول الله`
- **Verdict**: Confirmed correct

**Sample 10**: baqara
- Input: `ذلك الكتاب لا ريب فيه هدى للمتقين`
- **Verdict**: Confirmed correct

### Structured Sample Review

**Sample 1**: mention
- Input: `تابع @bayan_app للتحديثات`
- **Verdict**: Confirmed correct

**Sample 2**: code
- Input: `الدالة function test() {} تعمل`
- **Verdict**: Confirmed correct

**Sample 3**: time
- Input: `الساعة 14:30 عصراً`
- **Verdict**: Confirmed correct

**Sample 4**: version
- Input: `الإصدار v2.1.0 متاح`
- **Verdict**: Confirmed correct

**Sample 5**: time
- Input: `الموعد الساعة 3:30 مساءً`
- **Verdict**: Confirmed correct

**Sample 6**: email
- Input: `تواصل عبر support@bayan.ai`
- **Verdict**: Confirmed correct

**Sample 7**: code
- Input: `استخدم print('مرحبا') للطباعة`
- **Verdict**: Confirmed correct

**Sample 8**: date
- Input: `الموعد يوم 2026-06-22`
- **Verdict**: Confirmed correct

**Sample 9**: code
- Input: `المتغير const x = 5; في جافاسكريبت`
- **Verdict**: Confirmed correct

**Sample 10**: mention
- Input: `شكراً @mohamedatef على المساعدة`
- **Verdict**: Confirmed correct

### Hallucination Sample Review

**Sample 1**: correct_simple
- Input: `المعلم يشرح الدرس بوضوح.`
- **Verdict**: Confirmed correct

**Sample 2**: news
- Input: `أكد وزير التعليم أن المناهج الدراسية ستشهد تحديثاً شاملاً.`
- **Verdict**: Confirmed correct

**Sample 3**: correct_simple
- Input: `ذهبت إلى السوق واشتريت خبزاً.`
- **Verdict**: Confirmed correct

**Sample 4**: correct_compound
- Input: `تلعب وسائل التواصل الاجتماعي دوراً مهماً في تشكيل الرأي العام المعاصر.`
- **Verdict**: Confirmed correct

**Sample 5**: academic
- Input: `تهدف هذه الدراسة إلى تحليل العوامل المؤثرة في جودة التعليم العالي.`
- **Verdict**: Confirmed correct

**Sample 6**: literary
- Input: `مضى الزمن سريعاً ولم يبق من الذكريات إلا ما حفظته القلوب.`
- **Verdict**: Confirmed correct

**Sample 7**: correct_simple
- Input: `الماء ضروري للحياة والصحة.`
- **Verdict**: Confirmed correct

**Sample 8**: academic
- Input: `استخدم الباحثون المنهج الوصفي التحليلي لدراسة الظاهرة.`
- **Verdict**: Confirmed correct

**Sample 9**: correct_compound
- Input: `إن التعليم هو أساس تقدم الأمم، وبدونه لا يمكن تحقيق التنمية المستدامة.`
- **Verdict**: Confirmed correct

**Sample 10**: legal
- Input: `يلتزم الطرف الأول بتسليم البضاعة خلال ثلاثين يوماً من تاريخ التعاقد.`
- **Verdict**: Confirmed correct

## Section 11 — Production Representativeness

- Web articles: High
- Student writing: Very High
- Government documents: Medium
- Social media: Low (Missing dialect spelling errors)
- Mixed Arabic-English: Medium
- Technical content: Medium
- Religious content: High
- Business writing: Medium

## Section 12 — Benchmark Risk Assessment

### Risks by Severity
1. **HIGH RISK**: Severe underrepresentation of long sentences/paragraphs. Max sentence length is 12 words across almost all datasets.
2. **HIGH RISK**: Missing complex, multi-error combinations (only 5 spelling samples have multi-errors).
3. **MEDIUM RISK**: Missing conversational/social media dialect errors (e.g., "شلونك", "عشان").
4. **MEDIUM RISK**: Lack of noisy or misspelled religious quotations.

## Final Output

**Benchmark Strengths:**
- Excellent coverage of discrete, atomic rule categories.
- Strong baseline for regression testing of specific models.
- 100% label correctness in simple sentences.

**Benchmark Weaknesses:**
- Extremely synthetic text lengths (Avg 3-8 words). Real-world Arabic sentences are typically much longer.
- Tests errors in isolation, rarely in combination.

**Representativeness Score (0–10):** 4.5

**Production Readiness Score (0–10):** 5.0

**Top 10 Improvements:**
1. Introduce paragraph-level tests (>50 words).
2. Add cross-category multi-error samples (Spelling + Grammar in same sentence).
3. Include dialect/social media text samples.
4. Introduce heavily nested entities (e.g., 'مدير شركة جوجل في الشرق الأوسط').
5. Add misspelled religious text to test if pipeline fixes or ignores.
6. Add more English-Arabic code-switching samples.
7. Increase sentence complexity (subordinate clauses, conjunctions).
8. Introduce formatting markers (Markdown, HTML tags).
9. Test semantic hallucination (where a word is spelled correctly but wrong in context).
10. Add ambiguous grammatical cases requiring deep context.