Upload BENCHMARK_REPORT.md with huggingface_hub

BENCHMARK_REPORT.md

@@ -0,0 +1,179 @@
+# Comprehensive Test & Benchmark Report
+
+**Package:** deeplatent-nlp v0.3.8
+**Date:** 2026-02-04
+**Tokenizer:** almaghrabima/SARFTokenizer (from HuggingFace)
+**Rust Available:** Yes
+
+---
+
+## Executive Summary
+
+All tests passed with **100% roundtrip accuracy** on 100,000 samples and **58/58 edge case tests** passed. SARFTokenizer shows excellent parallel scaling, outperforming tiktoken at higher thread counts.
+
+---
+
+## 1. Roundtrip Accuracy Tests
+
+### 100,000 Sample Test Results
+
+| Category | Samples | Success | Failed | Accuracy |
+|----------|---------|---------|--------|----------|
+| Arabic | 33,333 | 33,333 | 0 | **100.00%** |
+| English | 33,333 | 33,333 | 0 | **100.00%** |
+| Mixed | 33,333 | 33,333 | 0 | **100.00%** |
+| **TOTAL** | **99,999** | **99,999** | **0** | **100.00%** |
+
+### Timing Breakdown
+- Arabic encode: 4.55s (7,329 texts/sec)
+- English encode: 1.18s (28,294 texts/sec)
+- Mixed encode: 10.80s (3,086 texts/sec)
+
+---
+
+## 2. Edge Case Tests (12 Categories)
+
+All **58 tests passed** across 12 edge case categories:
+
+| Category | Tests | Passed | Failed |
+|----------|-------|--------|--------|
+| Unicode Normalization | 6 | 6 | 0 |
+| Zero-Width Characters | 6 | 6 | 0 |
+| Unicode Whitespace | 6 | 6 | 0 |
+| Grapheme Clusters | 6 | 6 | 0 |
+| Apostrophes | 4 | 4 | 0 |
+| Dashes | 4 | 4 | 0 |
+| Decimal Separators | 3 | 3 | 0 |
+| URLs/Emails | 4 | 4 | 0 |
+| File Paths | 3 | 3 | 0 |
+| Code Identifiers | 4 | 4 | 0 |
+| Mixed Scripts/RTL | 6 | 6 | 0 |
+| Robustness | 6 | 6 | 0 |
+| **TOTAL** | **58** | **58** | **0** |
+
+### pytest Results
+- `test_edge_cases.py`: **107 tests passed**
+- `test_roundtrip.py`: **62 tests passed**
+
+---
+
+## 3. Performance Benchmarks
+
+### Single-Threaded Performance
+
+| Mode | Throughput | MB/s |
+|------|------------|------|
+| Single encode | 4,522 texts/sec | 1.9 MB/s |
+| Batch (1K) | 20,929 texts/sec | 8.7 MB/s |
+| Batch (10K) | 29,796 texts/sec | 12.4 MB/s |
+
+### Parallel Benchmark Comparison
+
+| Tokenizer | Vocab | 1 Thread | 2 Threads | 4 Threads | 8 Threads | Peak Memory |
+|-----------|-------|----------|-----------|-----------|-----------|-------------|
+| **SARFTokenizer (HF)** | 64,641 | 8,010/s | 41,282/s | 39,911/s | **43,148/s** | 0.2 MB |
+| SARFTokenizer (Local) | 64,641 | 7,714/s | 41,155/s | 42,578/s | 41,036/s | 0.2 MB |
+| tiktoken (o200k) | 200,019 | 21,733/s | 18,489/s | 20,683/s | 11,939/s | 0.2 MB |
+| tiktoken (cl100k) | 100,277 | 29,908/s | 19,036/s | 15,785/s | 11,093/s | 0.4 MB |
+
+### Thread Scaling Efficiency
+
+| Tokenizer | 2T Speedup | 4T Speedup | 8T Speedup |
+|-----------|------------|------------|------------|
+| **SARFTokenizer (HF)** | **5.15x** | **4.98x** | **5.39x** |
+| SARFTokenizer (Local) | 5.33x | 5.52x | 5.32x |
+| tiktoken (o200k) | 0.89x | 0.97x | 0.57x |
+| tiktoken (cl100k) | 0.65x | 0.66x | 0.41x |
+
+### Key Findings
+
+1. **SARFTokenizer outperforms tiktoken at higher thread counts**
+   - At 8 threads: SARFTokenizer (43,148/s) vs tiktoken cl100k (11,093/s) = **3.9x faster**
+
+2. **Excellent parallel scaling**
+   - SARFTokenizer achieves 5x+ speedup with parallel processing
+   - tiktoken shows negative scaling (slower with more threads)
+
+3. **Memory efficient**
+   - Peak memory usage: 0.2-0.4 MB for all tokenizers
+
+---
+
+## 4. Memory Usage
+
+| Metric | Value |
+|--------|-------|
+| Peak Memory | 0.28 MB |
+| Samples Processed | 10,000 |
+| Memory per Sample | ~0.03 KB |
+
+---
+
+## 5. PyPI vs Local Tokenizer Comparison
+
+| Metric | PyPI (HF) | Local |
+|--------|-----------|-------|
+| Vocab Size | 64,641 | 64,641 |
+| 1T Throughput | 8,010/s | 7,714/s |
+| 8T Throughput | 43,148/s | 41,036/s |
+| Memory | 0.2 MB | 0.2 MB |
+
+**Conclusion:** Both produce identical results. PyPI version is slightly faster.
+
+---
+
+## 6. Test Data Source
+
+- **Location:** `/root/.cache/deeplatent/base_data/`
+- **Format:** Parquet shards (748 files)
+- **Total Samples Available:** ~150M texts
+- **Languages:** Arabic, English, Mixed
+
+---
+
+## 7. Scripts Created
+
+| Script | Purpose |
+|--------|---------|
+| `test_comprehensive_million.py` | Million-scale roundtrip + edge case tests |
+| `benchmark_parallel.py` | Parallel benchmark comparison with tiktoken |
+
+### Usage
+
+```bash
+# Quick test (3,000 samples)
+python test_comprehensive_million.py --samples 3000
+
+# Full test (100,000 samples)
+python test_comprehensive_million.py --samples 100000 --report
+
+# Million-scale test
+python test_comprehensive_million.py --samples 1000000 --report
+
+# Parallel benchmark
+python benchmark_parallel.py --samples 10000 --threads 1 2 4 8
+
+# Compare specific tokenizers
+python benchmark_parallel.py --tokenizers sarf tiktoken
+```
+
+---
+
+## 8. Conclusion
+
+The SARFTokenizer from `almaghrabima/SARFTokenizer` on HuggingFace demonstrates:
+
+1. **Perfect roundtrip accuracy** (100% on 100K samples)
+2. **Robust edge case handling** (58/58 tests passed)
+3. **Superior parallel performance** (5x speedup, outperforms tiktoken at scale)
+4. **Minimal memory footprint** (0.2 MB peak)
+
+The tokenizer is production-ready for Arabic and multilingual NLP applications.
+
+---
+
+## Appendix: Raw JSON Results
+
+Results saved to:
+- `test_comprehensive_results.json`
+- `benchmark_parallel_results.json`