|
Download docs/BEST_PRACTICES.md from veripazari/durak: direct link, hf CLI and curl.
- Browser
- Download file 9.96 kB
-
https://huggingface.co/veripazari/durak/resolve/main/docs/BEST_PRACTICES.md
- Command line
-
hf download hf://veripazari/durak/docs/BEST_PRACTICES.md
-
curl -L -o BEST_PRACTICES.md https://huggingface.co/veripazari/durak/resolve/main/docs/BEST_PRACTICES.md
9.96 kB
| # Best Practices Guide | |
| This guide provides recommendations for using and extending Durak effectively. | |
| ## 🎯 General Principles | |
| ### 1. Resource Management | |
| **✅ DO**: Use embedded Rust resources for production | |
| ```python | |
| from durak import _durak_core | |
| # Fast, no file I/O | |
| stopwords = set(_durak_core.get_stopwords_base()) | |
| suffixes = _durak_core.get_detached_suffixes() | |
| ``` | |
| **❌ DON'T**: Load from files in hot paths | |
| ```python | |
| # Slower, file I/O overhead | |
| from durak import load_stopword_resource | |
| stopwords = load_stopword_resource("base/turkish") # OK for development, not production loops | |
| ``` | |
| **When to use file-based loading:** | |
| - Development and debugging | |
| - Custom resource files not yet embedded | |
| - Dynamic resource updates | |
| - Testing with modified resources | |
| ### 2. Performance Optimization | |
| **✅ DO**: Use Rust functions directly for performance-critical code | |
| ```python | |
| from durak import _durak_core | |
| # 5-10x faster | |
| normalized = _durak_core.fast_normalize(text) | |
| tokens = _durak_core.tokenize_with_offsets(text) | |
| ``` | |
| **✅ DO**: Batch process when possible | |
| ```python | |
| from durak import _durak_core | |
| # Process all texts before filtering | |
| normalized_texts = [_durak_core.fast_normalize(t) for t in texts] | |
| tokenized = [_durak_core.tokenize_with_offsets(t) for t in normalized_texts] | |
| ``` | |
| **❌ DON'T**: Mix Python/Rust unnecessarily in loops | |
| ```python | |
| # Inefficient: context switching overhead | |
| for text in texts: | |
| cleaned = durak.clean_text(text) # Python | |
| normalized = _durak_core.fast_normalize(cleaned) # Rust | |
| # Better: do all cleaning in one layer, then all normalization | |
| ``` | |
| ### 3. Pipeline Design | |
| **✅ DO**: Create reusable pipeline objects | |
| ```python | |
| from durak import StopwordManager, Pipeline, Normalizer | |
| # Create once, reuse many times | |
| stopword_mgr = StopwordManager(additions=["custom"], keep=["important"]) | |
| normalizer = Normalizer() | |
| pipeline = Pipeline([ | |
| normalizer, | |
| lambda tokens: [t for t in tokens if not stopword_mgr.is_stopword(t)] | |
| ]) | |
| # Reuse efficiently | |
| results = [pipeline(text) for text in texts] | |
| ``` | |
| **❌ DON'T**: Recreate expensive objects in loops | |
| ```python | |
| # Bad: recreates manager every iteration | |
| for text in texts: | |
| mgr = StopwordManager(additions=["custom"]) # Expensive! | |
| tokens = process_text(text) | |
| filtered = [t for t in tokens if not mgr.is_stopword(t)] | |
| ``` | |
| ## 📝 Configuration Management | |
| ### Externalizing Hard-Coded Values | |
| **✅ DO**: Use resource files for configuration | |
| ```python | |
| # resources/tr/config/custom_apostrophes.txt | |
| # resources/tr/config/domain_stopwords.txt | |
| from pathlib import Path | |
| config_dir = Path("resources/tr/config") | |
| apostrophes = (config_dir / "custom_apostrophes.txt").read_text().strip().split("\n") | |
| ``` | |
| **Best practice for new configurations:** | |
| 1. Add resource file under `resources/tr/config/` | |
| 2. Embed in Rust if needed frequently (see next section) | |
| 3. Load once at module initialization | |
| 4. Cache in module-level variable | |
| ### Adding New Embedded Resources | |
| **To add new resources to Rust binary:** | |
| 1. **Add file** to `resources/tr/` | |
| 2. **Embed in Rust** (`src/lib.rs`): | |
| ```rust | |
| static MY_DATA: &str = include_str!("../resources/tr/config/mydata.txt"); | |
| ``` | |
| 3. **Create accessor function**: | |
| ```rust | |
| #[pyfunction] | |
| fn get_my_data() -> Vec<String> { | |
| MY_DATA.lines().map(|s| s.trim().to_string()).collect() | |
| } | |
| ``` | |
| 4. **Export in module**: | |
| ```rust | |
| m.add_function(wrap_pyfunction!(get_my_data, m)?)?; | |
| ``` | |
| 5. **Add type stub** (`python/durak/_durak_core.pyi`): | |
| ```python | |
| def get_my_data() -> list[str]: ... | |
| ``` | |
| 6. **Rebuild**: `maturin develop --release` | |
| ## 🔧 Extension Patterns | |
| ### Custom Stopword Management | |
| **✅ DO**: Use domain-specific stopword files | |
| ```python | |
| # resources/tr/stopwords/domains/medical.txt | |
| # resources/tr/stopwords/domains/legal.txt | |
| manager = StopwordManager.from_resources([ | |
| "base/turkish", | |
| "domains/medical", # Domain-specific | |
| ], keep=["hasta", "doktor"]) # Keep domain terms | |
| ``` | |
| **✅ DO**: Export and version stopword sets | |
| ```python | |
| manager = StopwordManager(additions=["term1", "term2"]) | |
| manager.export("my_project_stopwords_v1.txt") | |
| # Later, load exact same set | |
| from durak import load_stopwords | |
| stopwords = load_stopwords("my_project_stopwords_v1.txt") | |
| ``` | |
| ### Morphological Analysis | |
| **Current limitation**: Basic suffix stripping only | |
| **Recommended approach for production:** | |
| ```python | |
| # For simple cases: use built-in | |
| from durak import _durak_core | |
| stripped = _durak_core.strip_suffixes("kitaplardan") # "kitap" | |
| # For complex morphology: use external tools | |
| # Option 1: Zemberek (Java-based, most accurate for Turkish) | |
| # Option 2: spaCy tr_core_news_lg | |
| # Option 3: Stanza Turkish model | |
| # Then integrate with Durak pipeline: | |
| from durak import Pipeline | |
| pipeline = Pipeline([ | |
| lambda text: durak.clean_text(text), | |
| lambda text: durak.tokenize(text), | |
| lambda tokens: your_lemmatizer.lemmatize(tokens), # External tool | |
| lambda tokens: durak.remove_stopwords(tokens), | |
| ]) | |
| ``` | |
| ### Suffix Configuration | |
| **Current state**: Rust suffixes are hard-coded for demo purposes | |
| **To customize suffix stripping:** | |
| 1. **Edit** `resources/tr/config/lemma_suffixes.txt` | |
| 2. **Load in Python**: | |
| ```python | |
| from pathlib import Path | |
| suffix_file = Path("resources/tr/config/lemma_suffixes.txt") | |
| custom_suffixes = [ | |
| line.strip() | |
| for line in suffix_file.read_text().splitlines() | |
| if line.strip() and not line.startswith("#") | |
| ] | |
| # Use with Python-based stripping | |
| def strip_custom(word: str, suffixes: list[str]) -> str: | |
| for suffix in sorted(suffixes, key=len, reverse=True): | |
| if word.endswith(suffix) and len(word) > len(suffix) + 2: | |
| return word[:-len(suffix)] | |
| return word | |
| lemma = strip_custom("kitaplardan", custom_suffixes) | |
| ``` | |
| 3. **For production**: Implement proper morphological analysis with vowel harmony | |
| ## 🧪 Testing Best Practices | |
| ### Unit Tests | |
| **✅ DO**: Test with realistic Turkish text | |
| ```python | |
| def test_preserves_diacritics(): | |
| text = "İstanbul'dan Türkiye'ye" | |
| tokens = durak.tokenize(text) | |
| assert "İstanbul'dan" in tokens # Preserves İ | |
| assert "Türkiye'ye" in tokens # Preserves ü | |
| ``` | |
| **✅ DO**: Test edge cases | |
| ```python | |
| def test_apostrophe_variants(): | |
| # Both straight and curly apostrophes | |
| assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"] | |
| assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"] # Curly | |
| ``` | |
| ### Performance Testing | |
| **✅ DO**: Benchmark critical paths | |
| ```python | |
| import time | |
| def benchmark(func, *args, iterations=1000): | |
| start = time.perf_counter() | |
| for _ in range(iterations): | |
| func(*args) | |
| return (time.perf_counter() - start) / iterations * 1000 | |
| # Measure | |
| py_time = benchmark(durak.normalize_case, text) | |
| rust_time = benchmark(_durak_core.fast_normalize, text) | |
| print(f"Speedup: {py_time / rust_time:.2f}x") | |
| ``` | |
| ## 🚀 Deployment | |
| ### Production Checklist | |
| - [ ] Use `maturin build --release` for optimized builds | |
| - [ ] Verify embedded resources with `_durak_core.get_*()` functions | |
| - [ ] Pre-load stopword managers at application startup | |
| - [ ] Use Rust functions for hot paths | |
| - [ ] Enable logging for resource loading issues | |
| - [ ] Version-lock dependencies in production | |
| - [ ] Test with production-like data volumes | |
| ### Distribution | |
| **For PyPI releases:** | |
| ```bash | |
| # Build optimized wheel | |
| maturin build --release | |
| # All resources are embedded in the wheel | |
| # No need to ship separate data files | |
| pip install durak-nlp # Just works! | |
| ``` | |
| **For custom deployments:** | |
| ```bash | |
| # Development mode (editable install) | |
| maturin develop | |
| # Production mode (optimized) | |
| maturin develop --release | |
| ``` | |
| ## 📊 Monitoring | |
| ### Resource Loading | |
| ```python | |
| import logging | |
| logging.basicConfig(level=logging.INFO) | |
| # Monitor resource loading | |
| try: | |
| from durak import _durak_core | |
| stopwords = _durak_core.get_stopwords_base() | |
| logging.info(f"Loaded {len(stopwords)} stopwords from embedded resources") | |
| except ImportError: | |
| logging.warning("Rust extension not available, using file-based loading") | |
| stopwords = durak.load_stopword_resource("base/turkish") | |
| ``` | |
| ### Performance Metrics | |
| ```python | |
| import time | |
| from functools import wraps | |
| def log_performance(func): | |
| @wraps(func) | |
| def wrapper(*args, **kwargs): | |
| start = time.perf_counter() | |
| result = func(*args, **kwargs) | |
| elapsed = (time.perf_counter() - start) * 1000 | |
| logging.info(f"{func.__name__}: {elapsed:.2f}ms") | |
| return result | |
| return wrapper | |
| @log_performance | |
| def process_corpus(texts): | |
| return [durak.process_text(t, remove_stopwords=True) for t in texts] | |
| ``` | |
| ## 🔐 Security | |
| ### Input Validation | |
| **✅ DO**: Validate external resources | |
| ```python | |
| from pathlib import Path | |
| def load_safe_stopwords(path: Path | str) -> set[str]: | |
| path = Path(path).resolve() | |
| # Ensure path is within allowed directory | |
| if not str(path).startswith(str(Path("resources/tr").resolve())): | |
| raise ValueError(f"Path outside resources directory: {path}") | |
| if not path.exists(): | |
| raise FileNotFoundError(f"Resource not found: {path}") | |
| return durak.load_stopwords(path) | |
| ``` | |
| **✅ DO**: Sanitize user input before processing | |
| ```python | |
| def process_user_text(text: str) -> list[str]: | |
| # Limit length | |
| if len(text) > 1_000_000: | |
| raise ValueError("Text too long") | |
| # Basic sanitization | |
| text = text.strip() | |
| # Process safely | |
| return durak.process_text(text, remove_stopwords=True) | |
| ``` | |
| ## Further Reading | |
| - [ARCHITECTURE.md](ARCHITECTURE.md) - System design and component architecture | |
| - [examples/](../examples/) - Working code examples | |
| - [benchmarks/](../benchmarks/) - Performance measurement tools | |
| - [CONTRIBUTING.md](../CONTRIBUTING.md) - How to contribute improvements | |
| --- | |
| **Questions or suggestions?** Open an issue at [github.com/fbkaragoz/durak/issues](https://github.com/fbkaragoz/durak/issues) | |