Turkish
durak / docs /BEST_PRACTICES.md
Taklaxbr's picture
Archive open-source github repository (cdliai/durak)
7c16940 verified
|
Raw History Blame Contribute Delete
9.96 kB
# Best Practices Guide
This guide provides recommendations for using and extending Durak effectively.
## 🎯 General Principles
### 1. Resource Management
**✅ DO**: Use embedded Rust resources for production
```python
from durak import _durak_core
# Fast, no file I/O
stopwords = set(_durak_core.get_stopwords_base())
suffixes = _durak_core.get_detached_suffixes()
```
**❌ DON'T**: Load from files in hot paths
```python
# Slower, file I/O overhead
from durak import load_stopword_resource
stopwords = load_stopword_resource("base/turkish") # OK for development, not production loops
```
**When to use file-based loading:**
- Development and debugging
- Custom resource files not yet embedded
- Dynamic resource updates
- Testing with modified resources
### 2. Performance Optimization
**✅ DO**: Use Rust functions directly for performance-critical code
```python
from durak import _durak_core
# 5-10x faster
normalized = _durak_core.fast_normalize(text)
tokens = _durak_core.tokenize_with_offsets(text)
```
**✅ DO**: Batch process when possible
```python
from durak import _durak_core
# Process all texts before filtering
normalized_texts = [_durak_core.fast_normalize(t) for t in texts]
tokenized = [_durak_core.tokenize_with_offsets(t) for t in normalized_texts]
```
**❌ DON'T**: Mix Python/Rust unnecessarily in loops
```python
# Inefficient: context switching overhead
for text in texts:
cleaned = durak.clean_text(text) # Python
normalized = _durak_core.fast_normalize(cleaned) # Rust
# Better: do all cleaning in one layer, then all normalization
```
### 3. Pipeline Design
**✅ DO**: Create reusable pipeline objects
```python
from durak import StopwordManager, Pipeline, Normalizer
# Create once, reuse many times
stopword_mgr = StopwordManager(additions=["custom"], keep=["important"])
normalizer = Normalizer()
pipeline = Pipeline([
normalizer,
lambda tokens: [t for t in tokens if not stopword_mgr.is_stopword(t)]
])
# Reuse efficiently
results = [pipeline(text) for text in texts]
```
**❌ DON'T**: Recreate expensive objects in loops
```python
# Bad: recreates manager every iteration
for text in texts:
mgr = StopwordManager(additions=["custom"]) # Expensive!
tokens = process_text(text)
filtered = [t for t in tokens if not mgr.is_stopword(t)]
```
## 📝 Configuration Management
### Externalizing Hard-Coded Values
**✅ DO**: Use resource files for configuration
```python
# resources/tr/config/custom_apostrophes.txt
# resources/tr/config/domain_stopwords.txt
from pathlib import Path
config_dir = Path("resources/tr/config")
apostrophes = (config_dir / "custom_apostrophes.txt").read_text().strip().split("\n")
```
**Best practice for new configurations:**
1. Add resource file under `resources/tr/config/`
2. Embed in Rust if needed frequently (see next section)
3. Load once at module initialization
4. Cache in module-level variable
### Adding New Embedded Resources
**To add new resources to Rust binary:**
1. **Add file** to `resources/tr/`
2. **Embed in Rust** (`src/lib.rs`):
```rust
static MY_DATA: &str = include_str!("../resources/tr/config/mydata.txt");
```
3. **Create accessor function**:
```rust
#[pyfunction]
fn get_my_data() -> Vec<String> {
MY_DATA.lines().map(|s| s.trim().to_string()).collect()
}
```
4. **Export in module**:
```rust
m.add_function(wrap_pyfunction!(get_my_data, m)?)?;
```
5. **Add type stub** (`python/durak/_durak_core.pyi`):
```python
def get_my_data() -> list[str]: ...
```
6. **Rebuild**: `maturin develop --release`
## 🔧 Extension Patterns
### Custom Stopword Management
**✅ DO**: Use domain-specific stopword files
```python
# resources/tr/stopwords/domains/medical.txt
# resources/tr/stopwords/domains/legal.txt
manager = StopwordManager.from_resources([
"base/turkish",
"domains/medical", # Domain-specific
], keep=["hasta", "doktor"]) # Keep domain terms
```
**✅ DO**: Export and version stopword sets
```python
manager = StopwordManager(additions=["term1", "term2"])
manager.export("my_project_stopwords_v1.txt")
# Later, load exact same set
from durak import load_stopwords
stopwords = load_stopwords("my_project_stopwords_v1.txt")
```
### Morphological Analysis
**Current limitation**: Basic suffix stripping only
**Recommended approach for production:**
```python
# For simple cases: use built-in
from durak import _durak_core
stripped = _durak_core.strip_suffixes("kitaplardan") # "kitap"
# For complex morphology: use external tools
# Option 1: Zemberek (Java-based, most accurate for Turkish)
# Option 2: spaCy tr_core_news_lg
# Option 3: Stanza Turkish model
# Then integrate with Durak pipeline:
from durak import Pipeline
pipeline = Pipeline([
lambda text: durak.clean_text(text),
lambda text: durak.tokenize(text),
lambda tokens: your_lemmatizer.lemmatize(tokens), # External tool
lambda tokens: durak.remove_stopwords(tokens),
])
```
### Suffix Configuration
**Current state**: Rust suffixes are hard-coded for demo purposes
**To customize suffix stripping:**
1. **Edit** `resources/tr/config/lemma_suffixes.txt`
2. **Load in Python**:
```python
from pathlib import Path
suffix_file = Path("resources/tr/config/lemma_suffixes.txt")
custom_suffixes = [
line.strip()
for line in suffix_file.read_text().splitlines()
if line.strip() and not line.startswith("#")
]
# Use with Python-based stripping
def strip_custom(word: str, suffixes: list[str]) -> str:
for suffix in sorted(suffixes, key=len, reverse=True):
if word.endswith(suffix) and len(word) > len(suffix) + 2:
return word[:-len(suffix)]
return word
lemma = strip_custom("kitaplardan", custom_suffixes)
```
3. **For production**: Implement proper morphological analysis with vowel harmony
## 🧪 Testing Best Practices
### Unit Tests
**✅ DO**: Test with realistic Turkish text
```python
def test_preserves_diacritics():
text = "İstanbul'dan Türkiye'ye"
tokens = durak.tokenize(text)
assert "İstanbul'dan" in tokens # Preserves İ
assert "Türkiye'ye" in tokens # Preserves ü
```
**✅ DO**: Test edge cases
```python
def test_apostrophe_variants():
# Both straight and curly apostrophes
assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"]
assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"] # Curly
```
### Performance Testing
**✅ DO**: Benchmark critical paths
```python
import time
def benchmark(func, *args, iterations=1000):
start = time.perf_counter()
for _ in range(iterations):
func(*args)
return (time.perf_counter() - start) / iterations * 1000
# Measure
py_time = benchmark(durak.normalize_case, text)
rust_time = benchmark(_durak_core.fast_normalize, text)
print(f"Speedup: {py_time / rust_time:.2f}x")
```
## 🚀 Deployment
### Production Checklist
- [ ] Use `maturin build --release` for optimized builds
- [ ] Verify embedded resources with `_durak_core.get_*()` functions
- [ ] Pre-load stopword managers at application startup
- [ ] Use Rust functions for hot paths
- [ ] Enable logging for resource loading issues
- [ ] Version-lock dependencies in production
- [ ] Test with production-like data volumes
### Distribution
**For PyPI releases:**
```bash
# Build optimized wheel
maturin build --release
# All resources are embedded in the wheel
# No need to ship separate data files
pip install durak-nlp # Just works!
```
**For custom deployments:**
```bash
# Development mode (editable install)
maturin develop
# Production mode (optimized)
maturin develop --release
```
## 📊 Monitoring
### Resource Loading
```python
import logging
logging.basicConfig(level=logging.INFO)
# Monitor resource loading
try:
from durak import _durak_core
stopwords = _durak_core.get_stopwords_base()
logging.info(f"Loaded {len(stopwords)} stopwords from embedded resources")
except ImportError:
logging.warning("Rust extension not available, using file-based loading")
stopwords = durak.load_stopword_resource("base/turkish")
```
### Performance Metrics
```python
import time
from functools import wraps
def log_performance(func):
@wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = (time.perf_counter() - start) * 1000
logging.info(f"{func.__name__}: {elapsed:.2f}ms")
return result
return wrapper
@log_performance
def process_corpus(texts):
return [durak.process_text(t, remove_stopwords=True) for t in texts]
```
## 🔐 Security
### Input Validation
**✅ DO**: Validate external resources
```python
from pathlib import Path
def load_safe_stopwords(path: Path | str) -> set[str]:
path = Path(path).resolve()
# Ensure path is within allowed directory
if not str(path).startswith(str(Path("resources/tr").resolve())):
raise ValueError(f"Path outside resources directory: {path}")
if not path.exists():
raise FileNotFoundError(f"Resource not found: {path}")
return durak.load_stopwords(path)
```
**✅ DO**: Sanitize user input before processing
```python
def process_user_text(text: str) -> list[str]:
# Limit length
if len(text) > 1_000_000:
raise ValueError("Text too long")
# Basic sanitization
text = text.strip()
# Process safely
return durak.process_text(text, remove_stopwords=True)
```
## Further Reading
- [ARCHITECTURE.md](ARCHITECTURE.md) - System design and component architecture
- [examples/](../examples/) - Working code examples
- [benchmarks/](../benchmarks/) - Performance measurement tools
- [CONTRIBUTING.md](../CONTRIBUTING.md) - How to contribute improvements
---
**Questions or suggestions?** Open an issue at [github.com/fbkaragoz/durak/issues](https://github.com/fbkaragoz/durak/issues)