Download docs/BEST_PRACTICES.md from veripazari/durak: direct link, hf CLI and curl.
- Browser
- Download file 9.96 kB
-
https://huggingface.co/veripazari/durak/resolve/main/docs/BEST_PRACTICES.md
- Command line
-
hf download hf://veripazari/durak/docs/BEST_PRACTICES.md
-
curl -L -o BEST_PRACTICES.md https://huggingface.co/veripazari/durak/resolve/main/docs/BEST_PRACTICES.md
Best Practices Guide
This guide provides recommendations for using and extending Durak effectively.
🎯 General Principles
1. Resource Management
✅ DO: Use embedded Rust resources for production
from durak import _durak_core
# Fast, no file I/O
stopwords = set(_durak_core.get_stopwords_base())
suffixes = _durak_core.get_detached_suffixes()
❌ DON'T: Load from files in hot paths
# Slower, file I/O overhead
from durak import load_stopword_resource
stopwords = load_stopword_resource("base/turkish") # OK for development, not production loops
When to use file-based loading:
- Development and debugging
- Custom resource files not yet embedded
- Dynamic resource updates
- Testing with modified resources
2. Performance Optimization
✅ DO: Use Rust functions directly for performance-critical code
from durak import _durak_core
# 5-10x faster
normalized = _durak_core.fast_normalize(text)
tokens = _durak_core.tokenize_with_offsets(text)
✅ DO: Batch process when possible
from durak import _durak_core
# Process all texts before filtering
normalized_texts = [_durak_core.fast_normalize(t) for t in texts]
tokenized = [_durak_core.tokenize_with_offsets(t) for t in normalized_texts]
❌ DON'T: Mix Python/Rust unnecessarily in loops
# Inefficient: context switching overhead
for text in texts:
cleaned = durak.clean_text(text) # Python
normalized = _durak_core.fast_normalize(cleaned) # Rust
# Better: do all cleaning in one layer, then all normalization
3. Pipeline Design
✅ DO: Create reusable pipeline objects
from durak import StopwordManager, Pipeline, Normalizer
# Create once, reuse many times
stopword_mgr = StopwordManager(additions=["custom"], keep=["important"])
normalizer = Normalizer()
pipeline = Pipeline([
normalizer,
lambda tokens: [t for t in tokens if not stopword_mgr.is_stopword(t)]
])
# Reuse efficiently
results = [pipeline(text) for text in texts]
❌ DON'T: Recreate expensive objects in loops
# Bad: recreates manager every iteration
for text in texts:
mgr = StopwordManager(additions=["custom"]) # Expensive!
tokens = process_text(text)
filtered = [t for t in tokens if not mgr.is_stopword(t)]
📝 Configuration Management
Externalizing Hard-Coded Values
✅ DO: Use resource files for configuration
# resources/tr/config/custom_apostrophes.txt
# resources/tr/config/domain_stopwords.txt
from pathlib import Path
config_dir = Path("resources/tr/config")
apostrophes = (config_dir / "custom_apostrophes.txt").read_text().strip().split("\n")
Best practice for new configurations:
- Add resource file under
resources/tr/config/ - Embed in Rust if needed frequently (see next section)
- Load once at module initialization
- Cache in module-level variable
Adding New Embedded Resources
To add new resources to Rust binary:
Add file to
resources/tr/Embed in Rust (
src/lib.rs):static MY_DATA: &str = include_str!("../resources/tr/config/mydata.txt");Create accessor function:
#[pyfunction] fn get_my_data() -> Vec<String> { MY_DATA.lines().map(|s| s.trim().to_string()).collect() }Export in module:
m.add_function(wrap_pyfunction!(get_my_data, m)?)?;Add type stub (
python/durak/_durak_core.pyi):def get_my_data() -> list[str]: ...Rebuild:
maturin develop --release
🔧 Extension Patterns
Custom Stopword Management
✅ DO: Use domain-specific stopword files
# resources/tr/stopwords/domains/medical.txt
# resources/tr/stopwords/domains/legal.txt
manager = StopwordManager.from_resources([
"base/turkish",
"domains/medical", # Domain-specific
], keep=["hasta", "doktor"]) # Keep domain terms
✅ DO: Export and version stopword sets
manager = StopwordManager(additions=["term1", "term2"])
manager.export("my_project_stopwords_v1.txt")
# Later, load exact same set
from durak import load_stopwords
stopwords = load_stopwords("my_project_stopwords_v1.txt")
Morphological Analysis
Current limitation: Basic suffix stripping only
Recommended approach for production:
# For simple cases: use built-in
from durak import _durak_core
stripped = _durak_core.strip_suffixes("kitaplardan") # "kitap"
# For complex morphology: use external tools
# Option 1: Zemberek (Java-based, most accurate for Turkish)
# Option 2: spaCy tr_core_news_lg
# Option 3: Stanza Turkish model
# Then integrate with Durak pipeline:
from durak import Pipeline
pipeline = Pipeline([
lambda text: durak.clean_text(text),
lambda text: durak.tokenize(text),
lambda tokens: your_lemmatizer.lemmatize(tokens), # External tool
lambda tokens: durak.remove_stopwords(tokens),
])
Suffix Configuration
Current state: Rust suffixes are hard-coded for demo purposes
To customize suffix stripping:
Edit
resources/tr/config/lemma_suffixes.txtLoad in Python:
from pathlib import Path suffix_file = Path("resources/tr/config/lemma_suffixes.txt") custom_suffixes = [ line.strip() for line in suffix_file.read_text().splitlines() if line.strip() and not line.startswith("#") ] # Use with Python-based stripping def strip_custom(word: str, suffixes: list[str]) -> str: for suffix in sorted(suffixes, key=len, reverse=True): if word.endswith(suffix) and len(word) > len(suffix) + 2: return word[:-len(suffix)] return word lemma = strip_custom("kitaplardan", custom_suffixes)For production: Implement proper morphological analysis with vowel harmony
🧪 Testing Best Practices
Unit Tests
✅ DO: Test with realistic Turkish text
def test_preserves_diacritics():
text = "İstanbul'dan Türkiye'ye"
tokens = durak.tokenize(text)
assert "İstanbul'dan" in tokens # Preserves İ
assert "Türkiye'ye" in tokens # Preserves ü
✅ DO: Test edge cases
def test_apostrophe_variants():
# Both straight and curly apostrophes
assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"]
assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"] # Curly
Performance Testing
✅ DO: Benchmark critical paths
import time
def benchmark(func, *args, iterations=1000):
start = time.perf_counter()
for _ in range(iterations):
func(*args)
return (time.perf_counter() - start) / iterations * 1000
# Measure
py_time = benchmark(durak.normalize_case, text)
rust_time = benchmark(_durak_core.fast_normalize, text)
print(f"Speedup: {py_time / rust_time:.2f}x")
🚀 Deployment
Production Checklist
- Use
maturin build --releasefor optimized builds - Verify embedded resources with
_durak_core.get_*()functions - Pre-load stopword managers at application startup
- Use Rust functions for hot paths
- Enable logging for resource loading issues
- Version-lock dependencies in production
- Test with production-like data volumes
Distribution
For PyPI releases:
# Build optimized wheel
maturin build --release
# All resources are embedded in the wheel
# No need to ship separate data files
pip install durak-nlp # Just works!
For custom deployments:
# Development mode (editable install)
maturin develop
# Production mode (optimized)
maturin develop --release
📊 Monitoring
Resource Loading
import logging
logging.basicConfig(level=logging.INFO)
# Monitor resource loading
try:
from durak import _durak_core
stopwords = _durak_core.get_stopwords_base()
logging.info(f"Loaded {len(stopwords)} stopwords from embedded resources")
except ImportError:
logging.warning("Rust extension not available, using file-based loading")
stopwords = durak.load_stopword_resource("base/turkish")
Performance Metrics
import time
from functools import wraps
def log_performance(func):
@wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = (time.perf_counter() - start) * 1000
logging.info(f"{func.__name__}: {elapsed:.2f}ms")
return result
return wrapper
@log_performance
def process_corpus(texts):
return [durak.process_text(t, remove_stopwords=True) for t in texts]
🔐 Security
Input Validation
✅ DO: Validate external resources
from pathlib import Path
def load_safe_stopwords(path: Path | str) -> set[str]:
path = Path(path).resolve()
# Ensure path is within allowed directory
if not str(path).startswith(str(Path("resources/tr").resolve())):
raise ValueError(f"Path outside resources directory: {path}")
if not path.exists():
raise FileNotFoundError(f"Resource not found: {path}")
return durak.load_stopwords(path)
✅ DO: Sanitize user input before processing
def process_user_text(text: str) -> list[str]:
# Limit length
if len(text) > 1_000_000:
raise ValueError("Text too long")
# Basic sanitization
text = text.strip()
# Process safely
return durak.process_text(text, remove_stopwords=True)
Further Reading
- ARCHITECTURE.md - System design and component architecture
- examples/ - Working code examples
- benchmarks/ - Performance measurement tools
- CONTRIBUTING.md - How to contribute improvements
Questions or suggestions? Open an issue at github.com/fbkaragoz/durak/issues