Turkish
durak / docs /BEST_PRACTICES.md
Taklaxbr's picture
Archive open-source github repository (cdliai/durak)
7c16940 verified
|
Raw History Blame Contribute Delete
9.96 kB

Best Practices Guide

This guide provides recommendations for using and extending Durak effectively.

🎯 General Principles

1. Resource Management

✅ DO: Use embedded Rust resources for production

from durak import _durak_core

# Fast, no file I/O
stopwords = set(_durak_core.get_stopwords_base())
suffixes = _durak_core.get_detached_suffixes()

❌ DON'T: Load from files in hot paths

# Slower, file I/O overhead
from durak import load_stopword_resource
stopwords = load_stopword_resource("base/turkish")  # OK for development, not production loops

When to use file-based loading:

  • Development and debugging
  • Custom resource files not yet embedded
  • Dynamic resource updates
  • Testing with modified resources

2. Performance Optimization

✅ DO: Use Rust functions directly for performance-critical code

from durak import _durak_core

# 5-10x faster
normalized = _durak_core.fast_normalize(text)
tokens = _durak_core.tokenize_with_offsets(text)

✅ DO: Batch process when possible

from durak import _durak_core

# Process all texts before filtering
normalized_texts = [_durak_core.fast_normalize(t) for t in texts]
tokenized = [_durak_core.tokenize_with_offsets(t) for t in normalized_texts]

❌ DON'T: Mix Python/Rust unnecessarily in loops

# Inefficient: context switching overhead
for text in texts:
    cleaned = durak.clean_text(text)  # Python
    normalized = _durak_core.fast_normalize(cleaned)  # Rust
    # Better: do all cleaning in one layer, then all normalization

3. Pipeline Design

✅ DO: Create reusable pipeline objects

from durak import StopwordManager, Pipeline, Normalizer

# Create once, reuse many times
stopword_mgr = StopwordManager(additions=["custom"], keep=["important"])
normalizer = Normalizer()

pipeline = Pipeline([
    normalizer,
    lambda tokens: [t for t in tokens if not stopword_mgr.is_stopword(t)]
])

# Reuse efficiently
results = [pipeline(text) for text in texts]

❌ DON'T: Recreate expensive objects in loops

# Bad: recreates manager every iteration
for text in texts:
    mgr = StopwordManager(additions=["custom"])  # Expensive!
    tokens = process_text(text)
    filtered = [t for t in tokens if not mgr.is_stopword(t)]

📝 Configuration Management

Externalizing Hard-Coded Values

✅ DO: Use resource files for configuration

# resources/tr/config/custom_apostrophes.txt
# resources/tr/config/domain_stopwords.txt

from pathlib import Path

config_dir = Path("resources/tr/config")
apostrophes = (config_dir / "custom_apostrophes.txt").read_text().strip().split("\n")

Best practice for new configurations:

  1. Add resource file under resources/tr/config/
  2. Embed in Rust if needed frequently (see next section)
  3. Load once at module initialization
  4. Cache in module-level variable

Adding New Embedded Resources

To add new resources to Rust binary:

  1. Add file to resources/tr/

  2. Embed in Rust (src/lib.rs):

    static MY_DATA: &str = include_str!("../resources/tr/config/mydata.txt");
    
  3. Create accessor function:

    #[pyfunction]
    fn get_my_data() -> Vec<String> {
        MY_DATA.lines().map(|s| s.trim().to_string()).collect()
    }
    
  4. Export in module:

    m.add_function(wrap_pyfunction!(get_my_data, m)?)?;
    
  5. Add type stub (python/durak/_durak_core.pyi):

    def get_my_data() -> list[str]: ...
    
  6. Rebuild: maturin develop --release

🔧 Extension Patterns

Custom Stopword Management

✅ DO: Use domain-specific stopword files

# resources/tr/stopwords/domains/medical.txt
# resources/tr/stopwords/domains/legal.txt

manager = StopwordManager.from_resources([
    "base/turkish",
    "domains/medical",  # Domain-specific
], keep=["hasta", "doktor"])  # Keep domain terms

✅ DO: Export and version stopword sets

manager = StopwordManager(additions=["term1", "term2"])
manager.export("my_project_stopwords_v1.txt")

# Later, load exact same set
from durak import load_stopwords
stopwords = load_stopwords("my_project_stopwords_v1.txt")

Morphological Analysis

Current limitation: Basic suffix stripping only

Recommended approach for production:

# For simple cases: use built-in
from durak import _durak_core
stripped = _durak_core.strip_suffixes("kitaplardan")  # "kitap"

# For complex morphology: use external tools
# Option 1: Zemberek (Java-based, most accurate for Turkish)
# Option 2: spaCy tr_core_news_lg
# Option 3: Stanza Turkish model

# Then integrate with Durak pipeline:
from durak import Pipeline

pipeline = Pipeline([
    lambda text: durak.clean_text(text),
    lambda text: durak.tokenize(text),
    lambda tokens: your_lemmatizer.lemmatize(tokens),  # External tool
    lambda tokens: durak.remove_stopwords(tokens),
])

Suffix Configuration

Current state: Rust suffixes are hard-coded for demo purposes

To customize suffix stripping:

  1. Edit resources/tr/config/lemma_suffixes.txt

  2. Load in Python:

    from pathlib import Path
    
    suffix_file = Path("resources/tr/config/lemma_suffixes.txt")
    custom_suffixes = [
        line.strip()
        for line in suffix_file.read_text().splitlines()
        if line.strip() and not line.startswith("#")
    ]
    
    # Use with Python-based stripping
    def strip_custom(word: str, suffixes: list[str]) -> str:
        for suffix in sorted(suffixes, key=len, reverse=True):
            if word.endswith(suffix) and len(word) > len(suffix) + 2:
                return word[:-len(suffix)]
        return word
    
    lemma = strip_custom("kitaplardan", custom_suffixes)
    
  3. For production: Implement proper morphological analysis with vowel harmony

🧪 Testing Best Practices

Unit Tests

✅ DO: Test with realistic Turkish text

def test_preserves_diacritics():
    text = "İstanbul'dan Türkiye'ye"
    tokens = durak.tokenize(text)
    assert "İstanbul'dan" in tokens  # Preserves İ
    assert "Türkiye'ye" in tokens    # Preserves ü

✅ DO: Test edge cases

def test_apostrophe_variants():
    # Both straight and curly apostrophes
    assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"]
    assert durak.tokenize("Ankara'da") == ["Ankara", "'", "da"]  # Curly

Performance Testing

✅ DO: Benchmark critical paths

import time

def benchmark(func, *args, iterations=1000):
    start = time.perf_counter()
    for _ in range(iterations):
        func(*args)
    return (time.perf_counter() - start) / iterations * 1000

# Measure
py_time = benchmark(durak.normalize_case, text)
rust_time = benchmark(_durak_core.fast_normalize, text)
print(f"Speedup: {py_time / rust_time:.2f}x")

🚀 Deployment

Production Checklist

  • Use maturin build --release for optimized builds
  • Verify embedded resources with _durak_core.get_*() functions
  • Pre-load stopword managers at application startup
  • Use Rust functions for hot paths
  • Enable logging for resource loading issues
  • Version-lock dependencies in production
  • Test with production-like data volumes

Distribution

For PyPI releases:

# Build optimized wheel
maturin build --release

# All resources are embedded in the wheel
# No need to ship separate data files
pip install durak-nlp  # Just works!

For custom deployments:

# Development mode (editable install)
maturin develop

# Production mode (optimized)
maturin develop --release

📊 Monitoring

Resource Loading

import logging

logging.basicConfig(level=logging.INFO)

# Monitor resource loading
try:
    from durak import _durak_core
    stopwords = _durak_core.get_stopwords_base()
    logging.info(f"Loaded {len(stopwords)} stopwords from embedded resources")
except ImportError:
    logging.warning("Rust extension not available, using file-based loading")
    stopwords = durak.load_stopword_resource("base/turkish")

Performance Metrics

import time
from functools import wraps

def log_performance(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = (time.perf_counter() - start) * 1000
        logging.info(f"{func.__name__}: {elapsed:.2f}ms")
        return result
    return wrapper

@log_performance
def process_corpus(texts):
    return [durak.process_text(t, remove_stopwords=True) for t in texts]

🔐 Security

Input Validation

✅ DO: Validate external resources

from pathlib import Path

def load_safe_stopwords(path: Path | str) -> set[str]:
    path = Path(path).resolve()

    # Ensure path is within allowed directory
    if not str(path).startswith(str(Path("resources/tr").resolve())):
        raise ValueError(f"Path outside resources directory: {path}")

    if not path.exists():
        raise FileNotFoundError(f"Resource not found: {path}")

    return durak.load_stopwords(path)

✅ DO: Sanitize user input before processing

def process_user_text(text: str) -> list[str]:
    # Limit length
    if len(text) > 1_000_000:
        raise ValueError("Text too long")

    # Basic sanitization
    text = text.strip()

    # Process safely
    return durak.process_text(text, remove_stopwords=True)

Further Reading


Questions or suggestions? Open an issue at github.com/fbkaragoz/durak/issues