Turkish
durak / docs /PROPERTY_TESTING.md
Taklaxbr's picture
Archive open-source github repository (cdliai/durak)
7c16940 verified
|
Raw History Blame Contribute Delete
7.07 kB
# Property-Based Testing Guide
Durak uses [Hypothesis](https://hypothesis.readthedocs.io/) for property-based testing to ensure robustness across a wide range of Turkish text inputs.
## What is Property-Based Testing?
Unlike traditional unit tests that check specific examples, property-based tests verify **mathematical properties** that should hold for **all inputs**.
### Example: Traditional vs Property-Based
**Traditional test:**
```python
def test_normalize_case():
assert normalize_case("İSTANBUL") == "istanbul"
assert normalize_case("ANKARA") == "ankara"
```
**Property-based test:**
```python
@given(turkish_word())
def test_normalize_case_is_idempotent(word):
"""Normalizing twice should equal normalizing once."""
normalized_once = normalize_case(word)
normalized_twice = normalize_case(normalized_once)
assert normalized_once == normalized_twice
```
The property test generates **hundreds of random Turkish words** and verifies the idempotence property holds for all of them.
## Why Property-Based Testing for Turkish NLP?
Turkish has unique challenges that make property testing especially valuable:
1. **Diacritic complexity**: İ/I vs i/ı conversion must be correct in all cases
2. **Apostrophe handling**: `İstanbul'a`, `Ankara'da` create complex token boundaries
3. **Unicode edge cases**: Zero-width characters, combining diacritics, RTL marks
4. **Morphological richness**: Agglutination creates extremely long, unpredictable words
5. **Capitalization rules**: Turkish-specific uppercase/lowercase conversions
Traditional unit tests can't cover the combinatorial space of these features.
## Text Generation Strategies
Durak provides specialized Hypothesis strategies in `tests/strategies.py`:
### Basic Strategies
```python
from tests.strategies import turkish_word, turkish_sentence
# Generate random Turkish words
turkish_word() # "çalışmak", "görmek", "İstanbul", etc.
# Generate words with possessive/case markers
turkish_word_with_suffix() # "İstanbul'a", "Ankara'dan", etc.
# Generate full sentences
turkish_sentence() # "İstanbul'a gittik ve müze'yi gördük."
```
### Advanced Strategies
```python
# Unicode edge cases (zero-width chars, combining diacritics)
turkish_text_with_unicode_edge_cases()
# Lists of Turkish stopwords
turkish_stopword_list()
```
## Writing Property Tests
### 1. Import the Strategy
```python
from hypothesis import given
from tests.strategies import turkish_sentence
```
### 2. Define the Property
```python
@given(turkish_sentence())
def test_tokenize_always_returns_list(text):
"""Tokenization must always return a list."""
tokens = tokenize(text)
assert isinstance(tokens, list)
```
### 3. Run with Hypothesis
Hypothesis will:
- Generate 100-200 random examples (configurable with `@settings(max_examples=N)`)
- Run your test on each example
- If a failure occurs, **shrink** to find the minimal failing case
- Cache failures for regression testing
## Common Properties to Test
### Idempotence
Functions that should give the same result when applied twice:
```python
@given(turkish_sentence())
def test_normalize_is_idempotent(text):
assert normalize_case(text) == normalize_case(normalize_case(text))
```
### Inverse Functions
Functions that should undo each other:
```python
@given(turkish_sentence())
def test_tokenize_rejoins(text):
tokens = tokenize(text)
rejoined = "".join(tokens)
# Should preserve non-whitespace content
assert set(rejoined.replace(" ", "")) == set(text.replace(" ", ""))
```
### Monotonicity
Functions that should never increase size:
```python
@given(turkish_word())
def test_normalization_not_expanding(word):
assert len(normalize_case(word)) <= len(word)
```
### Consistency
Related functions should agree:
```python
@given(turkish_sentence())
def test_offset_consistency(text):
tokens_with_offsets = tokenize_with_offsets(text)
for token, start, end in tokens_with_offsets:
# Offset must be valid
assert 0 <= start < end <= len(text)
# Extracted text should match token
extracted = text[start:end]
assert extracted.lower() == token.lower()
```
## Controlling Test Execution
### Number of Examples
```python
from hypothesis import settings
@given(turkish_sentence())
@settings(max_examples=500) # Generate 500 random examples
def test_something(text):
...
```
### Filtering Invalid Inputs
```python
from hypothesis import assume
@given(turkish_sentence())
def test_non_empty_tokenization(text):
assume(len(text.strip()) > 0) # Skip empty strings
tokens = tokenize(text)
assert len(tokens) > 0
```
### Debugging Failures
When a property test fails, Hypothesis prints:
1. The **minimal failing example** (after shrinking)
2. The **random seed** to reproduce the failure
```
Falsifying example: test_something(text='İ')
You can reproduce this by running:
pytest --hypothesis-seed=12345
```
## Running Property Tests
### Run all property tests:
```bash
pytest tests/test_properties.py -v
```
### Show Hypothesis statistics:
```bash
pytest tests/test_properties.py --hypothesis-show-statistics
```
Output:
```
- 200 passing examples, 0 failing examples, 0 invalid examples
- Typical runtimes: 0-1ms, ~ 50% in data generation
- Stopped because settings.max_examples=200
```
### Run with more examples (stress test):
```bash
pytest tests/test_properties.py --hypothesis-profile=ci
```
## Best Practices
### ✅ Do:
- Test **mathematical properties**, not specific examples
- Use `assume()` to filter out invalid inputs
- Keep properties simple and focused (one property per test)
- Use meaningful assertion messages
- Combine traditional and property-based tests
### ❌ Don't:
- Test implementation details
- Make tests too complex (they become hard to debug)
- Over-constrain input generation (reduces coverage)
- Ignore failures (Hypothesis finds real edge cases!)
## Example: Real Bug Caught by Property Testing
**Property test:**
```python
@given(turkish_word())
def test_normalize_case_removes_uppercase(word):
normalized = normalize_case(word)
assert "İ" not in normalized
```
**Bug found:**
```
Falsifying example: test_normalize_case_removes_uppercase(word='İİİ')
AssertionError: 'İ' found in normalized output
```
**Root cause:** Normalization was using Python's `.lower()` instead of Turkish-aware lowercase conversion. Property testing caught it instantly!
## Resources
- [Hypothesis Documentation](https://hypothesis.readthedocs.io/)
- [Property-Based Testing Patterns](https://hypothesis.works/articles/what-is-property-based-testing/)
- [Hypothesis for Python Developers](https://hypothesis.works/articles/quickcheck-in-every-language/)
## Contributing
When adding new features to Durak:
1. **Add traditional unit tests** for specific examples
2. **Add property tests** for general behavior
3. **Document invariants** that your function guarantees
Property tests serve as **executable specifications** and protect against regressions.