|
Download docs/PROPERTY_TESTING.md from veripazari/durak: direct link, hf CLI and curl.
- Browser
- Download file 7.07 kB
-
https://huggingface.co/veripazari/durak/resolve/main/docs/PROPERTY_TESTING.md
- Command line
-
hf download hf://veripazari/durak/docs/PROPERTY_TESTING.md
-
curl -L -o PROPERTY_TESTING.md https://huggingface.co/veripazari/durak/resolve/main/docs/PROPERTY_TESTING.md
7.07 kB
| # Property-Based Testing Guide | |
| Durak uses [Hypothesis](https://hypothesis.readthedocs.io/) for property-based testing to ensure robustness across a wide range of Turkish text inputs. | |
| ## What is Property-Based Testing? | |
| Unlike traditional unit tests that check specific examples, property-based tests verify **mathematical properties** that should hold for **all inputs**. | |
| ### Example: Traditional vs Property-Based | |
| **Traditional test:** | |
| ```python | |
| def test_normalize_case(): | |
| assert normalize_case("İSTANBUL") == "istanbul" | |
| assert normalize_case("ANKARA") == "ankara" | |
| ``` | |
| **Property-based test:** | |
| ```python | |
| @given(turkish_word()) | |
| def test_normalize_case_is_idempotent(word): | |
| """Normalizing twice should equal normalizing once.""" | |
| normalized_once = normalize_case(word) | |
| normalized_twice = normalize_case(normalized_once) | |
| assert normalized_once == normalized_twice | |
| ``` | |
| The property test generates **hundreds of random Turkish words** and verifies the idempotence property holds for all of them. | |
| ## Why Property-Based Testing for Turkish NLP? | |
| Turkish has unique challenges that make property testing especially valuable: | |
| 1. **Diacritic complexity**: İ/I vs i/ı conversion must be correct in all cases | |
| 2. **Apostrophe handling**: `İstanbul'a`, `Ankara'da` create complex token boundaries | |
| 3. **Unicode edge cases**: Zero-width characters, combining diacritics, RTL marks | |
| 4. **Morphological richness**: Agglutination creates extremely long, unpredictable words | |
| 5. **Capitalization rules**: Turkish-specific uppercase/lowercase conversions | |
| Traditional unit tests can't cover the combinatorial space of these features. | |
| ## Text Generation Strategies | |
| Durak provides specialized Hypothesis strategies in `tests/strategies.py`: | |
| ### Basic Strategies | |
| ```python | |
| from tests.strategies import turkish_word, turkish_sentence | |
| # Generate random Turkish words | |
| turkish_word() # "çalışmak", "görmek", "İstanbul", etc. | |
| # Generate words with possessive/case markers | |
| turkish_word_with_suffix() # "İstanbul'a", "Ankara'dan", etc. | |
| # Generate full sentences | |
| turkish_sentence() # "İstanbul'a gittik ve müze'yi gördük." | |
| ``` | |
| ### Advanced Strategies | |
| ```python | |
| # Unicode edge cases (zero-width chars, combining diacritics) | |
| turkish_text_with_unicode_edge_cases() | |
| # Lists of Turkish stopwords | |
| turkish_stopword_list() | |
| ``` | |
| ## Writing Property Tests | |
| ### 1. Import the Strategy | |
| ```python | |
| from hypothesis import given | |
| from tests.strategies import turkish_sentence | |
| ``` | |
| ### 2. Define the Property | |
| ```python | |
| @given(turkish_sentence()) | |
| def test_tokenize_always_returns_list(text): | |
| """Tokenization must always return a list.""" | |
| tokens = tokenize(text) | |
| assert isinstance(tokens, list) | |
| ``` | |
| ### 3. Run with Hypothesis | |
| Hypothesis will: | |
| - Generate 100-200 random examples (configurable with `@settings(max_examples=N)`) | |
| - Run your test on each example | |
| - If a failure occurs, **shrink** to find the minimal failing case | |
| - Cache failures for regression testing | |
| ## Common Properties to Test | |
| ### Idempotence | |
| Functions that should give the same result when applied twice: | |
| ```python | |
| @given(turkish_sentence()) | |
| def test_normalize_is_idempotent(text): | |
| assert normalize_case(text) == normalize_case(normalize_case(text)) | |
| ``` | |
| ### Inverse Functions | |
| Functions that should undo each other: | |
| ```python | |
| @given(turkish_sentence()) | |
| def test_tokenize_rejoins(text): | |
| tokens = tokenize(text) | |
| rejoined = "".join(tokens) | |
| # Should preserve non-whitespace content | |
| assert set(rejoined.replace(" ", "")) == set(text.replace(" ", "")) | |
| ``` | |
| ### Monotonicity | |
| Functions that should never increase size: | |
| ```python | |
| @given(turkish_word()) | |
| def test_normalization_not_expanding(word): | |
| assert len(normalize_case(word)) <= len(word) | |
| ``` | |
| ### Consistency | |
| Related functions should agree: | |
| ```python | |
| @given(turkish_sentence()) | |
| def test_offset_consistency(text): | |
| tokens_with_offsets = tokenize_with_offsets(text) | |
| for token, start, end in tokens_with_offsets: | |
| # Offset must be valid | |
| assert 0 <= start < end <= len(text) | |
| # Extracted text should match token | |
| extracted = text[start:end] | |
| assert extracted.lower() == token.lower() | |
| ``` | |
| ## Controlling Test Execution | |
| ### Number of Examples | |
| ```python | |
| from hypothesis import settings | |
| @given(turkish_sentence()) | |
| @settings(max_examples=500) # Generate 500 random examples | |
| def test_something(text): | |
| ... | |
| ``` | |
| ### Filtering Invalid Inputs | |
| ```python | |
| from hypothesis import assume | |
| @given(turkish_sentence()) | |
| def test_non_empty_tokenization(text): | |
| assume(len(text.strip()) > 0) # Skip empty strings | |
| tokens = tokenize(text) | |
| assert len(tokens) > 0 | |
| ``` | |
| ### Debugging Failures | |
| When a property test fails, Hypothesis prints: | |
| 1. The **minimal failing example** (after shrinking) | |
| 2. The **random seed** to reproduce the failure | |
| ``` | |
| Falsifying example: test_something(text='İ') | |
| You can reproduce this by running: | |
| pytest --hypothesis-seed=12345 | |
| ``` | |
| ## Running Property Tests | |
| ### Run all property tests: | |
| ```bash | |
| pytest tests/test_properties.py -v | |
| ``` | |
| ### Show Hypothesis statistics: | |
| ```bash | |
| pytest tests/test_properties.py --hypothesis-show-statistics | |
| ``` | |
| Output: | |
| ``` | |
| - 200 passing examples, 0 failing examples, 0 invalid examples | |
| - Typical runtimes: 0-1ms, ~ 50% in data generation | |
| - Stopped because settings.max_examples=200 | |
| ``` | |
| ### Run with more examples (stress test): | |
| ```bash | |
| pytest tests/test_properties.py --hypothesis-profile=ci | |
| ``` | |
| ## Best Practices | |
| ### ✅ Do: | |
| - Test **mathematical properties**, not specific examples | |
| - Use `assume()` to filter out invalid inputs | |
| - Keep properties simple and focused (one property per test) | |
| - Use meaningful assertion messages | |
| - Combine traditional and property-based tests | |
| ### ❌ Don't: | |
| - Test implementation details | |
| - Make tests too complex (they become hard to debug) | |
| - Over-constrain input generation (reduces coverage) | |
| - Ignore failures (Hypothesis finds real edge cases!) | |
| ## Example: Real Bug Caught by Property Testing | |
| **Property test:** | |
| ```python | |
| @given(turkish_word()) | |
| def test_normalize_case_removes_uppercase(word): | |
| normalized = normalize_case(word) | |
| assert "İ" not in normalized | |
| ``` | |
| **Bug found:** | |
| ``` | |
| Falsifying example: test_normalize_case_removes_uppercase(word='İİİ') | |
| AssertionError: 'İ' found in normalized output | |
| ``` | |
| **Root cause:** Normalization was using Python's `.lower()` instead of Turkish-aware lowercase conversion. Property testing caught it instantly! | |
| ## Resources | |
| - [Hypothesis Documentation](https://hypothesis.readthedocs.io/) | |
| - [Property-Based Testing Patterns](https://hypothesis.works/articles/what-is-property-based-testing/) | |
| - [Hypothesis for Python Developers](https://hypothesis.works/articles/quickcheck-in-every-language/) | |
| ## Contributing | |
| When adding new features to Durak: | |
| 1. **Add traditional unit tests** for specific examples | |
| 2. **Add property tests** for general behavior | |
| 3. **Document invariants** that your function guarantees | |
| Property tests serve as **executable specifications** and protect against regressions. | |