Turkish
durak / docs /PROPERTY_TESTING.md
Taklaxbr's picture
Archive open-source github repository (cdliai/durak)
7c16940 verified
|
Raw History Blame Contribute Delete
7.07 kB

Property-Based Testing Guide

Durak uses Hypothesis for property-based testing to ensure robustness across a wide range of Turkish text inputs.

What is Property-Based Testing?

Unlike traditional unit tests that check specific examples, property-based tests verify mathematical properties that should hold for all inputs.

Example: Traditional vs Property-Based

Traditional test:

def test_normalize_case():
    assert normalize_case("İSTANBUL") == "istanbul"
    assert normalize_case("ANKARA") == "ankara"

Property-based test:

@given(turkish_word())
def test_normalize_case_is_idempotent(word):
    """Normalizing twice should equal normalizing once."""
    normalized_once = normalize_case(word)
    normalized_twice = normalize_case(normalized_once)
    assert normalized_once == normalized_twice

The property test generates hundreds of random Turkish words and verifies the idempotence property holds for all of them.

Why Property-Based Testing for Turkish NLP?

Turkish has unique challenges that make property testing especially valuable:

  1. Diacritic complexity: İ/I vs i/ı conversion must be correct in all cases
  2. Apostrophe handling: İstanbul'a, Ankara'da create complex token boundaries
  3. Unicode edge cases: Zero-width characters, combining diacritics, RTL marks
  4. Morphological richness: Agglutination creates extremely long, unpredictable words
  5. Capitalization rules: Turkish-specific uppercase/lowercase conversions

Traditional unit tests can't cover the combinatorial space of these features.

Text Generation Strategies

Durak provides specialized Hypothesis strategies in tests/strategies.py:

Basic Strategies

from tests.strategies import turkish_word, turkish_sentence

# Generate random Turkish words
turkish_word()  # "çalışmak", "görmek", "İstanbul", etc.

# Generate words with possessive/case markers
turkish_word_with_suffix()  # "İstanbul'a", "Ankara'dan", etc.

# Generate full sentences
turkish_sentence()  # "İstanbul'a gittik ve müze'yi gördük."

Advanced Strategies

# Unicode edge cases (zero-width chars, combining diacritics)
turkish_text_with_unicode_edge_cases()

# Lists of Turkish stopwords
turkish_stopword_list()

Writing Property Tests

1. Import the Strategy

from hypothesis import given
from tests.strategies import turkish_sentence

2. Define the Property

@given(turkish_sentence())
def test_tokenize_always_returns_list(text):
    """Tokenization must always return a list."""
    tokens = tokenize(text)
    assert isinstance(tokens, list)

3. Run with Hypothesis

Hypothesis will:

  • Generate 100-200 random examples (configurable with @settings(max_examples=N))
  • Run your test on each example
  • If a failure occurs, shrink to find the minimal failing case
  • Cache failures for regression testing

Common Properties to Test

Idempotence

Functions that should give the same result when applied twice:

@given(turkish_sentence())
def test_normalize_is_idempotent(text):
    assert normalize_case(text) == normalize_case(normalize_case(text))

Inverse Functions

Functions that should undo each other:

@given(turkish_sentence())
def test_tokenize_rejoins(text):
    tokens = tokenize(text)
    rejoined = "".join(tokens)
    # Should preserve non-whitespace content
    assert set(rejoined.replace(" ", "")) == set(text.replace(" ", ""))

Monotonicity

Functions that should never increase size:

@given(turkish_word())
def test_normalization_not_expanding(word):
    assert len(normalize_case(word)) <= len(word)

Consistency

Related functions should agree:

@given(turkish_sentence())
def test_offset_consistency(text):
    tokens_with_offsets = tokenize_with_offsets(text)
    for token, start, end in tokens_with_offsets:
        # Offset must be valid
        assert 0 <= start < end <= len(text)
        # Extracted text should match token
        extracted = text[start:end]
        assert extracted.lower() == token.lower()

Controlling Test Execution

Number of Examples

from hypothesis import settings

@given(turkish_sentence())
@settings(max_examples=500)  # Generate 500 random examples
def test_something(text):
    ...

Filtering Invalid Inputs

from hypothesis import assume

@given(turkish_sentence())
def test_non_empty_tokenization(text):
    assume(len(text.strip()) > 0)  # Skip empty strings
    tokens = tokenize(text)
    assert len(tokens) > 0

Debugging Failures

When a property test fails, Hypothesis prints:

  1. The minimal failing example (after shrinking)
  2. The random seed to reproduce the failure
Falsifying example: test_something(text='İ')
You can reproduce this by running:
    pytest --hypothesis-seed=12345

Running Property Tests

Run all property tests:

pytest tests/test_properties.py -v

Show Hypothesis statistics:

pytest tests/test_properties.py --hypothesis-show-statistics

Output:

  - 200 passing examples, 0 failing examples, 0 invalid examples
  - Typical runtimes: 0-1ms, ~ 50% in data generation
  - Stopped because settings.max_examples=200

Run with more examples (stress test):

pytest tests/test_properties.py --hypothesis-profile=ci

Best Practices

✅ Do:

  • Test mathematical properties, not specific examples
  • Use assume() to filter out invalid inputs
  • Keep properties simple and focused (one property per test)
  • Use meaningful assertion messages
  • Combine traditional and property-based tests

❌ Don't:

  • Test implementation details
  • Make tests too complex (they become hard to debug)
  • Over-constrain input generation (reduces coverage)
  • Ignore failures (Hypothesis finds real edge cases!)

Example: Real Bug Caught by Property Testing

Property test:

@given(turkish_word())
def test_normalize_case_removes_uppercase(word):
    normalized = normalize_case(word)
    assert "İ" not in normalized

Bug found:

Falsifying example: test_normalize_case_removes_uppercase(word='İİİ')
AssertionError: 'İ' found in normalized output

Root cause: Normalization was using Python's .lower() instead of Turkish-aware lowercase conversion. Property testing caught it instantly!

Resources

Contributing

When adding new features to Durak:

  1. Add traditional unit tests for specific examples
  2. Add property tests for general behavior
  3. Document invariants that your function guarantees

Property tests serve as executable specifications and protect against regressions.