tests/bug_discovery/TEMPLATES/PROPERTY_TEMPLATE.md

# Property-Based Test: [Invariant Name]

## Purpose

Validate invariant: [invariant statement] for [target module/function]

**What this test validates:**
- Invariant holds for all valid inputs (not just hand-picked examples)
- Edge cases discovered through automatic test generation
- Counterexample shrinking to minimal failing case

**Target:**
- Module: `backend/core/[module].py`
- Function: `[function_name]`
- Input type: `[strings, integers, lists, JSON, etc.]`

## Dependencies

**Required Libraries:**
```bash

pip install hypothesis==6.92.0

```

**Target Module:**
- `backend/core/[module].py` - [description of target]
- `backend/api/[routes].py` - [description of target API]

**Hypothesis Strategies:**
- `hypothesis.strategies.text()` - String generation
- `hypothesis.strategies.integers()` - Integer generation
- `hypothesis.strategies.lists()` - List generation
- `hypothesis.strategies.dictionaries()` - Dictionary generation
- `hypothesis.strategies.builds()` - Custom object generation

## Invariant (Document Before Writing Test)

**CRITICAL: Document invariant FIRST, then write test.**

**Property:** [What must be true for all inputs]

Example invariants:
- "Workflow serialization is lossless for all step lists"
- "Agent execution is idempotent for all agent IDs"
- "JSON round-trip preserves data for all valid JSON objects"
- "Episode segmentation produces contiguous time segments for all message lists"

**Domain:** [Input space: strings, integers, lists, JSON objects, etc.]

**Preconditions:** [Required conditions for invariant to hold]

Example:
- Input must be valid UTF-8 string
- List must contain 0-100 items
- JSON must conform to schema

**Postconditions:** [What must be true after operation]

Example:
- Output list has same length as input
- All IDs in output are valid UUIDs
- Timestamps are monotonically increasing

**Example Invariant Documentation:**
```python

"""

Invariant: Workflow serialization is lossless for all step lists.



Property:

- For any list of workflow steps, serializing and deserializing

  produces an equivalent workflow with the same steps.



Domain:

- Input: List of workflow steps (dict with 'action', 'params', 'order')

- Size: 0-100 steps per workflow

- Actions: 'create_agent', 'execute_workflow', 'present_canvas'



Preconditions:

- All steps have valid 'action' field

- All steps have 'order' field (integer, 0-1000)

- 'params' is a dict (can be empty)



Postconditions:

- Deserialized workflow has same number of steps as input

- All steps are present in same order

- All step fields are preserved (action, params, order)

"""

```

## Setup

**Hypothesis settings configuration:**
```python

from hypothesis import given, settings, strategies as st

from tests.property_tests.conftest import DEFAULT_PROFILE, CI_PROFILE



# Use CI profile for faster tests (50 examples, 5s deadline)

# Use local profile for thorough testing (200 examples, 30s deadline)



# Settings profiles defined in tests/property_tests/conftest.py:

# - CI_PROFILE: max_examples=50, deadline=5s (fast for PR checks)

# - DEFAULT_PROFILE: max_examples=200, deadline=30s (thorough for local)



# Example: Use CI profile for fast tests

@settings(CI_PROFILE)



# Example: Use default profile for thorough tests

@settings(DEFAULT_PROFILE)



# Custom settings

@settings(

    max_examples=100,

    deadline=timedelta(seconds=10),

    phases=[Phase.generate]  # Skip reuse phase for faster tests

)

```

**Import strategies:**
```python

from hypothesis import strategies as st



# Common strategies

st.text()  # Random strings (unicode, min_size=0)

st.integers(min_value=0, max_value=100)  # Bounded integers

st.lists(st.integers(), min_size=0, max_size=100)  # Lists

st.dictionaries(st.text(), st.integers())  # Dicts

st.builds(MyModel, id=st.uuid4(), name=st.text())  # Custom objects

```

## Test Procedure

**Step 1: Define invariant (BEFORE writing test)**
```python

# Document invariant in docstring

def test_workflow_serialization_roundtrip(steps):

    """

    Test that workflow serialization is lossless.



    Invariant: Serializing and deserializing a workflow produces

    an equivalent workflow with the same steps.



    Strategy:

    - Generate random workflow steps (0-100 steps)

    - Each step has: action, params, order

    - Actions from: create_agent, execute_workflow, present_canvas



    Expected:

    - Deserialized workflow has same steps as input

    - All fields preserved (action, params, order)

    - Steps in same order

    """

```

**Step 2: Write test with @given decorator**
```python

import pytest

from hypothesis import given, settings, strategies as st

from tests.property_tests.conftest import DEFAULT_PROFILE

from backend.core.workflow_engine import WorkflowDefinition



@pytest.mark.property

@given(st.lists(

    st.fixed_dictionaries({

        'action': st.sampled_from(['create_agent', 'execute_workflow', 'present_canvas']),

        'params': st.dictionaries(st.text(), st.text()),

        'order': st.integers(min_value=0, max_value=1000)

    }),

    min_size=0,

    max_size=100

))

@settings(DEFAULT_PROFILE)

def test_workflow_serialization_roundtrip(steps):

    """

    Test that workflow serialization is lossless for all step lists.



    Invariant: For any list of workflow steps, serializing and deserializing

    produces an equivalent workflow with the same steps.



    Strategy: Generate 0-100 workflow steps with random actions, params, order.



    Expected: All steps preserved in same order.

    """

    # Arrange: Create workflow from generated steps

    workflow = WorkflowDefinition(name="test", steps=steps)



    # Act: Serialize and deserialize

    serialized = workflow.serialize()

    deserialized = WorkflowDefinition.deserialize(serialized)



    # Assert: Invariant holds

    assert len(deserialized.steps) == len(steps), \

        f"Step count mismatch: {len(deserialized.steps)} != {len(steps)}"



    for i, (original, recovered) in enumerate(zip(steps, deserialized.steps)):

        assert recovered['action'] == original['action'], \

            f"Step {i}: action mismatch ({recovered['action']} != {original['action']})"

        assert recovered['order'] == original['order'], \

            f"Step {i}: order mismatch ({recovered['order']} != {original['order']})"

        assert recovered['params'] == original['params'], \

            f"Step {i}: params mismatch ({recovered['params']} != {original['params']})"

```

**Step 3: Run test and verify**
```bash

# Run property test

pytest backend/tests/property_tests/test_workflow_properties.py::test_workflow_serialization_roundtrip -v



# Hypothesis will:

# 1. Generate 100+ random examples (by default)

# 2. Shrink counterexample to minimal case (if invariant violated)

# 3. Print minimal failing input (for bug filing)



# Example output on failure:

# Falsifying example:

# test_workflow_serialization_roundtrip(

#     steps=[

#         {'action': 'create_agent', 'params': {}, 'order': 0},

#         {'action': 'execute_workflow', 'params': {'name': ''}, 'order': 1}

#     ]

# )

# Shrunk from 53 steps to 2 steps in 0.05s

```

**Step 4: Handle invariant violations**
```python

# If test fails, Hypothesis provides minimal counterexample

# Example: Steps with empty 'name' param cause serialization error



# Fix bug or refine invariant

# Option 1: Fix bug in serialization logic

# Option 2: Add precondition: 'name' must be non-empty

# Option 3: Update invariant to handle empty names correctly



# Example: Add precondition to strategy

@given(st.lists(

    st.fixed_dictionaries({

        'action': st.sampled_from(['create_agent', 'execute_workflow', 'present_canvas']),

        'params': st.dictionaries(

            st.text(min_size=1, max_size=10),  # Non-empty keys

            st.text(min_size=1)  # Non-empty values (precondition)

        ),

        'order': st.integers(min_value=0, max_value=1000)

    }),

    min_size=0,

    max_size=100

))

@settings(DEFAULT_PROFILE)

def test_workflow_serialization_roundtrip_with_preconditions(steps):

    """

    Test that workflow serialization is lossless for all step lists.



    Invariant: For any list of workflow steps with non-empty params,

    serializing and deserializing produces an equivalent workflow.



    Precondition: All param keys and values must be non-empty strings.

    """

    # ... same test logic

```

## Expected Behavior

**Invariant holds (test passes):**
- All generated examples satisfy invariant
- Hypothesis runs 100-200 examples (depending on profile)
- No counterexamples found
- Test completes in <30s (per TQ-03)

**Invariant violated (test fails):**
- Hypothesis finds counterexample
- Automatically shrinks to minimal failing case
- Prints minimal input that violates invariant
- Provides reproduction script

**Example failure output:**
```python

# ==================== FAILURES ====================

# ____________________ test_workflow_serialization_roundtrip ____________________

#

# Falsifying example:

# test_workflow_serialization_roundtrip(

#     steps=[

#         {'action': 'execute_workflow', 'params': {'name': ''}, 'order': 0}

#     ]

# )

# Shrunk from 47 steps to 1 step in 0.03s

#

# assert 1 == 0

#  +  where 1 = len([{'action': 'execute_workflow', 'params': {}, 'order': 0}])

#  +  and   0 = len([])

#

# Step 0: params mismatch ({'name': ''} != {})

```

**Hypothesis shrinking process:**
1. Find first failing example (may be complex: 53 steps)
2. Simplify example (remove steps, reduce values)
3. Find minimal counterexample (2 steps → 1 step)
4. Report minimal failing case for debugging

## Bug Filing

**Automatic bug filing on invariant violation:**
```python

from tests.bug_discovery.bug_filing_service import BugFilingService



@pytest.mark.property

@given(st.lists(st.integers(), min_size=0, max_size=100))

@settings(DEFAULT_PROFILE)

def test_[invariant_name](inputs):

    """

    Test that [invariant] holds for all [inputs].



    Invariant: [statement]

    Strategy: [strategy description]



    Fails on: [known counterexample]

    """

    try:

        # Test logic

        result = [function_under_test](inputs)

        assert [invariant_check](result), f"Invariant violated: {result}"



    except AssertionError as e:

        # File bug with counterexample

        BugFilingService.file_bug(

            test_name=f"test_{[invariant_name]}_violation",

            error_message=f"Invariant violation: {str(e)}",

            metadata={

                "test_type": "property",

                "invariant": "[invariant_name]",

                "counterexample": str(inputs),

                "shrunk_input": str(inputs),  # Hypothesis already shrunk

                "hypothesis_examples": 100,  # Number of examples run

                "strategy": "st.lists(st.integers(), min_size=0, max_size=100)"

            },

            expected_behavior=f"Invariant should hold: {[invariant_statement]}",

            actual_behavior=f"Invariant violated for input: {inputs}"

        )

        raise  # Re-raise to fail test

```

**Manual bug filing (if not automatic):**
```bash

# Bug title: [Bug] Invariant violation: [Invariant Name]



# Bug body:

## Bug Description



Property-based test discovered invariant violation in [function_name].



## Invariant



**Statement:** [invariant statement]



**Domain:** [input space]



**Preconditions:** [required conditions]



## Counterexample



```python

# Minimal failing input (shrunk by Hypothesis)

inputs = [paste counterexample from test output]



# Reproducer

from backend.core.[module] import [function_name]

result = [function_name](inputs)

# Expected: [expected behavior]

# Actual: [actual behavior]

```

## Steps to Reproduce

1. Run property test: `pytest backend/tests/property_tests/test_[module]_properties.py::test_[invariant_name] -v`
2. Hypothesis finds counterexample after N examples
3. Counterexample shrunk to minimal case: [paste input]
4. Invariant violated: [description of violation]

## Shrinking Process

- Original failing example: [N] steps/items
- Shrunk to: [M] steps/items (minimal case)
- Shrinking time: [seconds]

## Hypothesis Output

```

[paste Hypothesis output with counterexample]

```

## Expected Behavior

Invariant should hold: [invariant statement]

For input: [counterexample], expected: [expected result]

## Actual Behavior

Invariant violated: [description of violation]

For input: [counterexample], actual: [actual result]

## Test Context

- **Test:** `test_[invariant_name]`
- **Hypothesis examples run:** [N]
- **Strategy:** [Hypothesis strategy used]
- **Settings:** [max_examples, deadline]

- **Platform:** [output of `uname -a`]

- **Python:** [output of `python --version`]

```



## TQ Compliance



**TQ-01 (Test Independence):**

- Each test generates fresh inputs (Hypothesis @given decorator)

- No shared state between property tests

- Each invariant tested independently



**TQ-02 (Pass Rate):**

- Property tests have 100% pass rate (invariant violations = real bugs)

- Same input always produces same output (deterministic target function)

- No flaky tests (Hypothesis provides reproducible examples)



**TQ-03 (Performance):**

- Hypothesis settings enforce deadline (30s default)

- CI profile: 50 examples, 5s deadline (fast for PR checks)

- Default profile: 200 examples, 30s deadline (thorough for local)



**TQ-04 (Determinism):**

- Same input produces same output (deterministic target function required)

- Hypothesis uses fixed random seed (reproducible examples)

- Counterexamples are reproducible (same test run = same failure)



**TQ-05 (Coverage Quality):**

- Tests invariant (observable behavior), not implementation

- Hypothesis explores input space systematically (edge cases discovered)

- Property-based: tests general property, not specific examples



## pytest.ini Marker



Add to `backend/pytest.ini`:

```ini

[pytest]

markers =

    property: Property-based tests (Hypothesis, slow, thorough)

```



Run only property tests:

```bash

pytest backend/tests/property_tests/ -v -m property
```



Skip property tests in fast CI:

```bash

pytest backend/tests/ -v -m "not property"

```

## Invariant-First Thinking

**Process:**
1. **Document invariant first** (before writing test)
2. Write test that validates invariant
3. Run test to discover counterexamples
4. Fix bugs or refine invariant (add preconditions)
5. Re-run test to verify fix

**Why invariant-first?**
- Forces clarity about what must be true
- Prevents implementation-driven tests
- Catches edge cases early
- Makes tests maintainable (invariant is documentation)

**Bad example (not invariant-first):**
```python

# BAD: Test specific examples, no invariant documented

def test_workflow_serialization():

    workflow = Workflow(steps=[{'action': 'create_agent'}])

    serialized = workflow.serialize()

    deserialized = Workflow.deserialize(serialized)

    assert deserialized.steps == workflow.steps

```

**Good example (invariant-first):**
```python

# GOOD: Invariant documented, tested for all inputs

@given(st.lists(st.builds(WorkflowStep)))

@settings(DEFAULT_PROFILE)

def test_workflow_serialization_roundtrip(steps):

    """

    Test that workflow serialization is lossless for all step lists.



    Invariant: For any list of workflow steps, serializing and

    deserializing produces an equivalent workflow.

    """

    workflow = Workflow(steps=steps)

    serialized = workflow.serialize()

    deserialized = Workflow.deserialize(serialized)

    assert deserialized.steps == workflow.steps

```

## See Also

- [Hypothesis Documentation](https://hypothesis.readthedocs.io)
- [Property-Based Testing](https://hypothesis.works/articles/what-is-property-based-testing/)
- `backend/docs/TEST_QUALITY_STANDARDS.md` - TQ-01 through TQ-05
- `backend/tests/bug_discovery/TEMPLATES/README.md` - Template usage guide