tests/bug_discovery/TEMPLATES/PROPERTY_TEMPLATE.md

Property-Based Test: [Invariant Name]

Purpose

Validate invariant: [invariant statement] for [target module/function]

What this test validates:

• Invariant holds for all valid inputs (not just hand-picked examples)
• Edge cases discovered through automatic test generation
• Counterexample shrinking to minimal failing case

Target:

• Module: backend/core/[module].py
• Function: [function_name]
• Input type: [strings, integers, lists, JSON, etc.]

Dependencies

Required Libraries:

pip install hypothesis==6.92.0

Target Module:

• backend/core/[module].py - [description of target]
• backend/api/[routes].py - [description of target API]

Hypothesis Strategies:

• hypothesis.strategies.text() - String generation
• hypothesis.strategies.integers() - Integer generation
• hypothesis.strategies.lists() - List generation
• hypothesis.strategies.dictionaries() - Dictionary generation
• hypothesis.strategies.builds() - Custom object generation

Invariant (Document Before Writing Test)

CRITICAL: Document invariant FIRST, then write test.

Property: [What must be true for all inputs]

Example invariants:

• "Workflow serialization is lossless for all step lists"
• "Agent execution is idempotent for all agent IDs"
• "JSON round-trip preserves data for all valid JSON objects"
• "Episode segmentation produces contiguous time segments for all message lists"

Domain: [Input space: strings, integers, lists, JSON objects, etc.]

Preconditions: [Required conditions for invariant to hold]

Example:

• Input must be valid UTF-8 string
• List must contain 0-100 items
• JSON must conform to schema

Postconditions: [What must be true after operation]

Example:

• Output list has same length as input
• All IDs in output are valid UUIDs
• Timestamps are monotonically increasing

Example Invariant Documentation:

"""
Invariant: Workflow serialization is lossless for all step lists.

Property:
- For any list of workflow steps, serializing and deserializing
  produces an equivalent workflow with the same steps.

Domain:
- Input: List of workflow steps (dict with 'action', 'params', 'order')
- Size: 0-100 steps per workflow
- Actions: 'create_agent', 'execute_workflow', 'present_canvas'

Preconditions:
- All steps have valid 'action' field
- All steps have 'order' field (integer, 0-1000)
- 'params' is a dict (can be empty)

Postconditions:
- Deserialized workflow has same number of steps as input
- All steps are present in same order
- All step fields are preserved (action, params, order)
"""

Setup

Hypothesis settings configuration:

from hypothesis import given, settings, strategies as st
from tests.property_tests.conftest import DEFAULT_PROFILE, CI_PROFILE

# Use CI profile for faster tests (50 examples, 5s deadline)
# Use local profile for thorough testing (200 examples, 30s deadline)

# Settings profiles defined in tests/property_tests/conftest.py:
# - CI_PROFILE: max_examples=50, deadline=5s (fast for PR checks)
# - DEFAULT_PROFILE: max_examples=200, deadline=30s (thorough for local)

# Example: Use CI profile for fast tests
@settings(CI_PROFILE)

# Example: Use default profile for thorough tests
@settings(DEFAULT_PROFILE)

# Custom settings
@settings(
    max_examples=100,
    deadline=timedelta(seconds=10),
    phases=[Phase.generate]  # Skip reuse phase for faster tests
)

Import strategies:

from hypothesis import strategies as st

# Common strategies
st.text()  # Random strings (unicode, min_size=0)
st.integers(min_value=0, max_value=100)  # Bounded integers
st.lists(st.integers(), min_size=0, max_size=100)  # Lists
st.dictionaries(st.text(), st.integers())  # Dicts
st.builds(MyModel, id=st.uuid4(), name=st.text())  # Custom objects

Test Procedure

Step 1: Define invariant (BEFORE writing test)

# Document invariant in docstring
def test_workflow_serialization_roundtrip(steps):
    """
    Test that workflow serialization is lossless.

    Invariant: Serializing and deserializing a workflow produces
    an equivalent workflow with the same steps.

    Strategy:
    - Generate random workflow steps (0-100 steps)
    - Each step has: action, params, order
    - Actions from: create_agent, execute_workflow, present_canvas

    Expected:
    - Deserialized workflow has same steps as input
    - All fields preserved (action, params, order)
    - Steps in same order
    """

Step 2: Write test with @given decorator

import pytest
from hypothesis import given, settings, strategies as st
from tests.property_tests.conftest import DEFAULT_PROFILE
from backend.core.workflow_engine import WorkflowDefinition

@pytest.mark.property
@given(st.lists(
    st.fixed_dictionaries({
        'action': st.sampled_from(['create_agent', 'execute_workflow', 'present_canvas']),
        'params': st.dictionaries(st.text(), st.text()),
        'order': st.integers(min_value=0, max_value=1000)
    }),
    min_size=0,
    max_size=100
))
@settings(DEFAULT_PROFILE)
def test_workflow_serialization_roundtrip(steps):
    """
    Test that workflow serialization is lossless for all step lists.

    Invariant: For any list of workflow steps, serializing and deserializing
    produces an equivalent workflow with the same steps.

    Strategy: Generate 0-100 workflow steps with random actions, params, order.

    Expected: All steps preserved in same order.
    """
    # Arrange: Create workflow from generated steps
    workflow = WorkflowDefinition(name="test", steps=steps)

    # Act: Serialize and deserialize
    serialized = workflow.serialize()
    deserialized = WorkflowDefinition.deserialize(serialized)

    # Assert: Invariant holds
    assert len(deserialized.steps) == len(steps), \
        f"Step count mismatch: {len(deserialized.steps)} != {len(steps)}"

    for i, (original, recovered) in enumerate(zip(steps, deserialized.steps)):
        assert recovered['action'] == original['action'], \
            f"Step {i}: action mismatch ({recovered['action']} != {original['action']})"
        assert recovered['order'] == original['order'], \
            f"Step {i}: order mismatch ({recovered['order']} != {original['order']})"
        assert recovered['params'] == original['params'], \
            f"Step {i}: params mismatch ({recovered['params']} != {original['params']})"

Step 3: Run test and verify

# Run property test
pytest backend/tests/property_tests/test_workflow_properties.py::test_workflow_serialization_roundtrip -v

# Hypothesis will:
# 1. Generate 100+ random examples (by default)
# 2. Shrink counterexample to minimal case (if invariant violated)
# 3. Print minimal failing input (for bug filing)

# Example output on failure:
# Falsifying example:
# test_workflow_serialization_roundtrip(
#     steps=[
#         {'action': 'create_agent', 'params': {}, 'order': 0},
#         {'action': 'execute_workflow', 'params': {'name': ''}, 'order': 1}
#     ]
# )
# Shrunk from 53 steps to 2 steps in 0.05s

Step 4: Handle invariant violations

# If test fails, Hypothesis provides minimal counterexample
# Example: Steps with empty 'name' param cause serialization error

# Fix bug or refine invariant
# Option 1: Fix bug in serialization logic
# Option 2: Add precondition: 'name' must be non-empty
# Option 3: Update invariant to handle empty names correctly

# Example: Add precondition to strategy
@given(st.lists(
    st.fixed_dictionaries({
        'action': st.sampled_from(['create_agent', 'execute_workflow', 'present_canvas']),
        'params': st.dictionaries(
            st.text(min_size=1, max_size=10),  # Non-empty keys
            st.text(min_size=1)  # Non-empty values (precondition)
        ),
        'order': st.integers(min_value=0, max_value=1000)
    }),
    min_size=0,
    max_size=100
))
@settings(DEFAULT_PROFILE)
def test_workflow_serialization_roundtrip_with_preconditions(steps):
    """
    Test that workflow serialization is lossless for all step lists.

    Invariant: For any list of workflow steps with non-empty params,
    serializing and deserializing produces an equivalent workflow.

    Precondition: All param keys and values must be non-empty strings.
    """
    # ... same test logic

Expected Behavior

Invariant holds (test passes):

• All generated examples satisfy invariant
• Hypothesis runs 100-200 examples (depending on profile)
• No counterexamples found
• Test completes in <30s (per TQ-03)

Invariant violated (test fails):

• Hypothesis finds counterexample
• Automatically shrinks to minimal failing case
• Prints minimal input that violates invariant
• Provides reproduction script

Example failure output:

# ==================== FAILURES ====================
# ____________________ test_workflow_serialization_roundtrip ____________________
#
# Falsifying example:
# test_workflow_serialization_roundtrip(
#     steps=[
#         {'action': 'execute_workflow', 'params': {'name': ''}, 'order': 0}
#     ]
# )
# Shrunk from 47 steps to 1 step in 0.03s
#
# assert 1 == 0
#  +  where 1 = len([{'action': 'execute_workflow', 'params': {}, 'order': 0}])
#  +  and   0 = len([])
#
# Step 0: params mismatch ({'name': ''} != {})

Hypothesis shrinking process:

1. Find first failing example (may be complex: 53 steps)
2. Simplify example (remove steps, reduce values)
3. Find minimal counterexample (2 steps → 1 step)
4. Report minimal failing case for debugging

Bug Filing

Automatic bug filing on invariant violation:

from tests.bug_discovery.bug_filing_service import BugFilingService

@pytest.mark.property
@given(st.lists(st.integers(), min_size=0, max_size=100))
@settings(DEFAULT_PROFILE)
def test_[invariant_name](inputs):
    """
    Test that [invariant] holds for all [inputs].

    Invariant: [statement]
    Strategy: [strategy description]

    Fails on: [known counterexample]
    """
    try:
        # Test logic
        result = [function_under_test](inputs)
        assert [invariant_check](result), f"Invariant violated: {result}"

    except AssertionError as e:
        # File bug with counterexample
        BugFilingService.file_bug(
            test_name=f"test_{[invariant_name]}_violation",
            error_message=f"Invariant violation: {str(e)}",
            metadata={
                "test_type": "property",
                "invariant": "[invariant_name]",
                "counterexample": str(inputs),
                "shrunk_input": str(inputs),  # Hypothesis already shrunk
                "hypothesis_examples": 100,  # Number of examples run
                "strategy": "st.lists(st.integers(), min_size=0, max_size=100)"
            },
            expected_behavior=f"Invariant should hold: {[invariant_statement]}",
            actual_behavior=f"Invariant violated for input: {inputs}"
        )
        raise  # Re-raise to fail test

Manual bug filing (if not automatic):

# Bug title: [Bug] Invariant violation: [Invariant Name]

# Bug body:
## Bug Description

Property-based test discovered invariant violation in [function_name].

## Invariant

**Statement:** [invariant statement]

**Domain:** [input space]

**Preconditions:** [required conditions]

## Counterexample

```python
# Minimal failing input (shrunk by Hypothesis)
inputs = [paste counterexample from test output]

# Reproducer
from backend.core.[module] import [function_name]
result = [function_name](inputs)
# Expected: [expected behavior]
# Actual: [actual behavior]

Steps to Reproduce

1. Run property test: pytest backend/tests/property_tests/test_[module]_properties.py::test_[invariant_name] -v
2. Hypothesis finds counterexample after N examples
3. Counterexample shrunk to minimal case: [paste input]
4. Invariant violated: [description of violation]

Shrinking Process

• Original failing example: [N] steps/items
• Shrunk to: [M] steps/items (minimal case)
• Shrinking time: [seconds]

Hypothesis Output

[paste Hypothesis output with counterexample]

Expected Behavior

Invariant should hold: [invariant statement]

For input: [counterexample], expected: [expected result]

Actual Behavior

Invariant violated: [description of violation]

For input: [counterexample], actual: [actual result]

Test Context

• Test: test_[invariant_name]
• Hypothesis examples run: [N]
• Strategy: [Hypothesis strategy used]
• Settings: [max_examples, deadline]
• Platform: [output of uname -a]
• Python: [output of python --version]

## TQ Compliance

**TQ-01 (Test Independence):**
- Each test generates fresh inputs (Hypothesis @given decorator)
- No shared state between property tests
- Each invariant tested independently

**TQ-02 (Pass Rate):**
- Property tests have 100% pass rate (invariant violations = real bugs)
- Same input always produces same output (deterministic target function)
- No flaky tests (Hypothesis provides reproducible examples)

**TQ-03 (Performance):**
- Hypothesis settings enforce deadline (30s default)
- CI profile: 50 examples, 5s deadline (fast for PR checks)
- Default profile: 200 examples, 30s deadline (thorough for local)

**TQ-04 (Determinism):**
- Same input produces same output (deterministic target function required)
- Hypothesis uses fixed random seed (reproducible examples)
- Counterexamples are reproducible (same test run = same failure)

**TQ-05 (Coverage Quality):**
- Tests invariant (observable behavior), not implementation
- Hypothesis explores input space systematically (edge cases discovered)
- Property-based: tests general property, not specific examples

## pytest.ini Marker

Add to `backend/pytest.ini`:
```ini
[pytest]
markers =
    property: Property-based tests (Hypothesis, slow, thorough)

Run only property tests:

pytest backend/tests/property_tests/ -v -m property

Skip property tests in fast CI:

pytest backend/tests/ -v -m "not property"

Invariant-First Thinking

Process:

1. Document invariant first (before writing test)
2. Write test that validates invariant
3. Run test to discover counterexamples
4. Fix bugs or refine invariant (add preconditions)
5. Re-run test to verify fix

Why invariant-first?

• Forces clarity about what must be true
• Prevents implementation-driven tests
• Catches edge cases early
• Makes tests maintainable (invariant is documentation)

Bad example (not invariant-first):

# BAD: Test specific examples, no invariant documented
def test_workflow_serialization():
    workflow = Workflow(steps=[{'action': 'create_agent'}])
    serialized = workflow.serialize()
    deserialized = Workflow.deserialize(serialized)
    assert deserialized.steps == workflow.steps

Good example (invariant-first):

# GOOD: Invariant documented, tested for all inputs
@given(st.lists(st.builds(WorkflowStep)))
@settings(DEFAULT_PROFILE)
def test_workflow_serialization_roundtrip(steps):
    """
    Test that workflow serialization is lossless for all step lists.

    Invariant: For any list of workflow steps, serializing and
    deserializing produces an equivalent workflow.
    """
    workflow = Workflow(steps=steps)
    serialized = workflow.serialize()
    deserialized = Workflow.deserialize(serialized)
    assert deserialized.steps == workflow.steps

See Also

• Hypothesis Documentation
• Property-Based Testing
• backend/docs/TEST_QUALITY_STANDARDS.md - TQ-01 through TQ-05
• backend/tests/bug_discovery/TEMPLATES/README.md - Template usage guide