Spaces:
Runtime error
Runtime error
| # evals | |
| Tests that check how a model or classifier behaves on labeled examples, not just that the code runs. | |
| | File | What it checks | | |
| |---|---| | |
| | `test_memory_gate.py` | The Jev gate that decides whether a chat message is worth a memory extraction call: the wording, the skip threshold, and 29 labeled messages | | |
| | `test_memory_extraction.py` | The prompt that decides which notes to save from a message, on labeled messages | | |
| | `test_chat_recall.py` | The chat memory organizer: files a chat's rounds under topics, reopens an old topic after a detour, writes synonyms as keywords, and picks the answers worth keeping | | |
| | `test_chat_tool_use.py` | The research agent's tool calls on a real bugged chat (Neo4j follow ups that stayed on one remembered post): does it list or search the whole archive, read the post devoted to the question, and cite it? Set `EVAL_BASELINE=1` to run the prompt from before the fix for comparison | | |
| The always-on tests in each file run with the normal suite. Tests that call a real service are opt-in: they use the keys | |
| from your `.env` (see `live.py`) and only run with `LIVE_EVALS=1`, for example: | |
| LIVE_EVALS=1 pytest tests/evals -k live | |