Coverage for tinytroupe / validation / validation_chamber.py: 0%

1from tinytroupe.experimentation import Proposition 

2 

3 

4class ValidationChamber: 

5 """  

6 An Validation Chamber is a container where autonomous agents can be put to be validated with respect to various custom validation tasks. 

7 

8 Validation tasks types include: 

9 - Question answering: given either a concrete question or a question pattern (to be instantitated via an LLM call), and an expectation,  

10 the agent is expected to answer the question correctly. To check correctness, an LLM-based Proposer is used. 

11 - Behavioral patterns: 

12 * repeated actions - does the agent keep repeating the same action like a crazy person? 

13 * self-consistency - does the agent contradict itself over time? 

14  

15 

16 The class also provides convenience auxiliary methods to: 

17 - generate reasonable question/answer pairs, given some general overall scenario and agent description. 

18 - generate reasonable behavioral patterns, given some general overall scenario and agent description. 

19  

20 """ 

21