examples/ai_testing/PROMPT_STRUCTURE.md

📨 Prompt Management Structure

Overview

This document explains how AI prompts are organized and managed in the PyCatan AI system.

Directory Structure

examples/ai_testing/my_games/ai_logs/
└── session_YYYYMMDD_HHMMSS/          # Game session folder
    ├── NH/                            # Player NH
    │   ├── prompts/                   # All prompts for this player
    │   │   ├── prompt_1.json          # First prompt
    │   │   ├── prompt_1.txt           # Human-readable version
    │   │   ├── prompt_2.json          # Second prompt
    │   │   ├── prompt_2.txt
    │   │   └── ...                    # More prompts as game progresses
    │   └── responses/                 # AI responses (to be implemented)
    │       ├── response_1.json
    │       └── ...
    ├── Alex/                          # Player Alex
    │   ├── prompts/
    │   │   ├── prompt_1.json
    │   │   └── ...
    │   └── responses/
    └── Sarah/                         # Player Sarah
        ├── prompts/
        └── responses/

How It Works

1. Prompt Generation

Every time the game state changes and it's a player's turn, the system generates a new prompt:

from examples.ai_testing.generate_prompts_from_state import main as generate_prompts

# This creates numbered prompts for each player
generate_prompts()

2. Prompt Numbering

• Each new prompt gets a sequential number: prompt_1.json, prompt_2.json, etc.
• Always use the highest numbered prompt - it's the most recent one
• Old prompts are kept for debugging and analysis

3. Prompt Format

Each .json file contains:

• response_schema: The expected format for LLM response
• system_instruction: Instructions for the LLM
• prompt: The actual game state and context

4. Getting the Latest Prompt

from examples.ai_testing.generate_prompts_from_state import get_latest_prompt

# Get the most recent prompt for a player
prompt_file, prompt_data = get_latest_prompt("NH")

if prompt_file:
    print(f"Latest prompt: {prompt_file}")
    # Use prompt_data to send to LLM

Or use the example script:

python examples/ai_testing/example_get_latest_prompt.py NH

File Types

JSON Files (prompt_N.json)

Send this to the LLM

• Complete request ready for API
• Includes response schema
• Machine-readable format

TXT Files (prompt_N.txt)

For human inspection

• Formatted for readability
• Includes headers and sections
• Helpful for debugging

Benefits of This Structure

✅ No Overwrites - Each prompt is saved with unique number ✅ Easy to Track - Can see full history of prompts ✅ Simple to Use - Always use the highest number ✅ Player-Specific - Each player has their own folder ✅ Session-Based - Each game session is separate ✅ Debugging - Can compare prompts across turns

Example Usage

Basic Usage

import json
from pathlib import Path
from examples.ai_testing.generate_prompts_from_state import get_latest_prompt

# 1. Get latest prompt for player
prompt_file, prompt_data = get_latest_prompt("NH")

# 2. Extract the prompt
if prompt_data:
    llm_request = prompt_data  # Already in correct format
    
    # 3. Send to LLM (pseudo-code)
    response = llm_client.send(
        system=llm_request["system_instruction"],
        prompt=llm_request["prompt"],
        response_format=llm_request["response_schema"]
    )

Finding All Prompts for a Player

from pathlib import Path

player_name = "NH"
session_dir = Path("examples/ai_testing/my_games/ai_logs/session_20260104_025733")
player_prompts = sorted((session_dir / player_name / "prompts").glob("prompt_*.json"))

print(f"Found {len(player_prompts)} prompts for {player_name}")
for prompt in player_prompts:
    print(f"  - {prompt.name}")

Comparing Two Prompts

import json

# Get two sequential prompts
prompt_1 = session_dir / "NH" / "prompts" / "prompt_1.json"
prompt_2 = session_dir / "NH" / "prompts" / "prompt_2.json"

with open(prompt_1) as f1, open(prompt_2) as f2:
    data1 = json.load(f1)
    data2 = json.load(f2)
    
    # Compare what changed
    turn1 = data1["prompt"]["game_state"]["meta"]["turn"]
    turn2 = data2["prompt"]["game_state"]["meta"]["turn"]
    
    print(f"Prompt 1 was for turn {turn1}")
    print(f"Prompt 2 was for turn {turn2}")

Future Enhancements

• Add responses/ directory for AI agent responses
• Create prompt-response pairs for training
• Add automatic cleanup of old sessions
• Add prompt compression for long games
• Add prompt diff tool to see what changed

Related Files

• generate_prompts_from_state.py - Main prompt generation logic
• example_get_latest_prompt.py - Example usage
• play_with_prompts.py - Auto-generate prompts during gameplay