CLI Usage

PromptLab provides a command-line interface (CLI) for managing prompts, versions, tests, and evaluations.

Basic Commands

Prompt Management

# Create a prompt
promptlab prompt create "Weather Forecast" --content "Provide a weather forecast for {location} on {date}" --tags "weather,forecast"

# List all prompts
promptlab prompt list

# Get prompt details
promptlab prompt get <prompt_id>

# Update a prompt
promptlab prompt update <prompt_id> --content "New content" --tags "new,tags"

# Delete a prompt
promptlab prompt delete <prompt_id>

Version Control

# Commit a version
promptlab version commit <prompt_id> --message "Version description"

# List versions
promptlab version list <prompt_id>

# Check out (revert to) a specific version
promptlab version checkout <prompt_id> <version_number>

# Compare versions
promptlab version diff <prompt_id> <version1> <version2>

Testing

# Create a test case
promptlab test create <prompt_id> --input '{"location": "New York", "date": "tomorrow"}' --expected "Expected output"

# List test cases
promptlab test list <prompt_id>

# Run a specific test case
promptlab test run <test_case_id> --llm openai

# Run all test cases for a prompt
promptlab test run-all <prompt_id> --llm openai

# Run an A/B test between two prompts
promptlab test ab <prompt_id_a> <prompt_id_b> --inputs '[{"var": "value1"}, {"var": "value2"}]' --llm openai

Evaluation

# Evaluate a prompt
promptlab eval run <prompt_id> --inputs '[{"var": "value1"}, {"var": "value2"}]' --llm openai

# List available metrics
promptlab eval metrics

# Register a custom metric
promptlab eval register-metric <metric_file.py>

Environment Configuration

The CLI supports environment variables for configuration:

PROMPTLAB_STORAGE: Path to store prompts and related data
PROMPTLAB_OPENAI_API_KEY: OpenAI API key for built-in LLM support
PROMPTLAB_DEFAULT_LLM: Default LLM to use for testing and evaluation

You can also create a config file at ~/.promptlab/config.json:

{
  "storage_path": "/path/to/storage",
  "default_llm": "openai",
  "api_keys": {
    "openai": "your-openai-key"
  }
}

Advanced Usage

Multiple Storage Locations

# Specify a storage location for a command
promptlab --storage /path/to/storage prompt list

# Export a prompt to another storage
promptlab prompt export <prompt_id> --output /path/to/output.json

# Import a prompt from a file
promptlab prompt import /path/to/prompt.json

Automation and Scripting

# Get output in JSON format
promptlab --json prompt list

# Use in shell scripts
PROMPT_ID=$(promptlab --json prompt create "Script Prompt" --content "Content" | jq -r '.id')
echo "Created prompt with ID: $PROMPT_ID"