Add tag categorization pipeline for e621 checklist

Implements structured tag suggestion system that organizes recommendations
by category from the e621 tagging checklist.

Key features:
- Parses e621 checklist into structured categories with constraints
- Defines processing tiers: FOUNDATIONAL → CHARACTER → APPEARANCE → SCENE → META
- Implements dependency logic (e.g., skip character tags for zero_pictured)
- Integrates with existing TF-IDF ranking for similarity-based suggestions
- Categories have constraint types: exactly_one, at_most_one, multi_select

Modules:
- psq_rag/tagging/category_parser.py: Parses checklist into TagCategory objects
- psq_rag/tagging/categorized_suggestions.py: Generates categorized suggestions
- scripts/test_parser_only.py: Unit tests for parser (all passing)

Workflow:
1. LLM predicts initial tags from prompt
2. System uses those tags to tf-idf rank remaining tags
3. Results organized by category for user review/selection

https://claude.ai/code/session_015ZwE7a5E6YVTrMpuB2pXX7

psq_rag/tagging/categorized_suggestions.py

@@ -0,0 +1,289 @@
+"""
+Generate categorized tag suggestions using TF-IDF similarity rankings.
+
+This module takes LLM-selected tags and generates organized suggestions
+for each category from the e621 checklist, ranked by TF-IDF similarity.
+"""
+from __future__ import annotations
+
+from collections import OrderedDict
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Set, Tuple, Optional
+
+from .category_parser import (
+    CategoryTier,
+    ConstraintType,
+    TagCategory,
+    parse_checklist,
+    should_skip_category,
+)
+from ..retrieval.psq_retrieval import get_tfidf_reduced_similar_tags
+
+
+@dataclass
+class CategorySuggestions:
+    """Suggestions for a single category."""
+    category: TagCategory
+    suggestions: List[Tuple[str, float]]  # [(tag, score), ...]
+    already_selected: Set[str]  # Tags from this category already in selected_tags
+
+
+@dataclass
+class CategorizedTagSuggestions:
+    """Organized tag suggestions by category."""
+    by_category: Dict[str, CategorySuggestions]  # category_name -> suggestions
+    other_suggestions: List[Tuple[str, float]]   # Tags not in any category
+    categories: Dict[str, TagCategory]            # All category definitions
+
+
+def load_categories(checklist_path: Optional[Path] = None) -> Dict[str, TagCategory]:
+    """
+    Load and parse category definitions from checklist.
+
+    Args:
+        checklist_path: Path to checklist file. If None, uses default location.
+
+    Returns:
+        Dict mapping category_name -> TagCategory
+    """
+    if checklist_path is None:
+        # Try to find it in the git repo from the other branch
+        import subprocess
+        try:
+            result = subprocess.run(
+                ['git', 'show', 'origin/claude/prompt-squirrel-rag-3PZn7:tagging_checklist.txt'],
+                capture_output=True,
+                text=True,
+                cwd=Path(__file__).parent.parent.parent
+            )
+            if result.returncode == 0:
+                # Write to temp file
+                import tempfile
+                with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as f:
+                    f.write(result.stdout)
+                    checklist_path = Path(f.name)
+        except Exception:
+            pass
+
+    if checklist_path is None or not checklist_path.exists():
+        raise FileNotFoundError(
+            "Could not find tagging_checklist.txt. "
+            "Please provide checklist_path or ensure it's in the repository."
+        )
+
+    return parse_checklist(checklist_path)
+
+
+def build_category_tag_index(categories: Dict[str, TagCategory]) -> Dict[str, str]:
+    """
+    Build reverse index: tag -> category_name.
+
+    Args:
+        categories: Category definitions
+
+    Returns:
+        Dict mapping tag -> category_name
+    """
+    tag_to_category = {}
+    for cat_name, category in categories.items():
+        for tag in category.tags:
+            # Normalize tag (the checklist has underscores, TF-IDF might have spaces)
+            normalized = tag.replace('_', ' ')
+            tag_to_category[normalized] = cat_name
+            tag_to_category[tag] = cat_name
+
+    return tag_to_category
+
+
+def generate_categorized_suggestions(
+    selected_tags: List[str],
+    *,
+    allow_nsfw_tags: bool = False,
+    top_n_per_category: int = 10,
+    top_n_other: int = 50,
+    checklist_path: Optional[Path] = None,
+) -> CategorizedTagSuggestions:
+    """
+    Generate tag suggestions organized by category.
+
+    Args:
+        selected_tags: Tags already selected/predicted by LLM
+        allow_nsfw_tags: Whether to include NSFW suggestions
+        top_n_per_category: Maximum suggestions per category
+        top_n_other: Maximum suggestions in "Other" category
+        checklist_path: Optional path to checklist file
+
+    Returns:
+        CategorizedTagSuggestions with organized suggestions
+    """
+    # Load category definitions
+    categories = load_categories(checklist_path)
+    tag_to_category = build_category_tag_index(categories)
+
+    # Get TF-IDF similarity scores for all tags based on selected tags
+    from collections import Counter
+    # Normalize selected tags (spaces -> underscores for TF-IDF)
+    normalized_selected = [tag.replace(' ', '_') for tag in selected_tags]
+    pseudo_doc = Counter(normalized_selected)
+
+    # Get all similar tags with scores
+    all_suggestions_ordered = get_tfidf_reduced_similar_tags(
+        pseudo_doc,
+        allow_nsfw_tags=allow_nsfw_tags
+    )
+
+    # Convert to list of (tag, score)
+    all_suggestions = list(all_suggestions_ordered.items())
+
+    # Track which tags are already selected
+    selected_set = set(selected_tags)
+    # Also check normalized versions
+    selected_set.update(tag.replace('_', ' ') for tag in selected_tags)
+    selected_set.update(tag.replace(' ', '_') for tag in selected_tags)
+
+    # Organize suggestions by category
+    by_category: Dict[str, List[Tuple[str, float]]] = {
+        cat_name: [] for cat_name in categories.keys()
+    }
+    other_suggestions: List[Tuple[str, float]] = []
+
+    for tag, score in all_suggestions:
+        # Skip if already selected
+        if tag in selected_set or tag.replace('_', ' ') in selected_set:
+            continue
+
+        # Find category
+        cat_name = tag_to_category.get(tag)
+        if cat_name:
+            by_category[cat_name].append((tag, score))
+        else:
+            other_suggestions.append((tag, score))
+
+    # Sort each category by score and limit to top N
+    sorted_by_category: Dict[str, CategorySuggestions] = {}
+
+    # Process in tier order
+    tier_order = sorted(categories.values(), key=lambda c: c.tier.value)
+
+    for category in tier_order:
+        cat_name = category.name
+
+        # Check if category should be skipped based on dependencies
+        if should_skip_category(category, selected_set, categories):
+            continue
+
+        suggestions = by_category.get(cat_name, [])
+
+        # Already sorted by score from TF-IDF, just limit
+        suggestions = suggestions[:top_n_per_category]
+
+        # Find tags from this category already selected
+        already_selected = set()
+        for tag in category.tags:
+            if tag in selected_set or tag.replace('_', ' ') in selected_set:
+                already_selected.add(tag)
+
+        sorted_by_category[cat_name] = CategorySuggestions(
+            category=category,
+            suggestions=suggestions,
+            already_selected=already_selected,
+        )
+
+    # Limit "Other" suggestions
+    other_suggestions = other_suggestions[:top_n_other]
+
+    return CategorizedTagSuggestions(
+        by_category=sorted_by_category,
+        other_suggestions=other_suggestions,
+        categories=categories,
+    )
+
+
+def format_suggestions_for_display(
+    categorized: CategorizedTagSuggestions,
+    show_scores: bool = True,
+) -> str:
+    """
+    Format categorized suggestions for display to user.
+
+    Args:
+        categorized: The categorized suggestions
+        show_scores: Whether to show similarity scores
+
+    Returns:
+        Formatted string for display
+    """
+    lines = []
+
+    # Process categories in tier order
+    tier_groups = {}
+    for cat_name, cat_sugg in categorized.by_category.items():
+        tier = cat_sugg.category.tier
+        if tier not in tier_groups:
+            tier_groups[tier] = []
+        tier_groups[tier].append((cat_name, cat_sugg))
+
+    for tier in sorted(tier_groups.keys(), key=lambda t: t.value):
+        tier_name = tier.name.title()
+        lines.append(f"\n{'='*60}")
+        lines.append(f"{tier_name} Categories")
+        lines.append('='*60)
+
+        for cat_name, cat_sugg in tier_groups[tier]:
+            category = cat_sugg.category
+
+            # Header
+            lines.append(f"\n{category.display_name}")
+            lines.append(f"  Constraint: {category.constraint.value}")
+
+            # Already selected tags
+            if cat_sugg.already_selected:
+                selected_str = ', '.join(sorted(cat_sugg.already_selected))
+                lines.append(f"  ✓ Selected: {selected_str}")
+
+            # Suggestions
+            if cat_sugg.suggestions:
+                lines.append("  Suggestions:")
+                for tag, score in cat_sugg.suggestions:
+                    if show_scores:
+                        lines.append(f"    • {tag} ({score:.3f})")
+                    else:
+                        lines.append(f"    • {tag}")
+            else:
+                lines.append("  (no suggestions)")
+
+    # Other suggestions
+    if categorized.other_suggestions:
+        lines.append(f"\n{'='*60}")
+        lines.append("Other Tags")
+        lines.append('='*60)
+        for tag, score in categorized.other_suggestions[:20]:  # Show top 20
+            if show_scores:
+                lines.append(f"  • {tag} ({score:.3f})")
+            else:
+                lines.append(f"  • {tag}")
+
+    return '\n'.join(lines)
+
+
+def get_category_suggestions_dict(
+    categorized: CategorizedTagSuggestions
+) -> Dict[str, List[str]]:
+    """
+    Get simple dict of category -> suggested tags (without scores).
+
+    Args:
+        categorized: The categorized suggestions
+
+    Returns:
+        Dict mapping category_name -> [tag1, tag2, ...]
+    """
+    result = {}
+
+    for cat_name, cat_sugg in categorized.by_category.items():
+        result[cat_name] = [tag for tag, _ in cat_sugg.suggestions]
+
+    result['other'] = [tag for tag, _ in categorized.other_suggestions]
+
+    return result

psq_rag/tagging/category_parser.py

@@ -0,0 +1,319 @@
+"""
+Parse e621 tagging checklist into structured category definitions.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from enum import Enum
+from pathlib import Path
+from typing import Dict, List, Set, Optional
+
+
+class ConstraintType(Enum):
+    """How many tags from this category can be selected."""
+    EXACTLY_ONE = "exactly_one"  # e.g., rating: safe/questionable/explicit
+    AT_MOST_ONE = "at_most_one"  # e.g., primary species (can have multiple for multi-character scenes)
+    MULTI_SELECT = "multi"        # e.g., perspectives, expressions
+    OPTIONAL = "optional"         # may or may not apply
+
+
+class CategoryTier(Enum):
+    """Processing order for categories based on dependencies."""
+    FOUNDATIONAL = 1   # count, rating
+    CHARACTER = 2      # body_type, species, gender
+    APPEARANCE = 3     # clothing, pose, expression
+    SCENE = 4          # location, perspective
+    META = 5           # quality, style, resolution
+
+
+@dataclass
+class TagCategory:
+    """Structured representation of a tag category from the checklist."""
+    name: str
+    display_name: str
+    description: str
+    tags: List[str]
+    constraint: ConstraintType
+    tier: CategoryTier
+    depends_on: List[str] = field(default_factory=list)  # Category names this depends on
+    skip_if: Dict[str, Set[str]] = field(default_factory=dict)  # {category: {values}} to skip
+
+
+def parse_checklist(checklist_path: Path) -> Dict[str, TagCategory]:
+    """
+    Parse the e621 tagging checklist text file into TagCategory objects.
+
+    Returns:
+        Dict mapping category_name -> TagCategory
+    """
+    with open(checklist_path, 'r', encoding='utf-8') as f:
+        lines = f.readlines()
+
+    categories = {}
+
+    # Track current section
+    current_section = None
+    in_basics = False
+    in_explicit = False
+    in_pose = False
+    in_info = False
+
+    for i, line in enumerate(lines):
+        stripped = line.strip()
+
+        # Skip header/navigation
+        if i < 10:
+            continue
+
+        # Section headers
+        if stripped == "Basics":
+            in_basics = True
+            in_explicit = in_pose = in_info = False
+            continue
+        elif stripped == "Sexually explicit":
+            in_explicit = True
+            in_basics = in_pose = in_info = False
+            continue
+        elif stripped == "Pose / Activity / Appearance":
+            in_pose = True
+            in_basics = in_explicit = in_info = False
+            continue
+        elif stripped == "Information and Requests":
+            in_info = True
+            in_basics = in_explicit = in_pose = False
+            continue
+        elif stripped in ("Heavily vetted tags.", "Do NOT tag"):
+            break  # Stop parsing
+
+        # Skip empty lines
+        if not stripped:
+            continue
+
+        # Parse category lines (those with question marks)
+        # They must be indented in the original (start with spaces)
+        if "?" in stripped and line.startswith("    "):
+            # Extract category question and tags
+            parts = stripped.split("?", 1)
+            if len(parts) != 2:
+                continue
+
+            question = parts[0].strip()
+            tags_text = parts[1].strip()
+
+            # Skip excluded categories
+            if question in ("Artist(s)", "Copyright", "Character", "Year of creation"):
+                continue
+
+            # Skip sexually explicit categories entirely
+            if in_explicit:
+                continue
+
+            # Special handling for Rating (tags are on following lines, not in description)
+            if question == "Rating":
+                tags = ["safe", "questionable", "explicit"]
+            else:
+                # Parse tags from the description
+                # Tags are either comma-separated or in parentheses
+                tags = parse_tags_from_description(tags_text)
+
+            if not tags:
+                continue
+
+            # Determine category metadata
+            category_name = normalize_category_name(question)
+            tier = determine_tier(question, in_basics, in_pose, in_info)
+            constraint = determine_constraint(question, tags)
+
+            # Debug: print categorization
+            # print(f"DEBUG: '{question}' -> tier={tier.name}, constraint={constraint.value}")
+
+            category = TagCategory(
+                name=category_name,
+                display_name=question,
+                description=tags_text,
+                tags=tags,
+                constraint=constraint,
+                tier=tier,
+            )
+
+            categories[category_name] = category
+
+    # Add dependency information
+    add_dependencies(categories)
+
+    return categories
+
+
+def parse_tags_from_description(text: str) -> List[str]:
+    """Extract individual tags from a description string."""
+    # Remove parenthetical explanations
+    text = re.sub(r'\([^)]*\)', '', text)
+
+    # Remove reference text like "See also:", "Common ones:", etc.
+    text = re.sub(r'(See also|Common ones|For more|click for)[^.]*\.', '', text, flags=re.IGNORECASE)
+
+    # Split by comma
+    parts = [p.strip() for p in text.split(',')]
+
+    tags = []
+    for part in parts:
+        # Clean up
+        part = part.strip()
+        if not part:
+            continue
+
+        # Remove "tag group:" prefixes
+        if part.startswith('tag group:'):
+            continue
+
+        # Remove explanatory text after tags
+        if ' for ' in part:
+            part = part.split(' for ')[0]
+        if ' if ' in part:
+            part = part.split(' if ')[0]
+        if ' such as ' in part:
+            part = part.split(' such as ')[0]
+
+        part = part.strip()
+
+        # Extract tags - can be multi-word with underscores or hyphens
+        # Tags can have parentheses like pencil_(artwork)
+        match = re.match(r'^([a-z_0-9-]+(?:_[a-z_0-9-]+)*(?:_\([a-z]+\))?)', part)
+        if match:
+            tags.append(match.group(1))
+
+    return tags
+
+
+def normalize_category_name(question: str) -> str:
+    """Convert question to category name."""
+    # Remove question mark and common words
+    name = question.lower().replace('?', '').strip()
+    name = name.replace('/', '_').replace(' ', '_')
+    name = name.replace('(', '').replace(')', '')
+
+    # Simplify common names
+    simplifications = {
+        'sex_gender': 'gender',
+        'how_many': 'count',
+        'quality_medium': 'quality',
+        'picture_organization': 'organization',
+        'text_and_languages': 'text',
+        'image_size': 'resolution',
+    }
+
+    return simplifications.get(name, name)
+
+
+def determine_tier(question: str, in_basics: bool, in_pose: bool, in_info: bool) -> CategoryTier:
+    """Determine processing tier based on question and section."""
+    q_lower = question.lower().rstrip('?')  # Remove trailing question mark if present
+
+    # Foundational
+    if q_lower in ('how many', 'rating'):
+        return CategoryTier.FOUNDATIONAL
+
+    # Character properties
+    if q_lower in ('body type', 'species', 'sex/gender'):
+        return CategoryTier.CHARACTER
+
+    # Appearance
+    if in_pose or q_lower in ('clothing', 'posture', 'general activity (if any)',
+                               'body decor', 'fur style', 'hair', 'breasts',
+                               'limbs', 'gaze', 'expression'):
+        return CategoryTier.APPEARANCE
+
+    # Scene
+    if q_lower in ('location', 'perspective'):
+        return CategoryTier.SCENE
+
+    # Meta/info
+    if in_info or q_lower in ('quality/medium', 'picture organization', 'style',
+                               'text and languages', 'information', 'requests',
+                               'image size'):
+        return CategoryTier.META
+
+    # Default
+    return CategoryTier.META
+
+
+def determine_constraint(question: str, tags: List[str]) -> ConstraintType:
+    """Determine selection constraint based on category semantics."""
+    q_lower = question.lower().rstrip('?')  # Remove trailing question mark if present
+
+    # Exactly one
+    if q_lower in ('rating', 'how many', 'body type'):
+        return ConstraintType.EXACTLY_ONE
+
+    # At most one per character (but can have multiple in multi-character scenes)
+    if q_lower in ('species', 'sex/gender'):
+        return ConstraintType.AT_MOST_ONE
+
+    # Multi-select
+    if q_lower in ('clothing', 'perspective', 'location', 'limbs',
+                   'expression', 'general activity (if any)', 'gaze',
+                   'posture', 'body decor', 'fur style', 'hair',
+                   'breasts', 'text and languages', 'quality/medium', 'style',
+                   'picture organization', 'information', 'requests',
+                   'image size'):
+        return ConstraintType.MULTI_SELECT
+
+    # Default optional
+    return ConstraintType.OPTIONAL
+
+
+def add_dependencies(categories: Dict[str, TagCategory]) -> None:
+    """Add dependency and skip rules to categories."""
+
+    # If zero_pictured, skip character/appearance categories
+    character_appearance_categories = [
+        'body_type', 'species', 'gender', 'clothing', 'posture',
+        'activity', 'body_decor', 'fur_style', 'hair', 'breasts',
+        'limbs', 'gaze', 'expression'
+    ]
+
+    for cat_name in character_appearance_categories:
+        if cat_name in categories:
+            categories[cat_name].skip_if['count'] = {'zero_pictured'}
+
+    # Character properties depend on having count first
+    for cat_name in ['body_type', 'species', 'gender']:
+        if cat_name in categories:
+            categories[cat_name].depends_on.append('count')
+
+    # Appearance depends on character properties
+    appearance_cats = ['clothing', 'posture', 'activity', 'body_decor',
+                       'fur_style', 'hair', 'breasts', 'limbs', 'gaze', 'expression']
+    for cat_name in appearance_cats:
+        if cat_name in categories:
+            categories[cat_name].depends_on.extend(['count', 'body_type'])
+
+
+def should_skip_category(category: TagCategory, selected_tags: Set[str],
+                         categories: Dict[str, TagCategory]) -> bool:
+    """
+    Check if a category should be skipped based on already-selected tags.
+
+    Args:
+        category: The category to check
+        selected_tags: Tags already selected
+        categories: All categories for dependency resolution
+
+    Returns:
+        True if category should be skipped
+    """
+    for dep_category_name, skip_values in category.skip_if.items():
+        dep_category = categories.get(dep_category_name)
+        if not dep_category:
+            continue
+
+        # Check if any of the selected tags from the dependency category
+        # are in the skip_values set
+        dep_tags = set(dep_category.tags)
+        selected_dep_tags = selected_tags & dep_tags
+
+        if selected_dep_tags & skip_values:
+            return True
+
+    return False

scripts/test_categorized_suggestions.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""
+Test script for categorized tag suggestions.
+"""
+from pathlib import Path
+import sys
+
+# Add parent directory to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from psq_rag.tagging.category_parser import parse_checklist
+from psq_rag.tagging.categorized_suggestions import (
+    generate_categorized_suggestions,
+    format_suggestions_for_display,
+    load_categories,
+)
+
+
+def test_parse_checklist():
+    """Test parsing the checklist file."""
+    print("=" * 80)
+    print("Testing checklist parsing...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+
+    if not checklist_path.exists():
+        print(f"ERROR: Checklist not found at {checklist_path}")
+        return False
+
+    categories = parse_checklist(checklist_path)
+
+    print(f"\nParsed {len(categories)} categories:")
+    for cat_name, category in categories.items():
+        print(f"\n  {cat_name}:")
+        print(f"    Display: {category.display_name}")
+        print(f"    Tier: {category.tier.name}")
+        print(f"    Constraint: {category.constraint.value}")
+        print(f"    Tags: {len(category.tags)} tags")
+        print(f"    Sample tags: {category.tags[:5]}")
+        if category.depends_on:
+            print(f"    Depends on: {category.depends_on}")
+        if category.skip_if:
+            print(f"    Skip if: {category.skip_if}")
+
+    return True
+
+
+def test_categorized_suggestions():
+    """Test generating categorized suggestions."""
+    print("\n" + "=" * 80)
+    print("Testing categorized suggestions...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+
+    # Example: User prompt resulted in these LLM-selected tags
+    selected_tags = [
+        "anthro",
+        "canine",
+        "male",
+        "solo",
+        "forest",
+        "standing",
+    ]
+
+    print(f"\nSelected tags: {', '.join(selected_tags)}")
+    print("\nGenerating categorized suggestions...")
+
+    try:
+        categorized = generate_categorized_suggestions(
+            selected_tags,
+            allow_nsfw_tags=False,
+            top_n_per_category=5,
+            top_n_other=10,
+            checklist_path=checklist_path,
+        )
+
+        print("\nFormatted output:")
+        print(format_suggestions_for_display(categorized, show_scores=True))
+
+        return True
+
+    except Exception as e:
+        print(f"ERROR: {e}")
+        import traceback
+        traceback.print_exc()
+        return False
+
+
+def test_zero_pictured():
+    """Test that character categories are skipped for zero_pictured."""
+    print("\n" + "=" * 80)
+    print("Testing zero_pictured dependency logic...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+
+    selected_tags = [
+        "zero_pictured",
+        "forest",
+        "outside",
+    ]
+
+    print(f"\nSelected tags: {', '.join(selected_tags)}")
+    print("(Should skip character/appearance categories)")
+
+    try:
+        categorized = generate_categorized_suggestions(
+            selected_tags,
+            allow_nsfw_tags=False,
+            top_n_per_category=5,
+            top_n_other=10,
+            checklist_path=checklist_path,
+        )
+
+        print("\nCategories with suggestions:")
+        for cat_name, cat_sugg in categorized.by_category.items():
+            if cat_sugg.suggestions or cat_sugg.already_selected:
+                print(f"  {cat_name}: {len(cat_sugg.suggestions)} suggestions")
+
+        # Check that character categories are empty
+        character_cats = ['body_type', 'species', 'gender', 'clothing']
+        all_skipped = True
+        for cat in character_cats:
+            if cat in categorized.by_category:
+                if categorized.by_category[cat].suggestions:
+                    print(f"  WARNING: {cat} should have been skipped!")
+                    all_skipped = False
+
+        if all_skipped:
+            print("\n✓ All character categories correctly skipped!")
+
+        return all_skipped
+
+    except Exception as e:
+        print(f"ERROR: {e}")
+        import traceback
+        traceback.print_exc()
+        return False
+
+
+if __name__ == "__main__":
+    success = True
+
+    success &= test_parse_checklist()
+    success &= test_categorized_suggestions()
+    success &= test_zero_pictured()
+
+    print("\n" + "=" * 80)
+    if success:
+        print("✓ All tests passed!")
+    else:
+        print("✗ Some tests failed")
+    print("=" * 80)
+
+    sys.exit(0 if success else 1)

scripts/test_parser_only.py

@@ -0,0 +1,166 @@
+#!/usr/bin/env python3
+"""
+Test script for category parser only (no TF-IDF dependencies).
+"""
+from pathlib import Path
+import sys
+
+# Add parent directory to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from psq_rag.tagging.category_parser import parse_checklist, should_skip_category
+
+
+def test_parse_checklist():
+    """Test parsing the checklist file."""
+    print("=" * 80)
+    print("Testing checklist parsing...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+
+    if not checklist_path.exists():
+        print(f"ERROR: Checklist not found at {checklist_path}")
+        return False
+
+    categories = parse_checklist(checklist_path)
+
+    print(f"\nParsed {len(categories)} categories:\n")
+
+    # Group by tier
+    by_tier = {}
+    for cat_name, category in categories.items():
+        tier = category.tier.name
+        if tier not in by_tier:
+            by_tier[tier] = []
+        by_tier[tier].append((cat_name, category))
+
+    for tier_name in sorted(by_tier.keys()):
+        print(f"\n{'='*60}")
+        print(f"{tier_name} Tier")
+        print('='*60)
+
+        for cat_name, category in by_tier[tier_name]:
+            print(f"\n  Category: {cat_name}")
+            print(f"    Display: {category.display_name}")
+            print(f"    Constraint: {category.constraint.value}")
+            print(f"    Tags ({len(category.tags)}): {', '.join(category.tags[:8])}")
+            if len(category.tags) > 8:
+                print(f"      ... and {len(category.tags) - 8} more")
+            if category.depends_on:
+                print(f"    Depends on: {category.depends_on}")
+            if category.skip_if:
+                print(f"    Skip if: {category.skip_if}")
+
+    return True
+
+
+def test_skip_logic():
+    """Test the skip logic for zero_pictured."""
+    print("\n" + "=" * 80)
+    print("Testing skip logic...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+    categories = parse_checklist(checklist_path)
+
+    # Test 1: zero_pictured should skip character categories
+    selected_tags = {'zero_pictured', 'forest', 'outside'}
+
+    character_cats = ['body_type', 'species', 'gender', 'clothing']
+    print(f"\nTest 1: Selected tags = {selected_tags}")
+    print("Should skip character/appearance categories:")
+
+    all_correct = True
+    for cat_name in character_cats:
+        if cat_name in categories:
+            should_skip = should_skip_category(
+                categories[cat_name],
+                selected_tags,
+                categories
+            )
+            status = "✓ SKIP" if should_skip else "✗ KEEP"
+            print(f"  {cat_name}: {status}")
+            if not should_skip:
+                all_correct = False
+
+    # Test 2: solo should NOT skip character categories
+    selected_tags = {'solo', 'anthro', 'male'}
+    print(f"\nTest 2: Selected tags = {selected_tags}")
+    print("Should NOT skip character categories:")
+
+    for cat_name in character_cats:
+        if cat_name in categories:
+            should_skip = should_skip_category(
+                categories[cat_name],
+                selected_tags,
+                categories
+            )
+            status = "✓ KEEP" if not should_skip else "✗ SKIP"
+            print(f"  {cat_name}: {status}")
+            if should_skip:
+                all_correct = False
+
+    return all_correct
+
+
+def test_tag_extraction():
+    """Test that tags are extracted correctly from descriptions."""
+    print("\n" + "=" * 80)
+    print("Testing tag extraction...")
+    print("=" * 80)
+
+    checklist_path = Path(__file__).parent.parent / "tagging_checklist.txt"
+    categories = parse_checklist(checklist_path)
+
+    # Check specific categories we care about
+    test_cases = {
+        'count': ['solo', 'duo', 'trio', 'group', 'zero_pictured'],
+        'rating': ['safe', 'questionable', 'explicit'],  # These might not parse correctly
+        'body_type': ['anthro', 'feral', 'humanoid', 'taur'],
+        'species': ['human', 'canine', 'feline', 'equine'],
+        'gender': ['male', 'female', 'intersex', 'ambiguous_gender'],
+        'clothing': ['fully_clothed', 'partially_clothed', 'nude'],
+        'location': ['inside', 'outside', 'bedroom', 'kitchen', 'forest'],
+    }
+
+    all_correct = True
+    for cat_name, expected_tags in test_cases.items():
+        if cat_name not in categories:
+            print(f"\n✗ Category '{cat_name}' not found!")
+            all_correct = False
+            continue
+
+        category = categories[cat_name]
+        found_tags = set(category.tags)
+        expected_set = set(expected_tags)
+
+        missing = expected_set - found_tags
+        extra = found_tags - expected_set
+
+        if missing:
+            print(f"\n{cat_name}:")
+            print(f"  ✗ Missing expected tags: {missing}")
+            all_correct = False
+        else:
+            print(f"\n✓ {cat_name}: All expected tags found")
+            print(f"    Tags: {', '.join(sorted(category.tags))}")
+
+    return all_correct
+
+
+if __name__ == "__main__":
+    success = True
+
+    success &= test_parse_checklist()
+    success &= test_tag_extraction()
+    success &= test_skip_logic()
+
+    print("\n" + "=" * 80)
+    if success:
+        print("✓ All tests passed!")
+    else:
+        print("✗ Some tests failed")
+    print("=" * 80)
+
+    sys.exit(0 if success else 1)

tagging_checklist.txt

@@ -0,0 +1,76 @@
+ e621:tagging checklist (locked)
+
+[Back: e621:index]
+
+This is an informal and unofficial supplement to the tagging rules and guidelines, meant to encourage better and more complete tagging.
+
+Make sure you're also familiar with our Tag What You See policy before editing tags: tag_what_you_see for the policy itself, and e621:Tag What You See (Explained) for a more in-depth explanation why we use TWYS.
+
+Each entry below poses a general question about a post, with some example tags that answer it. A good post will probably have most of these answered (but not necessarily all).
+Basics
+
+Tags that all posts should have, to maintain minimal searchability.
+
+    Artist(s)? Use their best known alias. If a picture has more than one artist, tag them all, along with collaboration. If you're not sure who the artist is, tag unknown_artist. If the artist wishes to remain anonymous, use anonymous_artist instead.
+    Rating?
+        Explicit for fully or partially exposed genitalia (penis, pussy, cloaca, sheath, balls, or anus), various sex acts even if no genitalia are visible, high amounts of violence/gore, sexual fluids such as cum or pussy_juice, and extreme sexual fetishes such as scat, watersports, or BDSM.
+        Safe for anything that can be viewed in public without much uproar: no genitals, no sexual overtones or poses, no realistic violence, or any questionable activity.
+        Questionable for everything in between, such as topless females and suggestive poses.
+            For more help on ratings please see e621: Ratings
+    Copyright? The original series or company a character or game is owned by.
+    Character? Tag the character's best known name. If not that, their full name. For more, see howto:tag_characters.
+    Body type? anthro, feral, humanoid, taur, anthrofied (pokemorph, digimorph), ponified, feralized
+    Species? human, canine, feline, bovine, cervine, equine, lagomorph, rodent, avian, insect, marine (cetacean, shark), scalie (click for detailed lists)
+    Sex/gender? male, female, intersex (herm, maleherm, gynomorph, andromorph), ambiguous_gender
+        See How To: Tag Genders for a detailed guide
+    How many? solo, duo, trio, group, zero_pictured
+    Clothing? fully_clothed, partially_clothed, skimpy, nude, bottomless, topless, underwear, open_shirt
+    Location? inside, outside, bedroom, kitchen, forest
+    Perspective? front_view, rear_view, side_view, three-quarter_view, low-angle_view, high-angle_view, worm's-eye_view, bird's-eye_view, first_person_view
+
+Sexually explicit
+
+    Male bits? penis, balls, sheath, knot, erection, half-erect, flaccid, humanoid_penis, equine_penis, tapering_penis, veiny_penis, uncut, circumcised
+    Female bits? pussy, clitoris, plump_labia, equine_pussy, canine_pussy
+    Other? butt, anus, puffy_anus, gaping_anus, urethra, genital_slit
+    Sex act? sex (male/female, female/female, male/male, bisexual), masturbation, handjob, footjob, fellatio, cunnilingus, vaginal_penetration, anal_penetration, threesome, foursome, orgy, gangbang, frottage, tribadism, orgasm, cum_inside
+    Position? Common ones: missionary_position, cowgirl_position, reverse_cowgirl_position, from_behind, 69_position, stand_and_carry_position.
+        See also: tag group:sex positions
+    Sexual themes? bondage, domination, rape, rough_sex, happy_sex, presenting, internal, impregnation, bestiality, interspecies, public, exhibitionism
+    Fluids? cum, cumshot, precum, pussy_juice, pussy_ejaculation, saliva
+    Toys? dildo, vibrator, buttplug, egg_vibrator, strapon, feeldoe
+
+Pose / Activity / Appearance
+
+    General activity (if any)? walking, running, fighting, sleeping, dancing, eating, kissing, licking
+    Posture? standing, bent_over, sitting, crouching, kneeling, all_fours, on_front, on_side, on_back, ass_up (see tag group:pose for full list)
+    Body decor? glasses, ring, necklace, bracelet, anklet, tattoo, piercing, collar, hat
+    Fur style? mane, chest_tuft, pubes
+    Hair? hair, long hair, short hair
+    Breasts? breasts (small_breasts, big_breasts, huge_breasts), nipples, under_boob, side_boob, teats
+    Limbs? crossed_arms, raised_arms, arms_behind_head, spread_legs, crossed_legs, raised_leg, legs_up, raised_tail, tailwag
+    Gaze? looking_at_viewer, looking_back, eye_contact, eyes_closed
+    Expression? blush, wink, smile, grin, tongue_out, naughty_face, embarrassed, happy, sad
+
+Information and Requests
+
+    Quality/medium? sketch, line_art, monochrome, shaded, pencil_(artwork), watercolor, 3D, digital_media_(artwork)
+    Picture organization? comic, multiple_scenes, sequence, close-up, portrait, pinup, solo_focus, wallpaper
+    Style? toony, detailed, realistic
+    Text and languages? english_text, japanese_text, spanish_text, runes, dialogue, speech_bubble, symbol
+    Information? translated, partially_translated, unknown_artist_signature, not_furry, bigger version at the source
+    Requests? translation_request, source_request, tagme
+    Image size? low_res, hi_res, absurd_res, superabsurd_res
+    Year of creation? 2016, 2015, and so on
+
+Heavily vetted tags.
+
+Tags that can be found on our global blacklist, and heavily vetted tags MUST be added upon upload.
+
+    young, gore, scat, watersports, diaper, my little pony, vore, not furry, rape, hyper, feral, nazi, politics, zoophile iconography.
+    Everything pedophilia 
+
+Do NOT tag
+
+    Subjective tags that express opinions. Common examples include beautiful, sexy, hot, good, crappy and most other adjectives. Subjective themes can be collected into a set instead. (See https://e621.net/help/sets )
+    Generic tags such as legs, eyes, big, image and organism.