to commit or not commit that is the question

.gitignore

@@ -0,0 +1,12 @@
+data
+ext
+ARCHIVE
+misc/api_keys.txt
+misc/training_tags_categories
+misc/credentials/*
+misc/credentials
+scripts/ARCHIVE
+scripts/CEMETARY
+cemetary
+.venv
+logs

README.md

@@ -1,29 +1,78 @@
-# Repository for "Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm"
+# Code for the Paper titled ["Perpetual Misogyny: How Gendered Tropes Shape Text-To-Image AI Personalization"](http://arxiv.org/pdf/2505.04600)
 
-   This repository contains the code for [Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm](https://arxiv.org/abs/2505.04600).
 
-   ## Related Datasets
-   
-   This project uses three datasets, all hosted on Hugging Face:
-   - Dataset 1: [dataset-name-1](https://huggingface.co/datasets/username/dataset-1)
-   - Dataset 2: [dataset-name-2](https://huggingface.co/datasets/username/dataset-2)  
-   - Dataset 3: [dataset-name-3](https://huggingface.co/datasets/username/dataset-3)
 
-   ## Installation
-```python
-   pip install -r requirements.txt
+
+### Repository Structure
+
+```
+CIVITAI_VISUALIZATIONS/
+├── .virtual_documents/                   # Temporary files from Jupyter
+├── data/                                 # Final curated datasets
+│   ├── subset1/                          # Specific data splits or versions
+│   ├── subset2/
+│   └── ...
+├── misc/
+│   └── credentials/                      # API keys and sensitive config (excluded from versioning)
+├── plots/                                # Output plots and figures used in the paper
+├── ├── jupyter_notebooks/                          # Main analysis notebooks
+│   ├── 0_Scraping_image_metadata.ipynb         # Scrapes CivitAI metadata via API
+│   ├── Section_1_Figure_1_image_grid.ipynb     # Grid of sample images
+│   ├── Section_3-2-1_Figure_3_histogram.ipynb  # Histogram of upload trends
+│   ├── Section_3-2-1_Figure_4_Mivolo.ipynb     # Model activity plot (Mivolo-focused)
+│   ├── Section_3-3-1_Figure_5_tags.ipynb       # Tag frequency and usage visualizations
+│   ├── Section_3-3-3_download_popular_models.ipynb # Download models for analysis
+│   ├── Section_3-3-3_Figure_6.ipynb            # Promotional tag usage patterns
+│   ├── Section_3-3-4_Figure_8a.ipynb           # Ranking of popular models
+│   ├── Section_3-3-4_Figure_8b.ipynb           # Continuation of model rankings
+│   ├── Section_3-3-4_Figure_9_Sankey.ipynb     # Sankey diagram: user-model contributions
+│   ├── Section_3-3-4_LLM_annotation.ipynb      # Annotations using large language models
+│   ├── Section_3-4_extract_LoRA_metadata.ipynb # LoRA metadata extraction
+│   ├── SuppM_Figure_12_Danbooru_Taxonomy.ipynb # Danbooru tag taxonomy: visualization
+│   ├── SuppM_Figure_13_Danbooru_taxonomy.ipynb # Tag grouping and structure
+│   └── SuppM_Figure_13.ipynb                   # Supplementary figure generation
+├── misc/                                       # Utility scripts and API credentials (excluded)
+├── plots/                                      # Output plots and visualizations
+├── public/                                     # Optional public-facing files
+├── .gitignore
+├── .gitmodules
+├── README.md                                   # This file
+└── requirements.txt                            # Python dependencies
 ```
 
-   ## Quick Start
-```python
-   from datasets import load_dataset
-   
-   # Load all three datasets
-   dataset1 = load_dataset("username/dataset-1")
-   dataset2 = load_dataset("username/dataset-2")
-   dataset3 = load_dataset("username/dataset-3")
-   
-   # Run your pipeline
-   from src.pipeline import run_full_pipeline
-   results = run_full_pipeline(dataset1, dataset2, dataset3)
-```
+# Project Setup
+
+## Requirements
+This project requires Python 3.8 or higher. Ensure you have it installed before proceeding.
+
+## Installation
+
+1. **Clone the Repository**  
+   ```sh
+   git clone https://gitlab.uzh.ch/latent-canon/pm-paper.git
+   cd pm-paper
+   ```
+
+2. **Create a Virtual Environment**  
+   (Recommended to avoid dependency conflicts)
+   ```sh
+   python -m venv venv
+   source venv/bin/activate  # On macOS/Linux
+   venv\Scripts\activate     # On Windows
+   ```
+
+3. **Install Dependencies**  
+   ```sh
+   pip install -r requirements.txt
+   ```
+
+4. **Jupyter Notebook Setup (Optional)**  
+   If running Jupyter notebooks, ensure the environment is linked:
+   ```sh
+   python -m ipykernel install --user --name=venv --display-name "Python (venv)"
+   ```
+
+## Notes
+- Ensure you have necessary system dependencies installed (e.g., `opencv` may require additional system libraries).
+- If you encounter any issues, ensure you're using the correct Python environment (`venv` activated).
+

gitlab-ci.yml

@@ -0,0 +1,11 @@
+
+# .gitlab-ci.yml
+pages:
+  stage: deploy
+  script:
+    - echo "Nothing to build, serving public/"
+  artifacts:
+    paths:
+      - public
+  only:
+    - main

jupyter_notebooks/.ipynb_checkpoints/GEMMA_3-checkpoint.ipynb

@@ -0,0 +1,400 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "471e0cef-678e-4403-8eca-f8e1991d86de",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "output_file = current_dir.parent / f\"data/CSV/gemma_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/gemma_local_query_index.txt\"\n",
+    "\n",
+    "# Model settings\n",
+    "MODEL_NAME =  MODEL_NAME = \"google/gemma-3-27b-it\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~65GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "# Load model with optimizations\n",
+    "print(\"Loading model (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    torch_dtype=torch.bfloat16,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "print(\"Loading raw input CSV...\")\n",
+    "df = pd.read_csv(input_file)  # ALWAYS load the full input\n",
+    "print(f\"Loaded {len(df)} rows from input file\")\n",
+    "\n",
+    "# If we have previous annotations, merge them\n",
+    "if output_file.exists():\n",
+    "    print(\"Found existing annotations, merging...\")\n",
+    "    existing_df = pd.read_csv(output_file)\n",
+    "    print(f\"Existing annotations has {len(existing_df)} rows\")\n",
+    "    \n",
+    "    # Update df with existing annotations\n",
+    "    # Only update the columns that were annotated\n",
+    "    annotation_cols = ['full_name', 'aliases', 'gender', 'profession_llm', 'country']\n",
+    "    for col in annotation_cols:\n",
+    "        if col in existing_df.columns:\n",
+    "            df[col] = existing_df[col][:len(df)]  # Make sure we don't exceed df length\n",
+    "    \n",
+    "    print(f\"Merged annotations, continuing with {len(df)} total rows\")\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Gemma annotation with specific profession categories.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "- For country respond with one word only, for example China or Columbia\n",
+    "- actress = actor\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "# === QUERY Gemma LOCAL ===\n",
+    "def query_gemma_local(prompt: str) -> str:\n",
+    "    \"\"\"Query Gemma locally via transformers.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for GEMMA\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are an assistant that extracts key data on a person based on the name. Respond with exactly 5 numbered lines. For professions, choose ONLY from these categories: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for older tokenizers\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate\n",
+    "        with torch.no_grad():\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=512,\n",
+    "                temperature=0.1,\n",
+    "                do_sample=True,\n",
+    "                top_p=0.9,\n",
+    "                pad_token_id=tokenizer.eos_token_id\n",
+    "            )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "output_file = current_dir.parent / f\"data/CSV/gemma_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/gemma_local_query_index.txt\"\n",
+    "\n",
+    "\n",
+    "# === PARSE RESPONSE ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Gemma response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"Gemma Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_gemma_local(prompt)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7a1be7c0-ce54-4445-8534-bf3ab5e70197",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/MISTRAL-checkpoint.ipynb

@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/QWEN-checkpoint.ipynb

@@ -0,0 +1,645 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0543d9c4-055b-49a3-a8d0-9dfb622b2b8c",
+   "metadata": {},
+   "source": [
+    "# QWEN 2.5-32B Local Inference\n",
+    "\n",
+    "## Hardware Requirements\n",
+    "- **GPU**: NVIDIA A100 (40GB or 80GB recommended)\n",
+    "- **VRAM Usage**: \n",
+    "  - 8-bit quantization: ~32GB\n",
+    "  - 4-bit quantization: ~16-20GB\n",
+    "  - bfloat16 (no quantization): ~64GB\n",
+    "- **System RAM**: 32GB minimum, 64GB recommended\n",
+    "- **Storage**: ~65GB for model download\n",
+    "\n",
+    "## Configuration\n",
+    "This notebook uses **8-bit quantization** via `bitsandbytes` for optimal performance on A100 GPUs:\n",
+    "- Reduces VRAM usage from 64GB to ~32GB\n",
+    "- Minimal quality degradation\n",
+    "- Faster inference than bfloat16\n",
+    "\n",
+    "## Model Details\n",
+    "- **Model**: Qwen/Qwen2.5-32B-Instruct\n",
+    "- **Task**: Entity annotation and profession classification\n",
+    "- **Quantization**: LLM.int8() (8-bit)\n",
+    "- **Device**: CUDA (auto device mapping)\n",
+    "\n",
+    "## Dependencies\n",
+    "Make sure to install:\n",
+    "```bash\n",
+    "pip install transformers>=4.35.0 bitsandbytes>=0.41.0 accelerate torch pandas tqdm\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "fe6ba282-896b-4272-b82b-ef24810732fb",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-11-29T20:14:34.671159Z",
+     "iopub.status.busy": "2025-11-29T20:14:34.671015Z",
+     "iopub.status.idle": "2025-11-29T20:26:40.952267Z",
+     "shell.execute_reply": "2025-11-29T20:26:40.951486Z",
+     "shell.execute_reply.started": "2025-11-29T20:14:34.671146Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading model: Qwen/Qwen2.5-32B-Instruct\n",
+      "Cache directory: /shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/data/models\n",
+      "This may take a while on first run (~65GB download)...\n",
+      "\n",
+      "Device: cuda\n",
+      "Loading tokenizer...\n",
+      "✅ Tokenizer loaded\n",
+      "Configuring 8-bit quantization...\n",
+      "Loading model with 8-bit quantization (this may take several minutes)...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Loading checkpoint shards: 100%|██████████| 17/17 [03:49<00:00, 13.50s/it]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Model loaded with 8-bit quantization\n",
+      "VRAM used: 32.72 GB\n",
+      "\n",
+      "Loading raw input CSV...\n",
+      "✅ Loaded professions.csv\n",
+      "✅ Loaded profession mapping with 9 categories\n",
+      "Loaded 50861 rows\n",
+      "\n",
+      "Profession categories (9):\n",
+      "  - actor\n",
+      "  - adult performer\n",
+      "  - singer/musician\n",
+      "  - model\n",
+      "  - online personality\n",
+      "  - public figure\n",
+      "  - voice actor/ASMR\n",
+      "  - sports professional\n",
+      "  - tv personality\n",
+      "\n",
+      "Creating prompts...\n",
+      "✅ Prompts created\n",
+      "Resuming from index 0\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 0/50861 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.\n",
+      "Qwen Local:   0%|          | 10/50861 [01:35<115:51:08,  8.20s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "💾 Progress saved after row 10\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 20/50861 [03:01<143:54:04, 10.19s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "💾 Progress saved after row 20\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 24/50861 [03:52<151:28:05, 10.73s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "❌ Row 23: parsed as all Unknown (likely model crash); skipping.\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 30/50861 [04:36<116:38:03,  8.26s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "💾 Progress saved after row 30\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 40/50861 [05:55<142:16:40, 10.08s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "💾 Progress saved after row 40\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Qwen Local:   0%|          | 45/50861 [06:45<127:20:20,  9.02s/it]\n"
+     ]
+    },
+    {
+     "ename": "KeyboardInterrupt",
+     "evalue": "",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
+      "\u001b[31mKeyboardInterrupt\u001b[39m                         Traceback (most recent call last)",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[1]\u001b[39m\u001b[32m, line 330\u001b[39m\n\u001b[32m    328\u001b[39m response = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m    329\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m attempt \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[32m3\u001b[39m):\n\u001b[32m--> \u001b[39m\u001b[32m330\u001b[39m     response = \u001b[43mquery_qwen_local\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprompt\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    332\u001b[39m     \u001b[38;5;66;03m# Valid response?\u001b[39;00m\n\u001b[32m    333\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m response \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(response.strip()) > \u001b[32m10\u001b[39m:\n",
+      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[1]\u001b[39m\u001b[32m, line 245\u001b[39m, in \u001b[36mquery_qwen_local\u001b[39m\u001b[34m(prompt)\u001b[39m\n\u001b[32m    243\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m timeout(\u001b[32m60\u001b[39m):\n\u001b[32m    244\u001b[39m     \u001b[38;5;28;01mwith\u001b[39;00m torch.no_grad():\n\u001b[32m--> \u001b[39m\u001b[32m245\u001b[39m         outputs = \u001b[43mmodel\u001b[49m\u001b[43m.\u001b[49m\u001b[43mgenerate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    246\u001b[39m \u001b[43m            \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    247\u001b[39m \u001b[43m            \u001b[49m\u001b[43mmax_new_tokens\u001b[49m\u001b[43m=\u001b[49m\u001b[32;43m100\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[32m    248\u001b[39m \u001b[43m            \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[43m=\u001b[49m\u001b[32;43m0.1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[32m    249\u001b[39m \u001b[43m            \u001b[49m\u001b[43mdo_sample\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[32m    250\u001b[39m \u001b[43m            \u001b[49m\u001b[43mpad_token_id\u001b[49m\u001b[43m=\u001b[49m\u001b[43mtokenizer\u001b[49m\u001b[43m.\u001b[49m\u001b[43meos_token_id\u001b[49m\n\u001b[32m    251\u001b[39m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    253\u001b[39m \u001b[38;5;66;03m# Decode\u001b[39;00m\n\u001b[32m    254\u001b[39m generated_ids = outputs[\u001b[32m0\u001b[39m][inputs[\u001b[33m'\u001b[39m\u001b[33minput_ids\u001b[39m\u001b[33m'\u001b[39m].shape[\u001b[32m1\u001b[39m]:]\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:120\u001b[39m, in \u001b[36mcontext_decorator.<locals>.decorate_context\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m    117\u001b[39m \u001b[38;5;129m@functools\u001b[39m.wraps(func)\n\u001b[32m    118\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mdecorate_context\u001b[39m(*args, **kwargs):\n\u001b[32m    119\u001b[39m     \u001b[38;5;28;01mwith\u001b[39;00m ctx_factory():\n\u001b[32m--> \u001b[39m\u001b[32m120\u001b[39m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2564\u001b[39m, in \u001b[36mGenerationMixin.generate\u001b[39m\u001b[34m(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, use_model_defaults, custom_generate, **kwargs)\u001b[39m\n\u001b[32m   2561\u001b[39m model_kwargs[\u001b[33m\"\u001b[39m\u001b[33muse_cache\u001b[39m\u001b[33m\"\u001b[39m] = generation_config.use_cache\n\u001b[32m   2563\u001b[39m \u001b[38;5;66;03m# 9. Call generation mode\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m2564\u001b[39m result = \u001b[43mdecoding_method\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m   2565\u001b[39m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[32m   2566\u001b[39m \u001b[43m    \u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2567\u001b[39m \u001b[43m    \u001b[49m\u001b[43mlogits_processor\u001b[49m\u001b[43m=\u001b[49m\u001b[43mprepared_logits_processor\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2568\u001b[39m \u001b[43m    \u001b[49m\u001b[43mstopping_criteria\u001b[49m\u001b[43m=\u001b[49m\u001b[43mprepared_stopping_criteria\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2569\u001b[39m \u001b[43m    \u001b[49m\u001b[43mgeneration_config\u001b[49m\u001b[43m=\u001b[49m\u001b[43mgeneration_config\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2570\u001b[39m \u001b[43m    \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mgeneration_mode_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2571\u001b[39m \u001b[43m    \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mmodel_kwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m   2572\u001b[39m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   2574\u001b[39m \u001b[38;5;66;03m# Convert to legacy cache format if requested\u001b[39;00m\n\u001b[32m   2575\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m (\n\u001b[32m   2576\u001b[39m     generation_config.return_legacy_cache \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[32m   2577\u001b[39m     \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(result, \u001b[33m\"\u001b[39m\u001b[33mpast_key_values\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m   2578\u001b[39m     \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mgetattr\u001b[39m(result.past_key_values, \u001b[33m\"\u001b[39m\u001b[33mto_legacy_cache\u001b[39m\u001b[33m\"\u001b[39m) \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   2579\u001b[39m ):\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2787\u001b[39m, in \u001b[36mGenerationMixin._sample\u001b[39m\u001b[34m(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)\u001b[39m\n\u001b[32m   2785\u001b[39m     is_prefill = \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[32m   2786\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m2787\u001b[39m     outputs = \u001b[43mmodel_forward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mmodel_inputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mreturn_dict\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[32m   2789\u001b[39m \u001b[38;5;66;03m# synced_gpus: don't waste resources running the code we don't need; kwargs must be updated before skipping\u001b[39;00m\n\u001b[32m   2790\u001b[39m model_kwargs = \u001b[38;5;28mself\u001b[39m._update_model_kwargs_for_generation(\n\u001b[32m   2791\u001b[39m     outputs,\n\u001b[32m   2792\u001b[39m     model_kwargs,\n\u001b[32m   2793\u001b[39m     is_encoder_decoder=\u001b[38;5;28mself\u001b[39m.config.is_encoder_decoder,\n\u001b[32m   2794\u001b[39m )\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1775\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1773\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1774\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1775\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1786\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1781\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1782\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1783\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1784\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1785\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1786\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1788\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1789\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/utils/generic.py:918\u001b[39m, in \u001b[36mcan_return_tuple.<locals>.wrapper\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m    916\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m return_dict_passed \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m    917\u001b[39m     return_dict = return_dict_passed\n\u001b[32m--> \u001b[39m\u001b[32m918\u001b[39m output = \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    919\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m return_dict \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(output, \u001b[38;5;28mtuple\u001b[39m):\n\u001b[32m    920\u001b[39m     output = output.to_tuple()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py:449\u001b[39m, in \u001b[36mQwen2ForCausalLM.forward\u001b[39m\u001b[34m(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, cache_position, logits_to_keep, **kwargs)\u001b[39m\n\u001b[32m    417\u001b[39m \u001b[38;5;129m@can_return_tuple\u001b[39m\n\u001b[32m    418\u001b[39m \u001b[38;5;129m@auto_docstring\u001b[39m\n\u001b[32m    419\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mforward\u001b[39m(\n\u001b[32m   (...)\u001b[39m\u001b[32m    430\u001b[39m     **kwargs: Unpack[TransformersKwargs],\n\u001b[32m    431\u001b[39m ) -> CausalLMOutputWithPast:\n\u001b[32m    432\u001b[39m \u001b[38;5;250m    \u001b[39m\u001b[33mr\u001b[39m\u001b[33;03m\"\"\"\u001b[39;00m\n\u001b[32m    433\u001b[39m \u001b[33;03m    Example:\u001b[39;00m\n\u001b[32m    434\u001b[39m \n\u001b[32m   (...)\u001b[39m\u001b[32m    447\u001b[39m \u001b[33;03m    \"Hey, are you conscious? Can you talk to me?\\nI'm not conscious, but I can talk to you.\"\u001b[39;00m\n\u001b[32m    448\u001b[39m \u001b[33;03m    ```\"\"\"\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m449\u001b[39m     outputs: BaseModelOutputWithPast = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    450\u001b[39m \u001b[43m        \u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43minput_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    451\u001b[39m \u001b[43m        \u001b[49m\u001b[43mattention_mask\u001b[49m\u001b[43m=\u001b[49m\u001b[43mattention_mask\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    452\u001b[39m \u001b[43m        \u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    453\u001b[39m \u001b[43m        \u001b[49m\u001b[43mpast_key_values\u001b[49m\u001b[43m=\u001b[49m\u001b[43mpast_key_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    454\u001b[39m \u001b[43m        \u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m=\u001b[49m\u001b[43minputs_embeds\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    455\u001b[39m \u001b[43m        \u001b[49m\u001b[43muse_cache\u001b[49m\u001b[43m=\u001b[49m\u001b[43muse_cache\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    456\u001b[39m \u001b[43m        \u001b[49m\u001b[43mcache_position\u001b[49m\u001b[43m=\u001b[49m\u001b[43mcache_position\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    457\u001b[39m \u001b[43m        \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    458\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    460\u001b[39m     hidden_states = outputs.last_hidden_state\n\u001b[32m    461\u001b[39m     \u001b[38;5;66;03m# Only compute necessary logits, and do not upcast them to float if we are not computing the loss\u001b[39;00m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1775\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1773\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1774\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1775\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1786\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1781\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1782\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1783\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1784\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1785\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1786\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1788\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1789\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/utils/generic.py:1064\u001b[39m, in \u001b[36mcheck_model_inputs.<locals>.wrapper\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1061\u001b[39m                 monkey_patched_layers.append((module, original_forward))\n\u001b[32m   1063\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1064\u001b[39m     outputs = \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1065\u001b[39m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m original_exception:\n\u001b[32m   1066\u001b[39m     \u001b[38;5;66;03m# If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\u001b[39;00m\n\u001b[32m   1067\u001b[39m     \u001b[38;5;66;03m# Get a TypeError even after removing the recordable kwargs -> re-raise the original exception\u001b[39;00m\n\u001b[32m   1068\u001b[39m     \u001b[38;5;66;03m# Otherwise -> we're probably missing `**kwargs` in the decorated function\u001b[39;00m\n\u001b[32m   1069\u001b[39m     kwargs_without_recordable = {k: v \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m kwargs.items() \u001b[38;5;28;01mif\u001b[39;00m k \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m recordable_keys}\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py:384\u001b[39m, in \u001b[36mQwen2Model.forward\u001b[39m\u001b[34m(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, cache_position, **kwargs)\u001b[39m\n\u001b[32m    381\u001b[39m position_embeddings = \u001b[38;5;28mself\u001b[39m.rotary_emb(hidden_states, position_ids)\n\u001b[32m    383\u001b[39m \u001b[38;5;28;01mfor\u001b[39;00m decoder_layer \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m.layers[: \u001b[38;5;28mself\u001b[39m.config.num_hidden_layers]:\n\u001b[32m--> \u001b[39m\u001b[32m384\u001b[39m     hidden_states = \u001b[43mdecoder_layer\u001b[49m\u001b[43m(\u001b[49m\n\u001b[32m    385\u001b[39m \u001b[43m        \u001b[49m\u001b[43mhidden_states\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    386\u001b[39m \u001b[43m        \u001b[49m\u001b[43mattention_mask\u001b[49m\u001b[43m=\u001b[49m\u001b[43mcausal_mask_mapping\u001b[49m\u001b[43m[\u001b[49m\u001b[43mdecoder_layer\u001b[49m\u001b[43m.\u001b[49m\u001b[43mattention_type\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    387\u001b[39m \u001b[43m        \u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m=\u001b[49m\u001b[43mposition_ids\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    388\u001b[39m \u001b[43m        \u001b[49m\u001b[43mpast_key_values\u001b[49m\u001b[43m=\u001b[49m\u001b[43mpast_key_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    389\u001b[39m \u001b[43m        \u001b[49m\u001b[43muse_cache\u001b[49m\u001b[43m=\u001b[49m\u001b[43muse_cache\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    390\u001b[39m \u001b[43m        \u001b[49m\u001b[43mcache_position\u001b[49m\u001b[43m=\u001b[49m\u001b[43mcache_position\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    391\u001b[39m \u001b[43m        \u001b[49m\u001b[43mposition_embeddings\u001b[49m\u001b[43m=\u001b[49m\u001b[43mposition_embeddings\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    392\u001b[39m \u001b[43m        \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[32m    393\u001b[39m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    395\u001b[39m hidden_states = \u001b[38;5;28mself\u001b[39m.norm(hidden_states)\n\u001b[32m    396\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m BaseModelOutputWithPast(\n\u001b[32m    397\u001b[39m     last_hidden_state=hidden_states,\n\u001b[32m    398\u001b[39m     past_key_values=past_key_values \u001b[38;5;28;01mif\u001b[39;00m use_cache \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[32m    399\u001b[39m )\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/modeling_layers.py:94\u001b[39m, in \u001b[36mGradientCheckpointingLayer.__call__\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m     91\u001b[39m         logger.warning_once(message)\n\u001b[32m     93\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._gradient_checkpointing_func(partial(\u001b[38;5;28msuper\u001b[39m().\u001b[34m__call__\u001b[39m, **kwargs), *args)\n\u001b[32m---> \u001b[39m\u001b[32m94\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m.\u001b[49m\u001b[34;43m__call__\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1775\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1773\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1774\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1775\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1786\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1781\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1782\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1783\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1784\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1785\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1786\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1788\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1789\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/utils/deprecation.py:172\u001b[39m, in \u001b[36mdeprecate_kwarg.<locals>.wrapper.<locals>.wrapped_func\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m    168\u001b[39m \u001b[38;5;28;01melif\u001b[39;00m minimum_action \u001b[38;5;129;01min\u001b[39;00m (Action.NOTIFY, Action.NOTIFY_ALWAYS) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m is_torchdynamo_compiling():\n\u001b[32m    169\u001b[39m     \u001b[38;5;66;03m# DeprecationWarning is ignored by default, so we use FutureWarning instead\u001b[39;00m\n\u001b[32m    170\u001b[39m     warnings.warn(message, \u001b[38;5;167;01mFutureWarning\u001b[39;00m, stacklevel=\u001b[32m2\u001b[39m)\n\u001b[32m--> \u001b[39m\u001b[32m172\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py:249\u001b[39m, in \u001b[36mQwen2DecoderLayer.forward\u001b[39m\u001b[34m(self, hidden_states, attention_mask, position_ids, past_key_values, use_cache, cache_position, position_embeddings, **kwargs)\u001b[39m\n\u001b[32m    247\u001b[39m residual = hidden_states\n\u001b[32m    248\u001b[39m hidden_states = \u001b[38;5;28mself\u001b[39m.post_attention_layernorm(hidden_states)\n\u001b[32m--> \u001b[39m\u001b[32m249\u001b[39m hidden_states = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mmlp\u001b[49m\u001b[43m(\u001b[49m\u001b[43mhidden_states\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    250\u001b[39m hidden_states = residual + hidden_states\n\u001b[32m    251\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m hidden_states\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1775\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1773\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1774\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1775\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1786\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1781\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1782\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1783\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1784\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1785\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1786\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1788\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1789\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/transformers/models/qwen2/modeling_qwen2.py:46\u001b[39m, in \u001b[36mQwen2MLP.forward\u001b[39m\u001b[34m(self, x)\u001b[39m\n\u001b[32m     45\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mforward\u001b[39m(\u001b[38;5;28mself\u001b[39m, x):\n\u001b[32m---> \u001b[39m\u001b[32m46\u001b[39m     down_proj = \u001b[38;5;28mself\u001b[39m.down_proj(\u001b[38;5;28mself\u001b[39m.act_fn(\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mgate_proj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m) * \u001b[38;5;28mself\u001b[39m.up_proj(x))\n\u001b[32m     47\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m down_proj\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1775\u001b[39m, in \u001b[36mModule._wrapped_call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1773\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m._compiled_call_impl(*args, **kwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m   1774\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1775\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_call_impl\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py:1786\u001b[39m, in \u001b[36mModule._call_impl\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m   1781\u001b[39m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[32m   1782\u001b[39m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[32m   1783\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m._backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m._forward_pre_hooks\n\u001b[32m   1784\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[32m   1785\u001b[39m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[32m-> \u001b[39m\u001b[32m1786\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mforward_call\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1788\u001b[39m result = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m   1789\u001b[39m called_always_called_hooks = \u001b[38;5;28mset\u001b[39m()\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/nn/modules.py:1071\u001b[39m, in \u001b[36mLinear8bitLt.forward\u001b[39m\u001b[34m(self, x)\u001b[39m\n\u001b[32m   1068\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m.bias \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m.bias.dtype != x.dtype:\n\u001b[32m   1069\u001b[39m     \u001b[38;5;28mself\u001b[39m.bias.data = \u001b[38;5;28mself\u001b[39m.bias.data.to(x.dtype)\n\u001b[32m-> \u001b[39m\u001b[32m1071\u001b[39m out = \u001b[43mbnb\u001b[49m\u001b[43m.\u001b[49m\u001b[43mmatmul\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mweight\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbias\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mbias\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstate\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43mstate\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1073\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m.state.has_fp16_weights \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28mself\u001b[39m.state.CB \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m   1074\u001b[39m     \u001b[38;5;28mself\u001b[39m.weight.data = \u001b[38;5;28mself\u001b[39m.state.CB\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:424\u001b[39m, in \u001b[36mmatmul\u001b[39m\u001b[34m(A, B, out, state, threshold, bias)\u001b[39m\n\u001b[32m    422\u001b[39m     \u001b[38;5;28;01mif\u001b[39;00m A.device.type \u001b[38;5;129;01min\u001b[39;00m (\u001b[33m\"\u001b[39m\u001b[33mcpu\u001b[39m\u001b[33m\"\u001b[39m, \u001b[33m\"\u001b[39m\u001b[33mxpu\u001b[39m\u001b[33m\"\u001b[39m):\n\u001b[32m    423\u001b[39m         \u001b[38;5;28;01mreturn\u001b[39;00m MatMul8bitFp.apply(A, B, out, bias, state)\n\u001b[32m--> \u001b[39m\u001b[32m424\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mMatMul8bitLt\u001b[49m\u001b[43m.\u001b[49m\u001b[43mapply\u001b[49m\u001b[43m(\u001b[49m\u001b[43mA\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mB\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mout\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbias\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstate\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/autograd/function.py:581\u001b[39m, in \u001b[36mFunction.apply\u001b[39m\u001b[34m(cls, *args, **kwargs)\u001b[39m\n\u001b[32m    578\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m torch._C._are_functorch_transforms_active():\n\u001b[32m    579\u001b[39m     \u001b[38;5;66;03m# See NOTE: [functorch vjp and autograd interaction]\u001b[39;00m\n\u001b[32m    580\u001b[39m     args = _functorch.utils.unwrap_dead_wrappers(args)\n\u001b[32m--> \u001b[39m\u001b[32m581\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m.\u001b[49m\u001b[43mapply\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[32m    583\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m is_setup_ctx_defined:\n\u001b[32m    584\u001b[39m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[32m    585\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33mIn order to use an autograd.Function with functorch transforms \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    586\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33m(vmap, grad, jvp, jacrev, ...), it must override the setup_context \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    587\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33mstaticmethod. For more details, please see \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    588\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33mhttps://pytorch.org/docs/main/notes/extending.func.html\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m    589\u001b[39m     )\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:192\u001b[39m, in \u001b[36mMatMul8bitLt.forward\u001b[39m\u001b[34m(ctx, A, B, out, bias, state)\u001b[39m\n\u001b[32m    189\u001b[39m     CA, CAt, SCA, SCAt, outlier_cols = F.int8_double_quant(A.to(torch.float16), threshold=state.threshold)\n\u001b[32m    190\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m    191\u001b[39m     \u001b[38;5;66;03m# Fast path\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m192\u001b[39m     CA, SCA, outlier_cols = \u001b[43mF\u001b[49m\u001b[43m.\u001b[49m\u001b[43mint8_vectorwise_quant\u001b[49m\u001b[43m(\u001b[49m\u001b[43mA\u001b[49m\u001b[43m.\u001b[49m\u001b[43mto\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtorch\u001b[49m\u001b[43m.\u001b[49m\u001b[43mfloat16\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mthreshold\u001b[49m\u001b[43m=\u001b[49m\u001b[43mstate\u001b[49m\u001b[43m.\u001b[49m\u001b[43mthreshold\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    193\u001b[39m     CAt = SCAt = \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[32m    195\u001b[39m has_grad = \u001b[38;5;28;01mFalse\u001b[39;00m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/functional.py:2058\u001b[39m, in \u001b[36mint8_vectorwise_quant\u001b[39m\u001b[34m(A, threshold)\u001b[39m\n\u001b[32m   2040\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mint8_vectorwise_quant\u001b[39m(A: torch.Tensor, threshold=\u001b[32m0.0\u001b[39m):\n\u001b[32m   2041\u001b[39m \u001b[38;5;250m    \u001b[39m\u001b[33;03m\"\"\"Quantizes a tensor with dtype `torch.float16` to `torch.int8` in accordance to the `LLM.int8()` algorithm.\u001b[39;00m\n\u001b[32m   2042\u001b[39m \n\u001b[32m   2043\u001b[39m \u001b[33;03m    For more information, see the [LLM.int8() paper](https://arxiv.org/abs/2208.07339).\u001b[39;00m\n\u001b[32m   (...)\u001b[39m\u001b[32m   2056\u001b[39m \u001b[33;03m        - `torch.Tensor` with dtype `torch.int32`, *optional*: A list of column indices which contain outlier features.\u001b[39;00m\n\u001b[32m   2057\u001b[39m \u001b[33;03m    \"\"\"\u001b[39;00m\n\u001b[32m-> \u001b[39m\u001b[32m2058\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mtorch\u001b[49m\u001b[43m.\u001b[49m\u001b[43mops\u001b[49m\u001b[43m.\u001b[49m\u001b[43mbitsandbytes\u001b[49m\u001b[43m.\u001b[49m\u001b[43mint8_vectorwise_quant\u001b[49m\u001b[43m.\u001b[49m\u001b[43mdefault\u001b[49m\u001b[43m(\u001b[49m\u001b[43mA\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mthreshold\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/_ops.py:841\u001b[39m, in \u001b[36mOpOverload.__call__\u001b[39m\u001b[34m(self, *args, **kwargs)\u001b[39m\n\u001b[32m    840\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m__call__\u001b[39m(\u001b[38;5;28mself\u001b[39m, /, *args: _P.args, **kwargs: _P.kwargs) -> _T:\n\u001b[32m--> \u001b[39m\u001b[32m841\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m_op\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/_compile.py:53\u001b[39m, in \u001b[36m_disable_dynamo.<locals>.inner\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m     50\u001b[39m     disable_fn = torch._dynamo.disable(fn, recursive, wrapping=\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[32m     51\u001b[39m     fn.__dynamo_disable = disable_fn  \u001b[38;5;66;03m# type: ignore[attr-defined]\u001b[39;00m\n\u001b[32m---> \u001b[39m\u001b[32m53\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mdisable_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:1044\u001b[39m, in \u001b[36mDisableContext.__call__.<locals>._fn\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m   1042\u001b[39m _maybe_set_eval_frame(_callback_from_stance(\u001b[38;5;28mself\u001b[39m.callback))\n\u001b[32m   1043\u001b[39m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[32m-> \u001b[39m\u001b[32m1044\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m   1045\u001b[39m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[32m   1046\u001b[39m     set_eval_frame(\u001b[38;5;28;01mNone\u001b[39;00m)\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/torch/library.py:732\u001b[39m, in \u001b[36m_impl.<locals>.register_.<locals>.func_no_dynamo\u001b[39m\u001b[34m(*args, **kwargs)\u001b[39m\n\u001b[32m    730\u001b[39m \u001b[38;5;129m@torch\u001b[39m._disable_dynamo\n\u001b[32m    731\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34mfunc_no_dynamo\u001b[39m(*args, **kwargs):\n\u001b[32m--> \u001b[39m\u001b[32m732\u001b[39m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[36mFile \u001b[39m\u001b[32m/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/backends/cuda/ops.py:148\u001b[39m, in \u001b[36m_\u001b[39m\u001b[34m(A, threshold)\u001b[39m\n\u001b[32m    145\u001b[39m         outlier_cols = torch.argwhere(outliers.any(dim=\u001b[32m0\u001b[39m)).view(-\u001b[32m1\u001b[39m)\n\u001b[32m    146\u001b[39m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[32m    147\u001b[39m         \u001b[38;5;66;03m# Needed for torch.compile support.\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m148\u001b[39m         outlier_cols = \u001b[43mtorch\u001b[49m\u001b[43m.\u001b[49m\u001b[43mempty\u001b[49m\u001b[43m(\u001b[49m\u001b[32;43m0\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice\u001b[49m\u001b[43m=\u001b[49m\u001b[43mA\u001b[49m\u001b[43m.\u001b[49m\u001b[43mdevice\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdtype\u001b[49m\u001b[43m=\u001b[49m\u001b[43mtorch\u001b[49m\u001b[43m.\u001b[49m\u001b[43mint64\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m    150\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m _cuda_device_of(A):\n\u001b[32m    151\u001b[39m     lib.cint8_vector_quant(\n\u001b[32m    152\u001b[39m         get_ptr(A),\n\u001b[32m    153\u001b[39m         get_ptr(out_row),\n\u001b[32m   (...)\u001b[39m\u001b[32m    158\u001b[39m         _get_tensor_stream(A),\n\u001b[32m    159\u001b[39m     )\n",
+      "\u001b[31mKeyboardInterrupt\u001b[39m: "
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "import signal\n",
+    "from contextlib import contextmanager\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/qwen_local_query_index.txt\"\n",
+    "output_file = current_dir.parent / f\"data/CSV/qwen_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "\n",
+    "# Model settings\n",
+    "MODEL_NAME = \"Qwen/Qwen2.5-32B-Instruct\"\n",
+    "#MODEL_NAME = \"Qwen/Qwen2.5-14B-Instruct\"\n",
+    "#MODEL_NAME =  \"Qwen/Qwen3-235B-A22B-Instruct-2507-FP8\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~65GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "# Configure 8-bit quantization for A100\n",
+    "print(\"Configuring 8-bit quantization...\")\n",
+    "quantization_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True,\n",
+    "    llm_int8_threshold=6.0,\n",
+    "    llm_int8_has_fp16_weight=False\n",
+    ")\n",
+    "\n",
+    "# Load model with 8-bit quantization\n",
+    "print(\"Loading model with 8-bit quantization (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    quantization_config=quantization_config,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded with 8-bit quantization\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "if output_file.exists():\n",
+    "    print(\"Loading annotated CSV...\")\n",
+    "    df = pd.read_csv(output_file)\n",
+    "else:\n",
+    "    print(\"Loading raw input CSV...\")\n",
+    "    df = pd.read_csv(input_file)\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS (OPTIMIZED FOR CLEAN OUTPUTS) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Qwen annotation with strict formatting requirements.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Extract information about '{name}' ({hint_text}).\n",
+    "\n",
+    "Respond with EXACTLY 5 numbered lines. Each line must contain ONLY the value, no labels or extra text.\n",
+    "\n",
+    "FORMAT REQUIREMENTS:\n",
+    "1. Full legal name in Western order (first last). VALUE ONLY.\n",
+    "2. Stage names/aliases, comma-separated. If none, write \"None\". VALUE ONLY.\n",
+    "3. Gender: MUST be exactly one word: Male, Female, Other, or Unknown. VALUE ONLY.\n",
+    "4. Professions: Choose up to 3 from this list ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality. Comma-separated. VALUE ONLY.\n",
+    "5. Primary country: Country name only (e.g., \"China\", \"United States\", \"Colombia\"). VALUE ONLY.\n",
+    "\n",
+    "RULES:\n",
+    "- Professions MUST match the exact categories listed (actress = actor)\n",
+    "- \"online personality\" includes streamers, cosplayers, YouTubers, influencers\n",
+    "- \"public figure\" includes politicians, activists, journalists, authors\n",
+    "- Use \"Unknown\" when uncertain or for fictional characters\n",
+    "- NO explanatory text, NO labels like \"Gender:\", NO prefixes\n",
+    "- For multi-role people, list up to 3 categories by relevance\n",
+    "\n",
+    "EXAMPLE FORMAT:\n",
+    "1. Taylor Swift\n",
+    "2. None\n",
+    "3. Female\n",
+    "4. singer/musician, public figure\n",
+    "5. United States\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "@contextmanager\n",
+    "def timeout(duration):\n",
+    "    def handler(signum, frame):\n",
+    "        raise TimeoutError(\"Operation timed out\")\n",
+    "    \n",
+    "    # Set the signal handler and alarm\n",
+    "    signal.signal(signal.SIGALRM, handler)\n",
+    "    signal.alarm(duration)\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        signal.alarm(0)  # Disable the alarm\n",
+    "\n",
+    "\n",
+    "def query_qwen_local(prompt: str) -> str:\n",
+    "    \"\"\"Query Qwen locally via transformers.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for Qwen with strict system prompt\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are a data extraction assistant. Respond with exactly 5 numbered lines containing ONLY values. No labels, no explanations, no prefixes. Follow the format precisely.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for older tokenizers\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate with timeout\n",
+    "        with timeout(60):\n",
+    "            with torch.no_grad():\n",
+    "                outputs = model.generate(\n",
+    "                    **inputs,\n",
+    "                    max_new_tokens=100,\n",
+    "                    temperature=0.1,\n",
+    "                    do_sample=False,\n",
+    "                    pad_token_id=tokenizer.eos_token_id\n",
+    "                )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except TimeoutError:\n",
+    "        print(f\"[ERROR] Generation timed out after 60 seconds\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        import traceback\n",
+    "        traceback.print_exc()\n",
+    "        return None\n",
+    "\n",
+    "    \n",
+    "# === PARSE RESPONSE WITH CLEANING ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Qwen response into structured fields with cleaning.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            # Clean gender field - remove any labels\n",
+    "            gender_raw = line[2:].strip()\n",
+    "            # Remove common prefixes\n",
+    "            gender_raw = re.sub(r'^(Gender:|gender:)\\s*', '', gender_raw, flags=re.IGNORECASE)\n",
+    "            # Extract just the gender word\n",
+    "            gender_match = re.search(r'\\b(Male|Female|Other|Unknown)\\b', gender_raw, re.IGNORECASE)\n",
+    "            fields['gender'] = gender_match.group(1).capitalize() if gender_match else gender_raw\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            # Clean country field - remove any labels\n",
+    "            country_raw = line[2:].strip()\n",
+    "            # Remove common prefixes like \"Primary country:\", \"Country:\", etc.\n",
+    "            country_raw = re.sub(r'^(Primary\\s+)?(associated\\s+)?country:\\s*', '', country_raw, flags=re.IGNORECASE)\n",
+    "            fields['country'] = country_raw\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"Qwen Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_qwen_local(prompt)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d9c7deb9-847a-472d-8055-f93dbfa6aa2e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-2-2_Figure_4_Demographic_patterns_in_gen_images-checkpoint.ipynb

@@ -0,0 +1,1795 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d8161f06-4fd2-436d-9e59-3f68b5a67f2c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:30:16.974712Z",
+     "iopub.status.busy": "2025-02-06T18:30:16.974296Z",
+     "iopub.status.idle": "2025-02-06T18:30:16.976909Z",
+     "shell.execute_reply": "2025-02-06T18:30:16.976526Z",
+     "shell.execute_reply.started": "2025-02-06T18:30:16.974692Z"
+    }
+   },
+   "source": [
+    "# Section 6.2:  Age and Gender Estimation using MiVOLO"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64aaedec-ef56-4a62-b61e-12de2675a1ae",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T19:52:51.171282Z",
+     "iopub.status.busy": "2025-02-06T19:52:51.170711Z",
+     "iopub.status.idle": "2025-02-06T19:52:55.405039Z",
+     "shell.execute_reply": "2025-02-06T19:52:55.404308Z",
+     "shell.execute_reply.started": "2025-02-06T19:52:51.171245Z"
+    }
+   },
+   "source": [
+    "![Alt text](../plots/mivolo.svg)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4293f307-44fd-455e-90fe-6e6928be9af5",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:59:21.970807Z",
+     "iopub.status.busy": "2025-02-08T21:59:21.969931Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.724295Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.723583Z",
+     "shell.execute_reply.started": "2025-02-08T21:59:21.970784Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "import csv\n",
+    "from pathlib import Path\n",
+    "import logging\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "import requests\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "import cv2\n",
+    "from io import BytesIO\n",
+    "from PIL import Image, UnidentifiedImageError\n",
+    "from datetime import datetime, timedelta\n",
+    "from dateutil.relativedelta import relativedelta\n",
+    "from mivolo.predictor import Predictor\n",
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.patches as mpatches"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "63a54f5f-900c-48dd-8932-2632e56c5670",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.726069Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.725699Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.730626Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.730099Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.726050Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "mini = current_dir.parent / 'data/CSV/image_subsets/Civiverse-mini.csv'\n",
+    "mivolo_in = current_dir.parent / 'data/CSV/image_subsets/Civiverse-mini-by-month/'\n",
+    "(current_dir.parent /  'data/CSV/image_subsets/Civiverse-mini-by-month/').mkdir(parents=True, exist_ok=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1fdfb89a-6094-4382-8755-fae213221ea5",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.731540Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.731359Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.825738Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.825258Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.731524Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def split_by_month(input_path, output_dir):\n",
+    "    # Load the dataset\n",
+    "    df = pd.read_csv(input_path)\n",
+    "    \n",
+    "    # Convert the 'createdAt' column to datetime\n",
+    "    df['createdAt'] = pd.to_datetime(df['createdAt'], errors='coerce')\n",
+    "    \n",
+    "    # Extract year and month\n",
+    "    df['year_month'] = df['createdAt'].dt.to_period('M')\n",
+    "    \n",
+    "    # Group the data by year and month and save each group as a CSV file\n",
+    "    unique_months = df['year_month'].unique()\n",
+    "\n",
+    "    for month in unique_months:\n",
+    "        # Filter data for the specific month\n",
+    "        df_month = df[df['year_month'] == month]\n",
+    "        \n",
+    "        # Define the file name based on the year and month\n",
+    "        file_name = f'{output_dir}/Civiverse-{month}.csv'\n",
+    "        \n",
+    "        # Save the file\n",
+    "        df_month.to_csv(file_name, index=False)\n",
+    "\n",
+    "    print(f\"Data has been split and saved to {output_dir}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "2c909d7c-7d16-4dc7-8364-7f1c0784414c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.827095Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.826919Z",
+     "iopub.status.idle": "2025-02-08T22:00:10.479484Z",
+     "shell.execute_reply": "2025-02-08T22:00:10.478777Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.827079Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/sctmp/lauwag/ipykernel_1497673/1825509207.py:9: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.\n",
+      "  df['year_month'] = df['createdAt'].dt.to_period('M')\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Data has been split and saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month\n"
+     ]
+    }
+   ],
+   "source": [
+    "split_by_month(mini, mivolo_in)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "4c543306-ffc8-4b9c-a3df-b49b2271caa9",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:10.480505Z",
+     "iopub.status.busy": "2025-02-08T22:00:10.480310Z",
+     "iopub.status.idle": "2025-02-08T22:00:10.483961Z",
+     "shell.execute_reply": "2025-02-08T22:00:10.483400Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:10.480486Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "mivolo_out = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'\n",
+    "mivolo_out.mkdir(parents=True, exist_ok=True)  # Create the output directory if it doesn't exist"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ffb7dd23",
+   "metadata": {},
+   "source": [
+    "## MiVOLO gender and age inference"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "304ed12f-c7b6-4129-b24d-7ccc793a62c7",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:10.484802Z",
+     "iopub.status.busy": "2025-02-08T22:00:10.484639Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/ultralytics/nn/tasks.py:634: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  return torch.load(file, map_location=\"cpu\"), file  # load\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Model summary (fused): 268 layers, 68125494 parameters, 0 gradients, 257.4 GFLOPs\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[W208 23:00:15.738708520 NNPACK.cpp:61] Could not initialize NNPACK! Reason: Unsupported hardware.\n",
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/mivolo/model/mi_volo.py:33: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  state = torch.load(ckpt_path, map_location=\"cpu\")\n",
+      "INFO:MiVOLO:Model meta:\n",
+      "min_age: 1, max_age: 95, avg_age: 48.0, num_classes: 3, in_chans: 6, with_persons_model: True, disable_faces: False, use_persons: True, only_age: False, num_classes_gender: 2, input_size: 224, use_person_crops: True, use_face_crops: True\n",
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/timm/models/_helpers.py:39: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  checkpoint = torch.load(checkpoint_path, map_location='cpu')\n",
+      "INFO:timm.models._helpers:Loaded state_dict from checkpoint '/shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/ext/MiVOLO/models/mivolo_imbd.pth.tar'\n",
+      "INFO:MiVOLO:Model mivolo_d1_224 created, param count: 27432414\n",
+      "INFO:timm.data.config:Data processing configuration for current model + dataset:\n",
+      "INFO:timm.data.config:\tinput_size: (3, 224, 224)\n",
+      "INFO:timm.data.config:\tinterpolation: bicubic\n",
+      "INFO:timm.data.config:\tmean: (0.485, 0.456, 0.406)\n",
+      "INFO:timm.data.config:\tstd: (0.229, 0.224, 0.225)\n",
+      "INFO:timm.data.config:\tcrop_pct: 0.96\n",
+      "INFO:timm.data.config:\tcrop_mode: center\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2022-11.csv\n",
+      "\n",
+      "0: 640x640 (no detections), 723.9ms\n",
+      "Speed: 12.1ms preprocess, 723.9ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2022-11.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2022-12.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-01.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-02.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-03.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-04.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-05.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-06.csv\n",
+      "\n",
+      "0: 416x640 1 person, 455.1ms\n",
+      "Speed: 3.5ms preprocess, 455.1ms inference, 33.5ms postprocess per image at shape (1, 3, 416, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 32.89\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-06.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-07.csv\n",
+      "\n",
+      "0: 640x320 1 person, 395.7ms\n",
+      "Speed: 2.9ms preprocess, 395.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 320)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 33.49\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 478.5ms\n",
+      "Speed: 1.9ms preprocess, 478.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.81\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 2 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-07.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-08.csv\n",
+      "\n",
+      "0: 640x448 1 person, 478.0ms\n",
+      "Speed: 2.9ms preprocess, 478.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 40.62\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-08.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-09.csv\n",
+      "\n",
+      "0: 416x640 (no detections), 567.1ms\n",
+      "Speed: 2.4ms preprocess, 567.1ms inference, 0.4ms postprocess per image at shape (1, 3, 416, 640)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 320x640 (no detections), 393.6ms\n",
+      "Speed: 1.7ms preprocess, 393.6ms inference, 0.4ms postprocess per image at shape (1, 3, 320, 640)\n",
+      "\n",
+      "0: 640x640 (no detections), 711.9ms\n",
+      "Speed: 3.4ms preprocess, 711.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x640 (no detections), 699.8ms\n",
+      "Speed: 2.3ms preprocess, 699.8ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x576 1 person, 629.6ms\n",
+      "Speed: 2.4ms preprocess, 629.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 576)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 28.65\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 598.3ms\n",
+      "Speed: 2.1ms preprocess, 598.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.85\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/98848c97-3d1e-4b52-9967-aeeca354a30e/width=656/98848c97-3d1e-4b52-9967-aeeca354a30e.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00133740>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-09.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-10.csv\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/e6469288-b487-4a06-99c1-59e7ac22fa77/width=1024/e6469288-b487-4a06-99c1-59e7ac22fa77.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00ecbdd0>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 448x640 (no detections), 536.6ms\n",
+      "Speed: 10.1ms preprocess, 536.6ms inference, 0.4ms postprocess per image at shape (1, 3, 448, 640)\n",
+      "Processed and saved 2 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-10.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-11.csv\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 662.9ms\n",
+      "Speed: 2.6ms preprocess, 662.9ms inference, 0.9ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.0\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x384 1 person, 895.9ms\n",
+      "Speed: 2.0ms preprocess, 895.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 43.33\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 529.4ms\n",
+      "Speed: 2.6ms preprocess, 529.4ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 539.3ms\n",
+      "Speed: 2.8ms preprocess, 539.3ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 39.15\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 708.6ms\n",
+      "Speed: 2.5ms preprocess, 708.6ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 29.64\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/ad4b5481-0a58-4196-a3e8-fca2fe22a3cc/width=1080/ad4b5481-0a58-4196-a3e8-fca2fe22a3cc.mp4: cannot identify image file <_io.BytesIO object at 0x14cb010c24d0>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-11.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-12.csv\n",
+      "\n",
+      "0: 640x384 1 person, 1 face, 461.0ms\n",
+      "Speed: 2.4ms preprocess, 461.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 19.61\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 501.3ms\n",
+      "Speed: 3.1ms preprocess, 501.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.58\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/3004b5fa-af81-4de7-829d-1d809d70b878/width=512/3004b5fa-af81-4de7-829d-1d809d70b878.mp4: cannot identify image file <_io.BytesIO object at 0x14cb0020ad40>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 842.5ms\n",
+      "Speed: 4.5ms preprocess, 842.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x416 (no detections), 446.8ms\n",
+      "Speed: 2.5ms preprocess, 446.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "Processed and saved 5 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-12.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-01.csv\n",
+      "\n",
+      "0: 640x448 (no detections), 638.5ms\n",
+      "Speed: 2.3ms preprocess, 638.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x416 (no detections), 441.7ms\n",
+      "Speed: 2.5ms preprocess, 441.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "\n",
+      "0: 640x448 (no detections), 470.3ms\n",
+      "Speed: 2.3ms preprocess, 470.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 693.9ms\n",
+      "Speed: 2.5ms preprocess, 693.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 808.6ms\n",
+      "Speed: 3.2ms preprocess, 808.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 15.01\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x320 1 person, 1 face, 345.6ms\n",
+      "Speed: 2.0ms preprocess, 345.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 320)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.86\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 6 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-01.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-02.csv\n",
+      "\n",
+      "0: 640x384 1 person, 1 face, 387.8ms\n",
+      "Speed: 1.9ms preprocess, 387.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.31\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x480 1 person, 1 face, 540.4ms\n",
+      "Speed: 2.5ms preprocess, 540.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.47\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 1 person, 1 face, 713.1ms\n",
+      "Speed: 3.8ms preprocess, 713.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.85\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 778.8ms\n",
+      "Speed: 28.7ms preprocess, 778.8ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 528.2ms\n",
+      "Speed: 2.3ms preprocess, 528.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.63\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 518.4ms\n",
+      "Speed: 3.9ms preprocess, 518.4ms inference, 0.9ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 18.25\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 470.7ms\n",
+      "Speed: 2.5ms preprocess, 470.7ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.51\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x480 1 person, 1 face, 647.1ms\n",
+      "Speed: 2.4ms preprocess, 647.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 58.87\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 469.8ms\n",
+      "Speed: 2.6ms preprocess, 469.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 477.5ms\n",
+      "Speed: 2.3ms preprocess, 477.5ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 23.79\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 10 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-02.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-03.csv\n",
+      "\n",
+      "0: 640x448 1 face, 511.4ms\n",
+      "Speed: 2.5ms preprocess, 511.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 24.87\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x544 (no detections), 576.5ms\n",
+      "Speed: 2.9ms preprocess, 576.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 544)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 687.1ms\n",
+      "Speed: 9.9ms preprocess, 687.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.76\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 498.3ms\n",
+      "Speed: 2.3ms preprocess, 498.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 (no detections), 573.2ms\n",
+      "Speed: 3.0ms preprocess, 573.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "Processed and saved 5 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-03.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-04.csv\n",
+      "\n",
+      "0: 640x384 (no detections), 518.2ms\n",
+      "Speed: 2.7ms preprocess, 518.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x512 (no detections), 707.7ms\n",
+      "Speed: 3.6ms preprocess, 707.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x416 (no detections), 453.7ms\n",
+      "Speed: 2.4ms preprocess, 453.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "\n",
+      "0: 640x384 (no detections), 391.0ms\n",
+      "Speed: 2.0ms preprocess, 391.0ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 449.8ms\n",
+      "Speed: 2.3ms preprocess, 449.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.39\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 618.4ms\n",
+      "Speed: 2.3ms preprocess, 618.4ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 631.0ms\n",
+      "Speed: 2.2ms preprocess, 631.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 24.05\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 1 person, 496.4ms\n",
+      "Speed: 2.6ms preprocess, 496.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.81\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 442.8ms\n",
+      "Speed: 2.3ms preprocess, 442.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 477.7ms\n",
+      "Speed: 2.4ms preprocess, 477.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.62\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 447.0ms\n",
+      "Speed: 2.2ms preprocess, 447.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 54.31\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 819.0ms\n",
+      "Speed: 3.6ms preprocess, 819.0ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 478.2ms\n",
+      "Speed: 1.8ms preprocess, 478.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.56\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 471.2ms\n",
+      "Speed: 2.7ms preprocess, 471.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.31\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 484.0ms\n",
+      "Speed: 2.2ms preprocess, 484.0ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x640 (no detections), 832.6ms\n",
+      "Speed: 3.0ms preprocess, 832.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 508.9ms\n",
+      "Speed: 2.5ms preprocess, 508.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 27.19\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6879c7b9-5cb3-42db-b409-30b4e2f71945/width=1080/6879c7b9-5cb3-42db-b409-30b4e2f71945.mp4: cannot identify image file <_io.BytesIO object at 0x14cb0020ad40>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 9 persons, 461.8ms\n",
+      "Speed: 2.4ms preprocess, 461.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([9, 3, 224, 224]), person_input: torch.Size([9, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 30.4\n",
+      "INFO:MiVOLO:\tgender: male [55%]\n",
+      "INFO:MiVOLO:\tage: 28.89\n",
+      "INFO:MiVOLO:\tgender: female [63%]\n",
+      "INFO:MiVOLO:\tage: 30.31\n",
+      "INFO:MiVOLO:\tgender: female [68%]\n",
+      "INFO:MiVOLO:\tage: 31.62\n",
+      "INFO:MiVOLO:\tgender: female [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 33.02\n",
+      "INFO:MiVOLO:\tgender: male [95%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 19 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-04.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-05.csv\n",
+      "\n",
+      "0: 640x448 1 person, 455.5ms\n",
+      "Speed: 2.2ms preprocess, 455.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 37.57\n",
+      "INFO:MiVOLO:\tgender: male [95%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 438.7ms\n",
+      "Speed: 2.2ms preprocess, 438.7ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 15.62\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 444.8ms\n",
+      "Speed: 2.3ms preprocess, 444.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6032bd70-6d53-4007-9e89-e69d4748efb5/width=528/6032bd70-6d53-4007-9e89-e69d4748efb5.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00ed0950>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 453.9ms\n",
+      "Speed: 2.3ms preprocess, 453.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 475.0ms\n",
+      "Speed: 1.6ms preprocess, 475.0ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.5\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 447.6ms\n",
+      "Speed: 2.5ms preprocess, 447.6ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 23.46\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 (no detections), 528.5ms\n",
+      "Speed: 3.2ms preprocess, 528.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 449.8ms\n",
+      "Speed: 2.3ms preprocess, 449.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 29.32\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 617.7ms\n",
+      "Speed: 2.4ms preprocess, 617.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.32\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 609.1ms\n",
+      "Speed: 2.3ms preprocess, 609.1ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 436.2ms\n",
+      "Speed: 2.5ms preprocess, 436.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 585.6ms\n",
+      "Speed: 3.1ms preprocess, 585.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.5\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 457.3ms\n",
+      "Speed: 2.1ms preprocess, 457.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.19\n",
+      "INFO:MiVOLO:\tgender: male [81%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 14 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-05.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-06.csv\n",
+      "\n",
+      "0: 640x448 (no detections), 484.5ms\n",
+      "Speed: 2.8ms preprocess, 484.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x512 (no detections), 524.8ms\n",
+      "Speed: 2.9ms preprocess, 524.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x480 1 person, 478.0ms\n",
+      "Speed: 2.6ms preprocess, 478.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 39.4\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 1 person, 1 face, 539.8ms\n",
+      "Speed: 2.6ms preprocess, 539.8ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.33\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 2 faces, 446.7ms\n",
+      "Speed: 2.4ms preprocess, 446.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([2, 3, 224, 224]), person_input: torch.Size([2, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.65\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "INFO:MiVOLO:\tage: 20.53\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 1 person, 1 face, 655.0ms\n",
+      "Speed: 3.3ms preprocess, 655.0ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 26.34\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x384 (no detections), 400.6ms\n",
+      "Speed: 2.1ms preprocess, 400.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "\n",
+      "0: 640x448 1 person, 587.9ms\n",
+      "Speed: 2.2ms preprocess, 587.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 30.4\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 610.3ms\n",
+      "Speed: 2.3ms preprocess, 610.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 453.6ms\n",
+      "Speed: 2.3ms preprocess, 453.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 511.3ms\n",
+      "Speed: 2.8ms preprocess, 511.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 34.28\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 441.2ms\n",
+      "Speed: 2.3ms preprocess, 441.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 586.3ms\n",
+      "Speed: 2.3ms preprocess, 586.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 437.5ms\n",
+      "Speed: 2.4ms preprocess, 437.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 448x640 1 person, 1 face, 437.4ms\n",
+      "Speed: 2.4ms preprocess, 437.4ms inference, 0.7ms postprocess per image at shape (1, 3, 448, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.81\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 436.8ms\n",
+      "Speed: 2.6ms preprocess, 436.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 448x640 (no detections), 433.0ms\n",
+      "Speed: 1.9ms preprocess, 433.0ms inference, 0.4ms postprocess per image at shape (1, 3, 448, 640)\n",
+      "\n",
+      "0: 640x448 (no detections), 599.7ms\n",
+      "Speed: 2.5ms preprocess, 599.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Set up logging\n",
+    "detector_weights = current_dir.parent / 'ext/MiVOLO/models/yolov8x_person_face.pt'\n",
+    "checkpoint = current_dir.parent / 'ext/MiVOLO/models/mivolo_imbd.pth.tar'\n",
+    "\n",
+    "_logger = logging.getLogger(\"inference\")\n",
+    "logging.basicConfig(level=logging.INFO)\n",
+    "\n",
+    "# Placeholder configuration and predictor initialization for MiVOLO\n",
+    "class Config:\n",
+    "    def __init__(self, detector_weights, checkpoint, device, with_persons=True, disable_faces=False, draw=False):\n",
+    "        self.detector_weights = detector_weights\n",
+    "        self.checkpoint = checkpoint\n",
+    "        self.device = device\n",
+    "        self.with_persons = with_persons\n",
+    "        self.disable_faces = disable_faces\n",
+    "        self.draw = draw\n",
+    "\n",
+    "\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "config = Config(detector_weights=detector_weights, checkpoint=checkpoint, device=device)\n",
+    "predictor = Predictor(config, verbose=True)\n",
+    "\n",
+    "def download_image(url):\n",
+    "    try:\n",
+    "        response = requests.get(url)\n",
+    "        response.raise_for_status()\n",
+    "        return Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
+    "    except requests.RequestException as e:\n",
+    "        _logger.error(f\"Failed to download image from {url}: {e}\")\n",
+    "        return None\n",
+    "    except UnidentifiedImageError as e:\n",
+    "        _logger.error(f\"Unidentified image error for URL {url}: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "def process_images_with_progress(data, predictor, output_file, start_idx=0):\n",
+    "    results = []\n",
+    "    total_images = len(data)\n",
+    "\n",
+    "    for idx, row in data.iterrows():\n",
+    "        if idx < start_idx:\n",
+    "            continue\n",
+    "\n",
+    "        img_url = row[\"url\"]\n",
+    "        pil_image = download_image(img_url)\n",
+    "        if pil_image is None:\n",
+    "            continue\n",
+    "\n",
+    "        np_image = np.array(pil_image)\n",
+    "        np_image = cv2.cvtColor(np_image, cv2.COLOR_RGB2BGR)\n",
+    "        detected_objects, _ = predictor.recognize(np_image)\n",
+    "\n",
+    "        row_result = row.to_dict()  # Start with the original row's data\n",
+    "\n",
+    "        if detected_objects and detected_objects.ages:\n",
+    "            for i in range(len(detected_objects.ages)):\n",
+    "                age = detected_objects.ages[i]\n",
+    "                gender = detected_objects.genders[i]\n",
+    "                gender_confidence = detected_objects.gender_scores[i]\n",
+    "\n",
+    "                if gender_confidence >= 0.83:\n",
+    "                    detection = {\n",
+    "                        \"detection_type\": 'face' if i in detected_objects.face_to_person_map else 'person',\n",
+    "                        \"gender\": gender,\n",
+    "                        \"gender_confidence\": gender_confidence,\n",
+    "                        \"age\": age,\n",
+    "                        \"n_persons\": detected_objects.n_persons,\n",
+    "                        \"n_faces\": detected_objects.n_faces,\n",
+    "                        \"detected\": True\n",
+    "                    }\n",
+    "                else:\n",
+    "                    detection = {\n",
+    "                        \"detection_type\": \"N/A\",\n",
+    "                        \"gender\": \"N/A\",\n",
+    "                        \"gender_confidence\": 0,\n",
+    "                        \"age\": 0,\n",
+    "                        \"n_persons\": detected_objects.n_persons if detected_objects else 0,\n",
+    "                        \"n_faces\": detected_objects.n_faces if detected_objects else 0,\n",
+    "                        \"detected\": False\n",
+    "                    }\n",
+    "\n",
+    "                results.append({**row_result, **detection})\n",
+    "        else:\n",
+    "            detection = {\n",
+    "                \"detection_type\": \"N/A\",\n",
+    "                \"gender\": \"N/A\",\n",
+    "                \"gender_confidence\": 0,\n",
+    "                \"age\": 0,\n",
+    "                \"n_persons\": detected_objects.n_persons if detected_objects else 0,\n",
+    "                \"n_faces\": detected_objects.n_faces if detected_objects else 0,\n",
+    "                \"detected\": False\n",
+    "            }\n",
+    "            results.append({**row_result, **detection})\n",
+    "\n",
+    "        if idx % 100 == 0 or idx == total_images - 1:\n",
+    "            df = pd.DataFrame(results)\n",
+    "            if os.path.exists(output_file):\n",
+    "                df.to_csv(output_file, mode='a', header=False, index=False)\n",
+    "            else:\n",
+    "                df.to_csv(output_file, mode='w', header=True, index=False)\n",
+    "            results = []\n",
+    "            print(f\"Processed and saved {idx + 1} images so far.\")\n",
+    "\n",
+    "def generate_months(start, end):\n",
+    "    start_date = datetime.strptime(start, '%Y-%m')\n",
+    "    end_date = datetime.strptime(end, '%Y-%m')\n",
+    "    while start_date <= end_date:\n",
+    "        yield start_date.strftime('%Y-%m')\n",
+    "        start_date += relativedelta(months=1)  # Increment by calendar months\n",
+    "\n",
+    "\n",
+    "start_month = '2022-11'\n",
+    "end_month = '2024-12'\n",
+    "\n",
+    "for month in generate_months(start_month, end_month):\n",
+    "    input_file_path = mivolo_in / f'Civiverse-{month}.csv'\n",
+    "    output_file_path = mivolo_out / f'{month}.csv'\n",
+    "\n",
+    "    if input_file_path.exists():\n",
+    "        print(f\"Processing: {input_file_path}\")\n",
+    "\n",
+    "        data = pd.read_csv(input_file_path)\n",
+    "        start_index = 0\n",
+    "        process_images_with_progress(data, predictor, output_file_path, start_idx=start_index)\n",
+    "\n",
+    "        print(f\"Processed and saved to: {output_file_path}\")\n",
+    "    else:\n",
+    "        print(f\"File not found: {input_file_path}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "26aeeef7",
+   "metadata": {},
+   "source": [
+    "## Visualization code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "88ec896a-bf9b-4cc6-a787-c1343f8acb41",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.patches as mpatches\n",
+    "\n",
+    "input_dir = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'\n",
+    "plot_dir = current_dir.parent / 'plots/'\n",
+    "\n",
+    "\n",
+    "all_data = pd.DataFrame()\n",
+    "for file_path in input_dir.glob('*.csv'):  # Reads all CSV files in the folder\n",
+    "    #print(f\"Loading: {file_path}\")\n",
+    "    data = pd.read_csv(file_path)\n",
+    "    all_data = pd.concat([all_data, data], ignore_index=True)\n",
+    "\n",
+    "# Filter rows where detection_type equals \"person\"\n",
+    "person_data = all_data[all_data['detection_type'] == 'person']\n",
+    "\n",
+    "# Count unique images and categorize by persons detected\n",
+    "n_images = all_data['id'].nunique()\n",
+    "images_with_zero_persons = all_data[all_data['n_persons'] == 0]['id'].nunique()\n",
+    "images_with_one_or_more_persons = all_data[all_data['n_persons'] > 0]['id'].nunique()\n",
+    "\n",
+    "n_persons_detected = person_data['id'].nunique()  # Unique images with at least one detected person\n",
+    "total_persons_detected = person_data.shape[0]  # Total number of persons detected\n",
+    "\n",
+    "\n",
+    "\n",
+    "n_total_female = person_data[person_data['gender'] == 'female']['id'].nunique()\n",
+    "n_total_male = person_data[person_data['gender'] == 'male']['id'].nunique()\n",
+    "\n",
+    "# Filter the data further for non-missing age and gender\n",
+    "filtered_data = person_data.dropna(subset=['age', 'gender'])\n",
+    "\n",
+    "# Round ages for consistent plotting\n",
+    "filtered_data['rounded_age'] = np.round(filtered_data['age'] * 4) / 4\n",
+    "\n",
+    "# Map browsingLevel to colors\n",
+    "def get_browsing_color(browsing_level):\n",
+    "    color_mapping = {\n",
+    "        1: 'silver',\n",
+    "        2: 'rosybrown',\n",
+    "        4: 'coral',\n",
+    "        8: 'crimson',\n",
+    "        16: 'blueviolet'\n",
+    "    }\n",
+    "    return color_mapping.get(browsing_level, 'black')  # Default to black for unknown values\n",
+    "\n",
+    "filtered_data['color'] = filtered_data['browsingLevel'].apply(get_browsing_color)\n",
+    "\n",
+    "# Aggregate data for plotting\n",
+    "aggregated_data = (\n",
+    "    filtered_data.groupby(['rounded_age', 'gender', 'color'])\n",
+    "    .size()\n",
+    "    .unstack(fill_value=0)\n",
+    ")\n",
+    "\n",
+    "# Define NSFW colors\n",
+    "nsfw_colors = ['blueviolet', 'crimson', 'coral', 'rosybrown', 'silver']\n",
+    "\n",
+    "# Plotting function\n",
+    "def plot_gender_data(ax, data, gender_label):\n",
+    "    ages = data.index\n",
+    "    bottom = np.zeros(len(ages))\n",
+    "    \n",
+    "    for color in nsfw_colors:\n",
+    "        counts = data[color] if color in data.columns else np.zeros(len(ages))\n",
+    "        ax.bar(\n",
+    "            ages,\n",
+    "            counts,\n",
+    "            color=color,\n",
+    "            edgecolor=color,\n",
+    "            linewidth=1,\n",
+    "            width=0.2,\n",
+    "            bottom=bottom,\n",
+    "            alpha=0.5\n",
+    "        )\n",
+    "        bottom += counts\n",
+    "\n",
+    "    x_min = 5\n",
+    "    x_max = filtered_data['rounded_age'].max()\n",
+    "    ax.set_xticks(np.arange(x_min, x_max + 0.5, 5))\n",
+    "    ax.set_xticklabels([f'{int(age)}' for age in np.arange(x_min, x_max + 0.5, 5)], fontsize=12, fontweight='bold')\n",
+    "    ax.set_xticks(np.arange(x_min, x_max + 0.5, 0.5), minor=True)\n",
+    "\n",
+    "    y_min = 0\n",
+    "    y_max = bottom.max() + 100\n",
+    "    y_ticks = np.arange(y_min, y_max + 1, 100)  # Fine-grained steps of 100\n",
+    "    ax.set_yticks(y_ticks)\n",
+    "    ax.set_yticklabels([str(int(y)) for y in y_ticks], fontsize=12, fontweight='bold')\n",
+    "\n",
+    "    ax.grid(True, which='major', color='lightgrey', linestyle='-', linewidth=0.5)\n",
+    "    ax.grid(True, which='minor', color='lightgrey', linestyle=':', linewidth=0.5)\n",
+    "\n",
+    "    ax.spines['top'].set_visible(False)\n",
+    "    ax.spines['right'].set_visible(False)\n",
+    "    ax.spines['left'].set_visible(False)\n",
+    "    ax.spines['bottom'].set_visible(False)\n",
+    "    \n",
+    "    ax.set_xlabel('Age', fontsize=12, fontweight='bold')\n",
+    "    if gender_label == 'Female':\n",
+    "        ax.set_ylabel('Number of Subjects', fontsize=14, fontweight='bold')\n",
+    "    ax.set_title(f'{gender_label} Read', fontsize=14, fontweight='bold')\n",
+    "\n",
+    "# Set up the subplots\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(14, 6.5), sharey=True)\n",
+    "\n",
+    "plot_gender_data(axes[0], aggregated_data.xs('male', level='gender'), 'Male')\n",
+    "plot_gender_data(axes[1], aggregated_data.xs('female', level='gender'), 'Female')\n",
+    "\n",
+    "legend_patches = [\n",
+    "    mpatches.Patch(facecolor='blueviolet', edgecolor='blueviolet', linewidth=2, label='Level 16: XXX'),\n",
+    "    mpatches.Patch(facecolor='crimson', edgecolor='crimson', linewidth=2, label='Level 8: X'),\n",
+    "    mpatches.Patch(facecolor='coral', edgecolor='coral', linewidth=2, label='Level 4: Mature'),\n",
+    "    mpatches.Patch(facecolor='rosybrown', edgecolor='rosybrown', linewidth=2, label='Level 2: Soft'),\n",
+    "    mpatches.Patch(facecolor='silver', edgecolor='silver', linewidth=2, label='Level 1: SFW'),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'n images: {n_images}', alpha=0),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'Total persons detected: {total_persons_detected}', alpha=0),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'Unique images containing persons: {n_persons_detected}', alpha=0),\n",
+    "]\n",
+    "\n",
+    "axes[0].legend(handles=legend_patches, title=\"Browsing Levels\", loc='upper left', fontsize=12, title_fontsize=12, frameon=True)\n",
+    "plt.savefig(f'{plot_dir}/mivolo.svg', format='svg', bbox_inches='tight')\n",
+    "plt.tight_layout()\n",
+    "\n",
+    "# Count images with at least one person\n",
+    "images_with_one_or_more_persons = all_data[all_data['n_persons'] > 0]['id'].nunique()\n",
+    "\n",
+    "# Count unique images in `person_data`\n",
+    "n_persons_detected = person_data['id'].nunique()\n",
+    "\n",
+    "# Count total persons detected\n",
+    "total_persons = person_data.shape[0]  # This counts all detected persons\n",
+    "\n",
+    "# Display potential inconsistencies\n",
+    "print(f\"Total images: {n_images}\")\n",
+    "print(f\"Images with at least one person: {images_with_one_or_more_persons}\")\n",
+    "print(f\"Unique images in `person_data`: {n_persons_detected}\")\n",
+    "print(f\"Total number of persons detected: {total_persons}\")\n",
+    "\n",
+    "\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "42b2a557-b8f4-4d0f-8907-98e3012a1b34",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T20:01:54.848400Z",
+     "iopub.status.busy": "2025-02-06T20:01:54.847713Z",
+     "iopub.status.idle": "2025-02-06T20:01:54.851533Z",
+     "shell.execute_reply": "2025-02-06T20:01:54.851102Z",
+     "shell.execute_reply.started": "2025-02-06T20:01:54.848376Z"
+    }
+   },
+   "source": [
+    "### Latex Table"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3e506c41-6497-4ece-99f3-73f09fe1129e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Define the directory containing CSV files\n",
+    "directory_path = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'  # Update this path with the actual directory path\n",
+    "\n",
+    "# Prepare data for LaTeX table\n",
+    "table_rows = []\n",
+    "\n",
+    "# Loop through each file in the directory\n",
+    "for file_name in os.listdir(directory_path):\n",
+    "    if file_name.endswith('.csv'):\n",
+    "        file_path = os.path.join(directory_path, file_name)\n",
+    "        print(f\"Processing file: {file_name}\")\n",
+    "\n",
+    "        # Load the data\n",
+    "        data = pd.read_csv(file_path)\n",
+    "\n",
+    "        # Total images analyzed\n",
+    "        total_images = data['id'].nunique()\n",
+    "\n",
+    "        # Count of images with no persons detected\n",
+    "        images_no_persons = data[data['n_persons'] == 0]['id'].nunique()\n",
+    "\n",
+    "        # Total persons detected (only using \"person\" detection type)\n",
+    "        total_persons_count = data[data['detection_type'] == 'person'].shape[0]\n",
+    "\n",
+    "        # Average age and standard deviation for male and female individuals\n",
+    "        male_age_stats = data[data['gender'] == 'male']['age'].agg(['mean', 'std']).fillna(0)\n",
+    "        female_age_stats = data[data['gender'] == 'female']['age'].agg(['mean', 'std']).fillna(0)\n",
+    "\n",
+    "        # Count of female and male subjects\n",
+    "        female_images_count = data[data['gender'] == 'female']['id'].nunique()\n",
+    "        male_images_count = data[data['gender'] == 'male']['id'].nunique()\n",
+    "\n",
+    "        # Female to male ratio\n",
+    "        female_to_male_ratio = female_images_count / male_images_count if male_images_count else None\n",
+    "\n",
+    "        # Browsing level analysis for females\n",
+    "        female_browsing_level_1 = data[(data['gender'] == 'female') & (data['browsingLevel'] == 1)]['id'].nunique()\n",
+    "        female_browsing_level_2_16 = data[(data['gender'] == 'female') & (data['browsingLevel'].isin([2, 4, 8, 16]))]['id'].nunique()\n",
+    "        \n",
+    "        female_browsing_level_1_percentage = (female_browsing_level_1 / female_images_count * 100) if female_images_count else 0\n",
+    "        female_browsing_level_2_16_percentage = (female_browsing_level_2_16 / female_images_count * 100) if female_images_count else 0\n",
+    "\n",
+    "        # Browsing level analysis for males\n",
+    "        male_browsing_level_1 = data[(data['gender'] == 'male') & (data['browsingLevel'] == 1)]['id'].nunique()\n",
+    "        male_browsing_level_2_16 = data[(data['gender'] == 'male') & (data['browsingLevel'].isin([2, 4, 8, 16]))]['id'].nunique()\n",
+    "\n",
+    "        male_browsing_level_1_percentage = (male_browsing_level_1 / male_images_count * 100) if male_images_count else 0\n",
+    "        male_browsing_level_2_16_percentage = (male_browsing_level_2_16 / male_images_count * 100) if male_images_count else 0\n",
+    "\n",
+    "        # Add row to table data\n",
+    "        table_rows.append([\n",
+    "            file_name.replace('.csv', ''),  # Remove file extension\n",
+    "            total_images,\n",
+    "            total_persons_count,\n",
+    "            images_no_persons,\n",
+    "            f\"{female_browsing_level_1_percentage:.2f}\",\n",
+    "            f\"{female_browsing_level_2_16_percentage:.2f}\",\n",
+    "            f\"{male_browsing_level_1_percentage:.2f}\",\n",
+    "            f\"{male_browsing_level_2_16_percentage:.2f}\",\n",
+    "            f\"{female_to_male_ratio:.2f}\" if female_to_male_ratio is not None else \"N/A\",\n",
+    "            f\"{female_age_stats['mean']:.2f} ({female_age_stats['std']:.2f})\",\n",
+    "            f\"{male_age_stats['mean']:.2f} ({male_age_stats['std']:.2f})\"\n",
+    "        ])\n",
+    "\n",
+    "# Sort table rows by the filename (assumes filenames are formatted with sortable dates)\n",
+    "table_rows = sorted(table_rows, key=lambda x: x[0])\n",
+    "\n",
+    "# Generate LaTeX table\n",
+    "latex_table = r\"\"\"\n",
+    "\\begin{table}[H]\n",
+    "\\centering\n",
+    "\\scriptsize\n",
+    "\\renewcommand{\\arraystretch}{0.9}\n",
+    "\\caption{Summary of Image Classification for 2023-2024}\n",
+    "\\label{table:image_classification_2023_2024}\n",
+    "\\begin{tabular}{lrrrrrrrrrr}\n",
+    "\\toprule\n",
+    "File Name & Total Images & Total Persons & No Persons & \\multicolumn{2}{c}{Female (\\%)} & \\multicolumn{2}{c}{Male (\\%)} & Female:Male & Female Age (Mean ± SD) & Male Age (Mean ± SD) \\\\\n",
+    " & & & & L1 & L2-16 & L1 & L2-16 & & & \\\\\n",
+    "\\midrule\n",
+    "\"\"\"\n",
+    "for row in table_rows:\n",
+    "    latex_table += \" & \".join(map(str, row)) + r\" \\\\\\\\\\n\"\n",
+    "\n",
+    "latex_table += r\"\"\"\n",
+    "\\bottomrule\n",
+    "\\end{tabular}\n",
+    "\\vspace{1em}\n",
+    "\\noindent\n",
+    "\\textbf{Disclaimer:} \\(\\female\\) and \\(\\male\\) refer to female-read and male-read classifications as determined by the MiVOLO system's weights. \n",
+    "We acknowledge the complexities of gender presentations and stress that these terms do not necessarily correspond to biological sex.\n",
+    "\\end{table}\n",
+    "\"\"\"\n",
+    "\n",
+    "# Output LaTeX table\n",
+    "print(\"\\nGenerated LaTeX Table:\")\n",
+    "print(latex_table)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3ef428be-856b-4c4c-b1b0-a052d181d03b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "HORDE",
+   "language": "python",
+   "name": "horde"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-1_Tag_occurences-checkpoint.ipynb

@@ -0,0 +1,6 @@
+{
+ "cells": [],
+ "metadata": {},
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4_Bloomz_query-checkpoint.ipynb

@@ -0,0 +1,370 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3b87c378-241e-41ab-be6e-84222594f22f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
+    "\n",
+    "# Import is used for pd.notna() and pd.isna() checks\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = True\n",
+    "TEST_SIZE = 10\n",
+    "MAX_ROWS = 20000\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "# Model settings - BLOOMZ (BigScience - European consortium)\n",
+    "MODEL_NAME = \"bigscience/bloomz-7b1\"  # Largest instruction-tuned BLOOM model\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"public figure\", \"voice actor/ASMR\",\n",
+    "    \"sports professional\", \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~14GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR)\n",
+    "    )\n",
+    "    print(\"✅ Tokenizer loaded\")\n",
+    "except Exception as e:\n",
+    "    print(f\"❌ Error loading tokenizer: {e}\")\n",
+    "    raise\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "    print(f\"Set pad_token to eos_token: {tokenizer.eos_token}\")\n",
+    "\n",
+    "# Load model with optimizations\n",
+    "print(\"Loading model (this may take several minutes)...\")\n",
+    "try:\n",
+    "    model = AutoModelForCausalLM.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        torch_dtype=torch.bfloat16,  # Use BF16 for efficiency\n",
+    "        device_map=\"auto\",  # Automatically distribute across GPUs\n",
+    "        low_cpu_mem_usage=True  # Optimize memory usage\n",
+    "    )\n",
+    "    model.eval()  # Set to evaluation mode\n",
+    "    print(\"✅ Model loaded\")\n",
+    "except Exception as e:\n",
+    "    print(f\"❌ Error loading model: {e}\")\n",
+    "    raise\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"Running in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPT (Exact DeepSeek style) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt.\"\"\"\n",
+    "    name = row.get('real_name', row.get('name', ''))\n",
+    "    if pd.isna(name):\n",
+    "        name = row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints exactly like DeepSeek version\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "\n",
+    "# === QUERY BLOOMZ LOCAL ===\n",
+    "def query_bloomz_local(prompt: str) -> str:\n",
+    "    \"\"\"Query BLOOMZ-7B1 locally via transformers, return raw response string.\"\"\"\n",
+    "    try:\n",
+    "        # BLOOMZ works better with instruction-response format\n",
+    "        full_prompt = f\"\"\"Instruction: Extract key data on a person based on the name and hints.\n",
+    "You must respond with exactly 5 numbered lines in this format:\n",
+    "1. Full legal name\n",
+    "2. Stage names/aliases  \n",
+    "3. Gender\n",
+    "4. Professions (comma-separated, choose ONLY from: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality)\n",
+    "5. Country\n",
+    "\n",
+    "{prompt}\n",
+    "\n",
+    "Response:\"\"\"\n",
+    "        \n",
+    "        inputs = tokenizer(\n",
+    "            full_prompt, \n",
+    "            return_tensors=\"pt\", \n",
+    "            truncation=True,\n",
+    "            max_length=2048\n",
+    "        ).to(device)\n",
+    "        \n",
+    "        # Generate with adjusted parameters for BLOOMZ\n",
+    "        with torch.no_grad():\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=256,\n",
+    "                temperature=0.3,  # Increased for more variability\n",
+    "                do_sample=True,\n",
+    "                top_p=0.9,\n",
+    "                top_k=40,\n",
+    "                repetition_penalty=1.1,\n",
+    "                pad_token_id=tokenizer.eos_token_id,  # Use EOS as pad token\n",
+    "                eos_token_id=tokenizer.eos_token_id,\n",
+    "                early_stopping=True\n",
+    "            )\n",
+    "        \n",
+    "        # Decode the entire output to see what's happening\n",
+    "        full_output = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
+    "        \n",
+    "        # Extract only the generated part (after the prompt)\n",
+    "        generated_text = full_output[len(tokenizer.decode(inputs['input_ids'][0], skip_special_tokens=True)):]\n",
+    "        \n",
+    "        # Debug output\n",
+    "        if not hasattr(query_bloomz_local, 'debug_count'):\n",
+    "            query_bloomz_local.debug_count = 0\n",
+    "        \n",
+    "        if query_bloomz_local.debug_count < 3:\n",
+    "            print(f\"\\n📝 BLOOMZ Debug #{query_bloomz_local.debug_count + 1}:\")\n",
+    "            print(f\"Prompt: {full_prompt[:200]}...\")\n",
+    "            print(f\"Full output: {full_output[:500]}...\")\n",
+    "            print(f\"Generated text: {generated_text}\")\n",
+    "            print(f\"{'='*60}\\n\")\n",
+    "            query_bloomz_local.debug_count += 1\n",
+    "        \n",
+    "        return generated_text.strip()\n",
+    "        \n",
+    "    except Exception as e:\n",
+    "        print(f\"Error querying BLOOMZ: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PARSE RESPONSE (Exact DeepSeek format) ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse numbered response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.') or line.startswith('1)'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.') or line.startswith('2)'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.') or line.startswith('3)'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.') or line.startswith('4)'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.') or line.startswith('5)'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/bloomz_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/bloomz_query_index.txt\"\n",
+    "\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    with open(index_file) as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "# Initialize columns (same as DeepSeek)\n",
+    "for col in ['full_name', 'gender', 'profession_llm', 'country', 'aliases']:\n",
+    "    if col not in df.columns:\n",
+    "        df[col] = 'Unknown'\n",
+    "\n",
+    "# Create prompts for all rows (same as DeepSeek)\n",
+    "print(\"Creating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "\n",
+    "print(f\"\\nAnnotating with BLOOMZ-7B1 LOCAL - rows {current_index} to {len(df)}...\")\n",
+    "print(f\"Model: {MODEL_NAME}\")\n",
+    "print(f\"This may take a while...\\n\")\n",
+    "\n",
+    "try:\n",
+    "    start_time = time.time()\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=\"Annotating\"):\n",
+    "        row = df.iloc[i]\n",
+    "        \n",
+    "        # Query BLOOMZ (equivalent to DeepSeek query)\n",
+    "        response = query_bloomz_local(row['prompt'])\n",
+    "        parsed_data = parse_response(response)\n",
+    "        \n",
+    "        # Update dataframe\n",
+    "        for key, value in parsed_data.items():\n",
+    "            df.at[i, key] = value\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        # Save progress at intervals\n",
+    "        if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            with open(index_file, 'w') as f:\n",
+    "                f.write(str(current_index))\n",
+    "            print(f\"✅ Progress saved after {i+1} rows\")\n",
+    "        \n",
+    "        # Optional: Add small delay to prevent overheating (not needed for rate limiting like DeepSeek)\n",
+    "        # time.sleep(0.1)\n",
+    "    \n",
+    "    elapsed_total = time.time() - start_time\n",
+    "    print(f\"\\n✅ Done! Final results saved to {output_file}\")\n",
+    "    \n",
+    "    # Summary statistics (same as DeepSeek)\n",
+    "    print(\"\\n=== Summary Statistics ===\")\n",
+    "    print(f\"Total processed: {len(df)}\")\n",
+    "    print(f\"\\nGender distribution:\")\n",
+    "    print(df['gender'].value_counts())\n",
+    "    print(f\"\\nTop 10 profession combinations:\")\n",
+    "    print(df['profession_llm'].value_counts().head(10))\n",
+    "    print(f\"\\nTop 10 countries:\")\n",
+    "    print(df['country'].value_counts().head(10))\n",
+    "    \n",
+    "    # Sample results\n",
+    "    print(\"\\n=== Sample Results ===\")\n",
+    "    display_cols = ['real_name', 'full_name', 'gender', 'profession_llm', 'country']\n",
+    "    available_cols = [col for col in display_cols if col in df.columns]\n",
+    "    print(df[available_cols].head(10).to_string(index=False))\n",
+    "    \n",
+    "    # Additional info for local model\n",
+    "    print(f\"\\nTotal time: {elapsed_total/60:.1f} minutes\")\n",
+    "    print(f\"Average speed: {len(df)/(elapsed_total/3600):.1f} samples/hour\")\n",
+    "    if torch.cuda.is_available():\n",
+    "        print(f\"Final VRAM usage: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB\")\n",
+    "\n",
+    "except Exception as e:\n",
+    "    print(f\"⚠️ Error encountered: {e}\")\n",
+    "    print(f\"⚠️ Last processed index: {current_index}\")\n",
+    "    \n",
+    "    # Save progress before exiting\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    with open(index_file, 'w') as f:\n",
+    "        f.write(str(current_index))\n",
+    "    \n",
+    "    print(f\"⚠️ Progress saved up to row {current_index}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4_Figure_8_Step_1_LLM_annotation-checkpoint.ipynb

@@ -0,0 +1,1941 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "23d0ae58",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset - LLM Annotation Pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4407358",
+   "metadata": {},
+   "source": [
+    "### Unified Model Loading & Inference\n",
+    "Code for querying Mistral, Gemma, and Qwen models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a1b9d0e",
+   "metadata": {},
+   "source": [
+    "## CLEANING & PREPROCESSING"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3df42c46",
+   "metadata": {},
+   "source": [
+    "#### Named Entity Recognitition (NER) using SpaCy "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a287eef4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "import emoji\n",
+    "import spacy\n",
+    "\n",
+    "# Load spaCy model\n",
+    "# You may need to download it first: python -m spacy download en_core_web_sm\n",
+    "try:\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model loaded: en_core_web_sm\")\n",
+    "except OSError:\n",
+    "    print(\"❌ spaCy model not found. Downloading...\")\n",
+    "    import subprocess\n",
+    "    subprocess.run([\"python\", \"-m\", \"spacy\", \"download\", \"en_core_web_sm\"])\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model downloaded and loaded\")\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "#input_file = current_dir.parent / \"data/CSV/real_person_adapters.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter.csv\"\n",
+    "\n",
+    "# Load dataset\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "def translate_leetspeak(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Translate common leetspeak patterns to normal letters.\n",
+    "    Examples: 4kira -> Akira, 3mma -> Emma, 1rene -> Irene\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return text\n",
+    "    \n",
+    "    # Common leetspeak mappings (order matters!)\n",
+    "    leetspeak_map = {\n",
+    "        '4': 'a',\n",
+    "        '3': 'e', \n",
+    "        '1': 'i',\n",
+    "        '0': 'o',\n",
+    "        '7': 't',\n",
+    "        '5': 's',\n",
+    "        '8': 'b',\n",
+    "        '9': 'g',\n",
+    "        '@': 'a',\n",
+    "        '$': 's',\n",
+    "        '!': 'i',\n",
+    "    }\n",
+    "    \n",
+    "    result = text\n",
+    "    # Apply mappings at word boundaries or start of string\n",
+    "    for leet, normal in leetspeak_map.items():\n",
+    "        # Replace at start of word\n",
+    "        result = re.sub(rf'\\b{re.escape(leet)}', normal, result, flags=re.IGNORECASE)\n",
+    "        # Replace standalone numbers that look like letters in context\n",
+    "        result = re.sub(rf'(?<=[a-z]){re.escape(leet)}(?=[a-z])', normal, result, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    return result\n",
+    "\n",
+    "def preprocess_for_ner(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Preprocess the name before spaCy NER.\n",
+    "    Remove noise but keep the actual name parts.\n",
+    "    \"\"\"\n",
+    "    if pd.isna(name):\n",
+    "        return \"\"\n",
+    "    \n",
+    "    name = str(name)\n",
+    "    \n",
+    "    # FIRST: Translate leetspeak\n",
+    "    name = translate_leetspeak(name)\n",
+    "    \n",
+    "    # Remove emoji\n",
+    "    name = emoji.replace_emoji(name, replace=' ')\n",
+    "    \n",
+    "    # Remove version indicators (v1, v2, v1.0, etc.)\n",
+    "    name = re.sub(r'\\s*[vV]\\d+(\\.\\d+)?\\s*', ' ', name)\n",
+    "    \n",
+    "    # Remove LoRA-related terms (case insensitive)\n",
+    "    lora_terms = ['lora', 'loha', 'lycoris', 'controlnet', 'textual inversion', \n",
+    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter', 'pony', 'sdxl', 'flux', 'illustrious', 'sd14', 'sd14', 'sd2', 'sd3', 'diffusion', 'stable', 'hunyuan']\n",
+    "    for term in lora_terms:\n",
+    "        name = re.sub(rf'\\b{term}\\b', '', name, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    # Remove content in parentheses or brackets (often metadata)\n",
+    "    name = re.sub(r'\\([^)]*\\)', '', name)\n",
+    "    name = re.sub(r'\\[[^\\]]*\\]', '', name)\n",
+    "    \n",
+    "    # Remove special characters like 「」\n",
+    "    name = re.sub(r'[「」『』【】〈〉《》]', '', name)\n",
+    "    \n",
+    "    # Handle pipe - keep first part\n",
+    "    if '|' in name:\n",
+    "        name = name.split('|')[0]\n",
+    "    \n",
+    "    # Handle forward slash - keep first part\n",
+    "    if '/' in name:\n",
+    "        name = name.split('/')[0]\n",
+    "    \n",
+    "    # Replace underscores with spaces\n",
+    "    name = name.replace('_', ' ')\n",
+    "    \n",
+    "    # Remove multiple spaces\n",
+    "    name = re.sub(r'\\s+', ' ', name)\n",
+    "    \n",
+    "    # Strip\n",
+    "    name = name.strip()\n",
+    "    \n",
+    "    return name\n",
+    "\n",
+    "def extract_person_name(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Use spaCy NER to extract person names from text.\n",
+    "    Falls back to cleaned text if no PERSON entity found.\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Run spaCy NER\n",
+    "    doc = nlp(text)\n",
+    "    \n",
+    "    # Extract PERSON entities\n",
+    "    person_entities = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    \n",
+    "    if person_entities:\n",
+    "        # Return the first (usually longest/best) person name\n",
+    "        return person_entities[0].strip()\n",
+    "    \n",
+    "    # If no PERSON entity found, try to extract capitalized words (likely names)\n",
+    "    # This helps with names spaCy might miss\n",
+    "    words = text.split()\n",
+    "    capitalized_words = [w for w in words if w and w[0].isupper() and len(w) > 1]\n",
+    "    \n",
+    "    if capitalized_words:\n",
+    "        # Join first few capitalized words (likely the name)\n",
+    "        return ' '.join(capitalized_words[:3]).strip()\n",
+    "    \n",
+    "    # Last resort: return cleaned text\n",
+    "    return text.strip()\n",
+    "\n",
+    "def clean_name_with_spacy(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Complete name cleaning pipeline with spaCy NER.\n",
+    "    \n",
+    "    Pipeline:\n",
+    "    1. Translate leetspeak (4→a, 3→e, 1→i, etc.)\n",
+    "    2. Remove noise (emoji, version tags, LoRA terms)\n",
+    "    3. Use spaCy to extract PERSON entities\n",
+    "    4. Fallback to capitalized words or cleaned text\n",
+    "    \"\"\"\n",
+    "    # Step 1 & 2: Preprocess (leetspeak + noise removal)\n",
+    "    preprocessed = preprocess_for_ner(name)\n",
+    "    \n",
+    "    if not preprocessed:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Step 3: Extract person name using spaCy NER\n",
+    "    person_name = extract_person_name(preprocessed)\n",
+    "    \n",
+    "    return person_name\n",
+    "\n",
+    "# Apply name cleaning with spaCy\n",
+    "print(\"\\n🔄 Processing names with spaCy NER...\")\n",
+    "df['real_name'] = df['name'].apply(clean_name_with_spacy)\n",
+    "\n",
+    "# Show examples with detailed comparison\n",
+    "print(\"\\n📊 Name cleaning examples (with spaCy NER):\")\n",
+    "print(\"=\" * 100)\n",
+    "print(f\"{'Original Name':<50} | {'Cleaned Name':<30}\")\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "examples = df[['name', 'real_name']].head(30)\n",
+    "shown = 0\n",
+    "for idx, row in examples.iterrows():\n",
+    "    if row['name'] != row['real_name'] and shown < 20:\n",
+    "        print(f\"{row['name']:<50} | {row['real_name']:<30}\")\n",
+    "        shown += 1\n",
+    "\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "# Show specific test cases\n",
+    "print(\"\\n🧪 Leetspeak translation examples:\")\n",
+    "test_names = ['4kira LoRA', '3mma Watson v2', '1rene LORA', 'L3vi Ackerman']\n",
+    "for test in test_names:\n",
+    "    result = clean_name_with_spacy(test)\n",
+    "    print(f\"  {test:<30} -> {result}\")\n",
+    "\n",
+    "# Statistics\n",
+    "print(f\"\\n📈 Statistics:\")\n",
+    "print(f\"  Total rows: {len(df)}\")\n",
+    "print(f\"  Non-empty names: {(df['real_name'] != '').sum()}\")\n",
+    "print(f\"  Empty names: {(df['real_name'] == '').sum()}\")\n",
+    "\n",
+    "# Show some examples of what spaCy identified\n",
+    "print(\"\\n🎯 Sample spaCy NER results:\")\n",
+    "sample_names = df['real_name'].head(20).tolist()\n",
+    "for i, name in enumerate(sample_names[:10], 1):\n",
+    "    if name:\n",
+    "        print(f\"  {i}. {name}\")\n",
+    "\n",
+    "print(f\"\\n✅ Cleaned {len(df)} names using spaCy NER\")\n",
+    "\n",
+    "# Save intermediate result\n",
+    "output_step1 = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "df.to_csv(output_step1, index=False)\n",
+    "print(f\"💾 Saved to {output_step1}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64687c72",
+   "metadata": {},
+   "source": [
+    "#### STEP 02: Nationality tag to Country hint\n",
+    "here tags related to nationality gets converted to the country equivalent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d6eaef5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries_file = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "output_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(input_file)\n",
+    "countries_df = pd.read_csv(countries_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Define uninhabited or non-relevant territories to exclude\n",
+    "excluded_territories = {\n",
+    "    'isle of man', 'bouvet island', 'heard island and mcdonald islands',\n",
+    "    'french southern territories', 'south georgia and the south sandwich islands',\n",
+    "    'svalbard and jan mayen', 'british indian ocean territory', 'antarctica',\n",
+    "    'christmas island', 'cocos (keeling) islands', 'norfolk island',\n",
+    "    'pitcairn', 'tokelau', 'united states minor outlying islands',\n",
+    "    'wallis and futuna', 'western sahara'\n",
+    "}\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag → (country, nationality) mapping with PRIORITIES\n",
+    "tag_to_country_nationality = {}\n",
+    "# We'll use a priority score: direct country name = 3, nationality = 2, word parts = 1\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "    \n",
+    "    # Skip excluded territories\n",
+    "    if country.lower() in excluded_territories:\n",
+    "        continue\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Store as (country, nationality, priority)\n",
+    "    # Exact country name match = highest priority\n",
+    "    if country_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_lc] = (country, \"\", 3)\n",
+    "    \n",
+    "    # Exact nationality match = medium priority  \n",
+    "    if nationality_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_lc] = (\"\", nationality, 2)\n",
+    "    \n",
+    "    # No-space versions\n",
+    "    country_no_space = country_lc.replace(\" \", \"\")\n",
+    "    nationality_no_space = nationality_lc.replace(\" \", \"\")\n",
+    "    \n",
+    "    if country_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_no_space] = (country, \"\", 3)\n",
+    "    if nationality_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_no_space] = (\"\", nationality, 2)\n",
+    "\n",
+    "    # Word parts = lowest priority (only for longer words to avoid false matches)\n",
+    "    for part in country_lc.split():\n",
+    "        if len(part) > 4:  # Only words longer than 4 chars\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (country, \"\", 1)\n",
+    "    for part in nationality_lc.split():\n",
+    "        if len(part) > 4:\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (\"\", nationality, 1)\n",
+    "\n",
+    "print(f\"Built country/nationality mapping with {len(tag_to_country_nationality)} entries\")\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality by checking ALL tags\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    \"\"\"\n",
+    "    Check ALL tags and return the best match based on priority.\n",
+    "    Priority: exact country name > nationality > word parts\n",
+    "    \"\"\"\n",
+    "    best_match = None\n",
+    "    best_priority = 0\n",
+    "    \n",
+    "    for tag in tags:\n",
+    "        # Try cleaned version (no spaces)\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        \n",
+    "        # Check cleaned version\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[cleaned]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "        \n",
+    "        # Also check original tag\n",
+    "        if tag in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[tag]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "    \n",
+    "    if best_match:\n",
+    "        return pd.Series(best_match)\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag → profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(output_file, index=False)\n",
+    "\n",
+    "# Preview results\n",
+    "print(f\"\\nProcessed {len(poi_df)} rows\")\n",
+    "print(f\"Rows with country: {(poi_df['likely_country'] != '').sum()}\")\n",
+    "print(f\"Rows with nationality: {(poi_df['likely_nationality'] != '').sum()}\")\n",
+    "print(f\"Rows with profession: {(poi_df['likely_profession'] != '').sum()}\")\n",
+    "\n",
+    "print(f\"\\nTop 10 countries:\")\n",
+    "print(poi_df[poi_df['likely_country'] != '']['likely_country'].value_counts().head(10))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a4a58b3",
+   "metadata": {},
+   "source": [
+    "## LLM ANNOTATION"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b298844d",
+   "metadata": {},
+   "source": [
+    "#### Model Configurations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "39f3d65e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "import signal\n",
+    "from contextlib import contextmanager\n",
+    "\n",
+    "# Configuration\n",
+    "current_dir = Path.cwd()\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Model configurations\n",
+    "MODEL_CONFIGS = {\n",
+    "    'mistral': {\n",
+    "        'name': 'mistralai/Mistral-7B-Instruct-v0.3',\n",
+    "        'dtype': torch.bfloat16,\n",
+    "        'quantization': None,\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.05,\n",
+    "            'do_sample': True,\n",
+    "            'top_p': 1.0,\n",
+    "        }\n",
+    "    },\n",
+    "    'gemma': {\n",
+    "        'name': 'google/gemma-3-27b-it',\n",
+    "        'dtype': torch.bfloat16,\n",
+    "        'quantization': None,\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.1,\n",
+    "            'do_sample': True,\n",
+    "            'top_p': 1.0,\n",
+    "        }\n",
+    "    },\n",
+    "    'qwen': {\n",
+    "        'name': 'Qwen/Qwen2.5-32B-Instruct',\n",
+    "        'dtype': None,  # Will use quantization\n",
+    "        'quantization': BitsAndBytesConfig(\n",
+    "            load_in_8bit=True,\n",
+    "            llm_int8_threshold=6.0,\n",
+    "            llm_int8_has_fp16_weight=False\n",
+    "        ),\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.1,\n",
+    "            'do_sample': False,\n",
+    "        }\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c215b38c",
+   "metadata": {},
+   "source": [
+    "#### Load Model Function"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfb5b13e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def load_model(model_type='mistral'):\n",
+    "    \"\"\"\n",
+    "    Load model and tokenizer based on type.\n",
+    "    \n",
+    "    Args:\n",
+    "        model_type: 'mistral', 'gemma', or 'qwen'\n",
+    "    \n",
+    "    Returns:\n",
+    "        tuple: (model, tokenizer, config)\n",
+    "    \"\"\"\n",
+    "    if model_type not in MODEL_CONFIGS:\n",
+    "        raise ValueError(f\"Unknown model type: {model_type}. Choose from {list(MODEL_CONFIGS.keys())}\")\n",
+    "    \n",
+    "    config = MODEL_CONFIGS[model_type]\n",
+    "    model_name = config['name']\n",
+    "    \n",
+    "    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "    print(f\"Loading model: {model_name}\")\n",
+    "    print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "    print(f\"Device: {device}\\n\")\n",
+    "    \n",
+    "    if device == \"cpu\":\n",
+    "        print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    \n",
+    "    # Load tokenizer\n",
+    "    try:\n",
+    "        tokenizer = AutoTokenizer.from_pretrained(\n",
+    "            model_name,\n",
+    "            cache_dir=str(CACHE_DIR),\n",
+    "            use_fast=True\n",
+    "        )\n",
+    "    except:\n",
+    "        tokenizer = AutoTokenizer.from_pretrained(\n",
+    "            model_name,\n",
+    "            cache_dir=str(CACHE_DIR),\n",
+    "            use_fast=False\n",
+    "        )\n",
+    "    \n",
+    "    if tokenizer.pad_token is None:\n",
+    "        tokenizer.pad_token = tokenizer.eos_token\n",
+    "    \n",
+    "    # Load model\n",
+    "    model_kwargs = {\n",
+    "        'cache_dir': str(CACHE_DIR),\n",
+    "        'device_map': 'auto',\n",
+    "        'trust_remote_code': False\n",
+    "    }\n",
+    "    \n",
+    "    if config['quantization']:\n",
+    "        model_kwargs['quantization_config'] = config['quantization']\n",
+    "    else:\n",
+    "        model_kwargs['torch_dtype'] = config['dtype']\n",
+    "    \n",
+    "    model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)\n",
+    "    model.eval()\n",
+    "    \n",
+    "    # Check VRAM\n",
+    "    if torch.cuda.is_available():\n",
+    "        vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "        print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "    \n",
+    "    return model, tokenizer, config\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11b2221a",
+   "metadata": {},
+   "source": [
+    "#### Inference Code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "229f96bd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@contextmanager\n",
+    "def timeout(duration):\n",
+    "    \"\"\"Context manager for timeout.\"\"\"\n",
+    "    def handler(signum, frame):\n",
+    "        raise TimeoutError(\"Operation timed out\")\n",
+    "    \n",
+    "    signal.signal(signal.SIGALRM, handler)\n",
+    "    signal.alarm(duration)\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        signal.alarm(0)\n",
+    "\n",
+    "def query_model(prompt, model, tokenizer, config, use_timeout=False):\n",
+    "    \"\"\"\n",
+    "    Query model with given prompt.\n",
+    "    \n",
+    "    Args:\n",
+    "        prompt: Input prompt string\n",
+    "        model: Loaded model\n",
+    "        tokenizer: Loaded tokenizer\n",
+    "        config: Model configuration dict\n",
+    "        use_timeout: Whether to use 60s timeout (for Qwen)\n",
+    "    \n",
+    "    Returns:\n",
+    "        str: Model response or None on error\n",
+    "    \"\"\"\n",
+    "    try:\n",
+    "        device = next(model.parameters()).device\n",
+    "        \n",
+    "        # Format as chat message\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are a data extraction assistant. Respond with exactly 5 numbered lines containing ONLY values. No labels, no explanations, no prefixes. Follow the format precisely.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generation parameters\n",
+    "        gen_kwargs = config['generation_params'].copy()\n",
+    "        gen_kwargs['pad_token_id'] = tokenizer.eos_token_id\n",
+    "        \n",
+    "        # Generate\n",
+    "        generation_fn = lambda: model.generate(**inputs, **gen_kwargs)\n",
+    "        \n",
+    "        if use_timeout:\n",
+    "            with timeout(60):\n",
+    "                with torch.no_grad():\n",
+    "                    outputs = generation_fn()\n",
+    "        else:\n",
+    "            with torch.no_grad():\n",
+    "                outputs = generation_fn()\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except TimeoutError:\n",
+    "        print(f\"[ERROR] Generation timed out after 60 seconds\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"[ERROR] Generation failed: {e}\")\n",
+    "        return None\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "88f005f8",
+   "metadata": {},
+   "source": [
+    "#### Prompt creation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dfe05463",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def create_prompt(row):\n",
+    "    \"\"\"Create annotation prompt from row data.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if needed\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Extract information about '{name}' ({hint_text}).\n",
+    "\n",
+    "Respond with EXACTLY 5 numbered lines. Each line must contain ONLY the value, no labels or extra text.\n",
+    "\n",
+    "FORMAT REQUIREMENTS:\n",
+    "1. Full legal name in Western order (first last). VALUE ONLY.\n",
+    "2. Stage names/aliases, comma-separated. If none, write \"None\". VALUE ONLY.\n",
+    "3. Gender: MUST be exactly one word: Male, Female, Other, or Unknown. VALUE ONLY.\n",
+    "4. Professions: Choose up to 3 from this list ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality. Comma-separated. VALUE ONLY.\n",
+    "5. Primary country: Country name only (e.g., \"China\", \"United States\", \"Colombia\"). VALUE ONLY.\n",
+    "\n",
+    "RULES:\n",
+    "- Professions MUST match the exact categories listed (actress = actor)\n",
+    "- \"online personality\" includes streamers, cosplayers, YouTubers, influencers\n",
+    "- \"public figure\" includes politicians, activists, journalists, authors\n",
+    "- Use \"Unknown\" when uncertain or for fictional characters\n",
+    "- NO explanatory text, NO labels like \"Gender:\", NO prefixes\n",
+    "- For multi-role people, list up to 3 categories by relevance\n",
+    "\n",
+    "EXAMPLE FORMAT:\n",
+    "1. Taylor Swift\n",
+    "2. None\n",
+    "3. Female\n",
+    "4. singer/musician, public figure\n",
+    "5. United States\"\"\"\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "854fa668",
+   "metadata": {},
+   "source": [
+    "#### Response parsing code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a4be2ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def parse_response(response):\n",
+    "    \"\"\"Parse model response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            gender_raw = line[2:].strip()\n",
+    "            gender_raw = re.sub(r'^(Gender:|gender:)\\s*', '', gender_raw, flags=re.IGNORECASE)\n",
+    "            gender_match = re.search(r'\\b(Male|Female|Other|Unknown)\\b', gender_raw, re.IGNORECASE)\n",
+    "            fields['gender'] = gender_match.group(1).capitalize() if gender_match else gender_raw\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            country_raw = line[2:].strip()\n",
+    "            country_raw = re.sub(r'^(Primary\\s+)?(associated\\s+)?country:\\s*', '', country_raw, flags=re.IGNORECASE)\n",
+    "            fields['country'] = country_raw\n",
+    "    \n",
+    "    return fields\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7e2f7a86",
+   "metadata": {},
+   "source": [
+    "#### CSV annotation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5f3dd5d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def annotate_dataset(model_type='mistral', test_mode=False, test_size=100, max_rows=50862, save_interval=10):\n",
+    "    \"\"\"\n",
+    "    Annotate dataset using specified model.\n",
+    "    \n",
+    "    Args:\n",
+    "        model_type: 'mistral', 'gemma', or 'qwen'\n",
+    "        test_mode: If True, only process test_size rows\n",
+    "        test_size: Number of rows to process in test mode\n",
+    "        max_rows: Maximum rows to process\n",
+    "        save_interval: Save progress every N rows\n",
+    "    \"\"\"\n",
+    "    # Setup paths\n",
+    "    input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "    output_file = current_dir.parent / f\"data/CSV/{model_type}_local_annotated_POI{'_test' if test_mode else ''}.csv\"\n",
+    "    index_file = current_dir.parent / f\"misc/query_indicies/{model_type}_local_query_index.txt\"\n",
+    "    index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "    \n",
+    "    # Load model\n",
+    "    model, tokenizer, config = load_model(model_type)\n",
+    "    \n",
+    "    # Load data\n",
+    "    print(f\"Loaded {len(df)} rows from input file\")\n",
+    "    df = pd.read_csv(input_file)\n",
+    "    \n",
+    "    # Merge existing annotations if available\n",
+    "    if output_file.exists():\n",
+    "        existing_df = pd.read_csv(output_file)\n",
+    "        annotation_cols = ['full_name', 'aliases', 'gender', 'profession_llm', 'country']\n",
+    "        for col in annotation_cols:\n",
+    "            if col in existing_df.columns:\n",
+    "                df[col] = existing_df[col][:len(df)]\n",
+    "    \n",
+    "    # Apply limits\n",
+    "    if test_mode:\n",
+    "        df = df.head(test_size).copy()\n",
+    "    elif max_rows:\n",
+    "        df = df.head(max_rows).copy()\n",
+    "    \n",
+    "    # Create prompts\n",
+    "    df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "    \n",
+    "    # Load progress index\n",
+    "    current_index = 0\n",
+    "    if index_file.exists():\n",
+    "        try:\n",
+    "            current_index = int(index_file.read_text().strip())\n",
+    "        except:\n",
+    "            current_index = 0\n",
+    "    \n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "    \n",
+    "    # Process rows\n",
+    "    use_timeout = (model_type == 'qwen')\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=f\"{model_type.capitalize()} Annotation\"):\n",
+    "        prompt = df.at[i, \"prompt\"]\n",
+    "        \n",
+    "        # Query with retries\n",
+    "        response = None\n",
+    "        for attempt in range(3):\n",
+    "            response = query_model(prompt, model, tokenizer, config, use_timeout)\n",
+    "            \n",
+    "            if response and len(response.strip()) > 10:\n",
+    "                break\n",
+    "            \n",
+    "            print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "            time.sleep(0.5)\n",
+    "        \n",
+    "        # Skip if invalid\n",
+    "        if not response or len(response.strip()) <= 10:\n",
+    "            print(f\"❌ Row {i}: failed after retries, skipping\")\n",
+    "            continue\n",
+    "        \n",
+    "        # Parse and validate\n",
+    "        parsed = parse_response(response)\n",
+    "        \n",
+    "        if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "            print(f\"❌ Row {i}: parsed as all Unknown, skipping\")\n",
+    "            continue\n",
+    "        \n",
+    "        # Write fields\n",
+    "        for key, value in parsed.items():\n",
+    "            df.at[i, key] = value\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        # GPU cleanup\n",
+    "        if torch.cuda.is_available():\n",
+    "            torch.cuda.empty_cache()\n",
+    "            torch.cuda.synchronize()\n",
+    "        \n",
+    "        # Save progress\n",
+    "        if (i + 1) % save_interval == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            index_file.write_text(str(current_index))\n",
+    "            print(f\"💾 Progress saved after row {i+1}\")\n",
+    "    \n",
+    "    # Final save\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    index_file.write_text(str(current_index))\n",
+    "    print(f\"✓ Finished annotation with {model_type}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55da2f4c",
+   "metadata": {},
+   "source": [
+    "### Usage Examples\n",
+    "Run annotation with your chosen model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "351ea40c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example 1: Annotate with Mistral (13.5 GB VRAM)\n",
+    "# annotate_dataset(model_type='mistral', test_mode=False)\n",
+    "\n",
+    "# Example 2: Annotate with Gemma (56.3 GB VRAM)\n",
+    "# annotate_dataset(model_type='gemma', test_mode=False)\n",
+    "\n",
+    "# Example 3: Annotate with Qwen (32.7 GB VRAM, 8-bit)\n",
+    "# annotate_dataset(model_type='qwen', test_mode=False)\n",
+    "\n",
+    "# Test mode (first 100 rows)\n",
+    "# annotate_dataset(model_type='mistral', test_mode=True, test_size=100)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6431d347-d80c-4e8b-83a7-531e5df95a72",
+   "metadata": {},
+   "source": [
+    "## EuroLLM-9B-Instruct"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8203abc-e7c3-4cb6-aaeb-fdc6933981fc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "import signal\n",
+    "from contextlib import contextmanager\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/eurollm_local_query_index.txt\"\n",
+    "output_file = current_dir.parent / f\"data/CSV/eurollm_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "\n",
+    "# Model settings\n",
+    "MODEL_NAME = \"utter-project/EuroLLM-9B-Instruct\"\n",
+    "#MODEL_NAME = \"Qwen/Qwen2.5-32B-Instruct\"\n",
+    "#MODEL_NAME = \"Qwen/Qwen2.5-14B-Instruct\"\n",
+    "#MODEL_NAME =  \"Qwen/Qwen3-235B-A22B-Instruct-2507-FP8\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Get HF token from credentials file\n",
+    "import os\n",
+    "credentials_dir = current_dir.parent / \"misc/credentials\"\n",
+    "hf_token_file = credentials_dir / \"hf_token.txt\"\n",
+    "\n",
+    "HF_TOKEN = None\n",
+    "if hf_token_file.exists():\n",
+    "    HF_TOKEN = hf_token_file.read_text().strip()\n",
+    "    print(\"✅ HF token loaded from credentials file\")\n",
+    "else:\n",
+    "    print(\"⚠️  HF token file not found at:\", hf_token_file)\n",
+    "    print(\"    The script will try to use cached credentials from 'huggingface-cli login'\")\n",
+    "    print(\"    Or create the file: misc/credentials/hf_token.txt with your token\")\n",
+    "    HF_TOKEN = None  # Will use cached token if available\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True,\n",
+    "        token=HF_TOKEN\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False,\n",
+    "        token=HF_TOKEN\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "# Configure 8-bit quantization for A100\n",
+    "print(\"Configuring 8-bit quantization...\")\n",
+    "quantization_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True,\n",
+    "    llm_int8_threshold=6.0,\n",
+    "    llm_int8_has_fp16_weight=False\n",
+    ")\n",
+    "\n",
+    "# Load model with 8-bit quantization\n",
+    "print(\"Loading model with 8-bit quantization (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    quantization_config=quantization_config,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False,\n",
+    "    token=HF_TOKEN\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded with 8-bit quantization\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "if output_file.exists():\n",
+    "    print(\"Loading annotated CSV...\")\n",
+    "    df = pd.read_csv(output_file)\n",
+    "else:\n",
+    "    print(\"Loading raw input CSV...\")\n",
+    "    df = pd.read_csv(input_file)\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS (OPTIMIZED FOR CLEAN OUTPUTS) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for EuroLLM annotation with strict formatting requirements.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Extract information about '{name}'. \n",
+    "Context hints (DO NOT copy these as professions): {hint_text}\n",
+    "\n",
+    "Respond with EXACTLY 5 numbered lines. Each line must contain ONLY the value, no labels or extra text.\n",
+    "\n",
+    "FORMAT REQUIREMENTS:\n",
+    "1. Full legal name in Western order (first last). VALUE ONLY.\n",
+    "2. Stage names/aliases, comma-separated. If none, write \"None\". VALUE ONLY.\n",
+    "3. Gender: MUST be exactly one word: Male, Female, Other, or Unknown. VALUE ONLY.\n",
+    "4. Professions: Choose up to 3 from this list ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality. Comma-separated. VALUE ONLY.\n",
+    "5. Primary country: Country name only (e.g., \"China\", \"United States\", \"Colombia\"). VALUE ONLY.\n",
+    "\n",
+    "CRITICAL RULES FOR PROFESSIONS (Line 4):\n",
+    "- ONLY use the exact profession categories listed above\n",
+    "- DO NOT use descriptive words like \"sexy\", \"photorealistic\", \"celebrity\"\n",
+    "- DO NOT copy the hint words as professions\n",
+    "- If uncertain about profession, write \"Unknown\"\n",
+    "- Valid professions are ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality\n",
+    "- Actress = actor, streamer = online personality, YouTuber = online personality\n",
+    "\n",
+    "OTHER RULES:\n",
+    "- Use \"Unknown\" when uncertain or for fictional characters\n",
+    "- NO explanatory text, NO labels like \"Gender:\", NO prefixes\n",
+    "- For multi-role people, list up to 3 categories by relevance\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "@contextmanager\n",
+    "def timeout(duration):\n",
+    "    def handler(signum, frame):\n",
+    "        raise TimeoutError(\"Operation timed out\")\n",
+    "    \n",
+    "    # Set the signal handler and alarm\n",
+    "    signal.signal(signal.SIGALRM, handler)\n",
+    "    signal.alarm(duration)\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        signal.alarm(0)  # Disable the alarm\n",
+    "\n",
+    "\n",
+    "def query_eurollm_local(prompt: str) -> str:\n",
+    "    \"\"\"Query EuroLLM locally via transformers with very low temperature.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for EuroLLM with strict system prompt\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are a data extraction assistant. Respond with exactly 5 numbered lines containing ONLY values. No labels, no explanations, no prefixes. Follow the format precisely.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template') and tokenizer.chat_template is not None:\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for models without chat template\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate with timeout and very low temperature\n",
+    "        with timeout(60):\n",
+    "            with torch.no_grad():\n",
+    "                outputs = model.generate(\n",
+    "                    **inputs,\n",
+    "                    max_new_tokens=100,\n",
+    "                    temperature=0.01,  # Very low temperature for more deterministic outputs\n",
+    "                    do_sample=True,     # Must be True when temperature is set\n",
+    "                    pad_token_id=tokenizer.eos_token_id\n",
+    "                )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except TimeoutError:\n",
+    "        print(f\"[ERROR] Generation timed out after 60 seconds\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        import traceback\n",
+    "        traceback.print_exc()\n",
+    "        return None\n",
+    "\n",
+    "    \n",
+    "# === PARSE RESPONSE WITH CLEANING ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse EuroLLM response into structured fields with cleaning.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Valid profession categories\n",
+    "    VALID_PROFESSIONS = {\n",
+    "        \"actor\", \"adult performer\", \"singer/musician\", \"model\", \n",
+    "        \"online personality\", \"public figure\", \"voice actor/asmr\", \n",
+    "        \"sports professional\", \"tv personality\"\n",
+    "    }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            # Clean gender field - remove any labels\n",
+    "            gender_raw = line[2:].strip()\n",
+    "            # Remove common prefixes\n",
+    "            gender_raw = re.sub(r'^(Gender:|gender:)\\s*', '', gender_raw, flags=re.IGNORECASE)\n",
+    "            # Extract just the gender word\n",
+    "            gender_match = re.search(r'\\b(Male|Female|Other|Unknown)\\b', gender_raw, re.IGNORECASE)\n",
+    "            fields['gender'] = gender_match.group(1).capitalize() if gender_match else gender_raw\n",
+    "        elif line.startswith('4.'):\n",
+    "            # Clean and validate profession field\n",
+    "            profession_raw = line[2:].strip()\n",
+    "            \n",
+    "            # Split by comma and validate each profession\n",
+    "            professions = [p.strip().lower() for p in profession_raw.split(',')]\n",
+    "            valid_profs = []\n",
+    "            \n",
+    "            for prof in professions:\n",
+    "                # Check if it's a valid profession\n",
+    "                if prof in VALID_PROFESSIONS:\n",
+    "                    valid_profs.append(prof)\n",
+    "                # Check for common invalid entries\n",
+    "                elif prof in ['unknown', '']:\n",
+    "                    continue\n",
+    "                # Reject descriptive words that aren't professions\n",
+    "                elif prof in ['sexy', 'photorealistic', 'celebrity', 'famous', 'popular', \n",
+    "                            'beautiful', 'attractive', 'hot', 'gorgeous']:\n",
+    "                    continue\n",
+    "                # If it looks like it might be close to a valid profession, keep it\n",
+    "                elif any(valid in prof for valid in VALID_PROFESSIONS):\n",
+    "                    # Try to extract the valid part\n",
+    "                    for valid in VALID_PROFESSIONS:\n",
+    "                        if valid in prof:\n",
+    "                            valid_profs.append(valid)\n",
+    "                            break\n",
+    "            \n",
+    "            # Set the cleaned professions or Unknown if none are valid\n",
+    "            if valid_profs:\n",
+    "                fields['profession_llm'] = ', '.join(valid_profs)\n",
+    "            else:\n",
+    "                fields['profession_llm'] = 'Unknown'\n",
+    "                \n",
+    "        elif line.startswith('5.'):\n",
+    "            # Clean country field - remove any labels\n",
+    "            country_raw = line[2:].strip()\n",
+    "            # Remove common prefixes like \"Primary country:\", \"Country:\", etc.\n",
+    "            country_raw = re.sub(r'^(Primary\\s+)?(associated\\s+)?country:\\s*', '', country_raw, flags=re.IGNORECASE)\n",
+    "            fields['country'] = country_raw\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"EuroLLM Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_eurollm_local(prompt)\n",
+    "        \n",
+    "        # DEBUG: Print first few responses to see what's happening\n",
+    "        if i < 5:\n",
+    "            print(f\"\\n=== DEBUG Row {i}, Attempt {attempt+1} ===\")\n",
+    "            print(f\"Response length: {len(response) if response else 0}\")\n",
+    "            print(f\"Response: {response[:500] if response else 'None'}\")\n",
+    "            print(\"=\" * 50)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # DEBUG: Print first few parsed results\n",
+    "    if i < 5:\n",
+    "        print(f\"\\n=== PARSED Row {i} ===\")\n",
+    "        for key, value in parsed.items():\n",
+    "            print(f\"  {key}: {value}\")\n",
+    "        print(\"=\" * 50)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "472e5ac2-ec04-4bfa-8a67-116277238c15",
+   "metadata": {},
+   "source": [
+    "## Mistral 24b instruct"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a55a5e30-83f3-4f7c-a537-b1216d4e8a07",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T22:16:21.002786Z",
+     "iopub.status.busy": "2025-12-09T22:16:21.002337Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading model: mistralai/Mistral-Small-Instruct-2409\n",
+      "Cache directory: /shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/data/models\n",
+      "This may take a while on first run (~65GB download)...\n",
+      "\n",
+      "Device: cuda\n",
+      "Loading tokenizer...\n",
+      "✅ Tokenizer loaded\n",
+      "Loading model (this may take several minutes)...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "`torch_dtype` is deprecated! Use `dtype` instead!\n",
+      "Loading checkpoint shards: 100%|██████████| 9/9 [02:42<00:00, 18.06s/it]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Model loaded\n",
+      "VRAM used: 21.40 GB\n",
+      "\n",
+      "Loading raw input CSV...\n",
+      "Loaded 50861 rows from input file\n",
+      "Found existing annotations, merging...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_3104208/1997558719.py:113: DtypeWarning: Columns (52,53,54,55,56) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  existing_df = pd.read_csv(output_file)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Existing annotations has 50861 rows\n",
+      "Merged annotations, continuing with 50861 total rows\n",
+      "✅ Loaded professions.csv\n",
+      "✅ Loaded profession mapping with 9 categories\n",
+      "Loaded 50861 rows\n",
+      "\n",
+      "Profession categories (9):\n",
+      "  - actor\n",
+      "  - adult performer\n",
+      "  - singer/musician\n",
+      "  - model\n",
+      "  - online personality\n",
+      "  - public figure\n",
+      "  - voice actor/ASMR\n",
+      "  - sports professional\n",
+      "  - tv personality\n",
+      "\n",
+      "Creating prompts...\n",
+      "✅ Prompts created\n",
+      "Resuming from index 8810\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Mistral Local:   0%|          | 0/42051 [00:00<?, ?it/s]/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:181: UserWarning: MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 during quantization\n",
+      "  warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+      "Mistral Local:   0%|          | 7/42051 [00:57<93:01:03,  7.96s/it] "
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "output_file = current_dir.parent / f\"data/CSV/mistral24_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/mistral24_local_query_index.txt\"\n",
+    "\n",
+    "\n",
+    "# Model settings\n",
+    "#MODEL_NAME = \"mistralai/Mistral-Small-3.1-24B-Instruct-2503\"\n",
+    "MODEL_NAME = \"mistralai/Mistral-Small-Instruct-2409\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~65GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "quantization_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Load model with optimizations\n",
+    "print(\"Loading model (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    torch_dtype=torch.bfloat16,\n",
+    "    quantization_config=quantization_config,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "print(\"Loading raw input CSV...\")\n",
+    "df = pd.read_csv(input_file)  # ALWAYS load the full input\n",
+    "print(f\"Loaded {len(df)} rows from input file\")\n",
+    "\n",
+    "# If we have previous annotations, merge them\n",
+    "if output_file.exists():\n",
+    "    print(\"Found existing annotations, merging...\")\n",
+    "    existing_df = pd.read_csv(output_file)\n",
+    "    print(f\"Existing annotations has {len(existing_df)} rows\")\n",
+    "    \n",
+    "    # Update df with existing annotations\n",
+    "    # Only update the columns that were annotated\n",
+    "    annotation_cols = ['full_name', 'aliases', 'gender', 'profession_llm', 'country']\n",
+    "    for col in annotation_cols:\n",
+    "        if col in existing_df.columns:\n",
+    "            df[col] = existing_df[col][:len(df)]  # Make sure we don't exceed df length\n",
+    "    \n",
+    "    print(f\"Merged annotations, continuing with {len(df)} total rows\")\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS (DEEPSEEK STYLE) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Mistral annotation with specific profession categories.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "- For country respond with one word only, for example China or Columbia\n",
+    "- actress = actor\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "# === QUERY MISTRAL LOCAL ===\n",
+    "def query_mistral_local(prompt: str) -> str:\n",
+    "    \"\"\"Query Mistral locally via transformers.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for Mistral\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are an assistant that extracts key data on a person based on the name. Respond with exactly 5 numbered lines. For professions, choose ONLY from these categories: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for older tokenizers\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate\n",
+    "        with torch.no_grad():\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=512,\n",
+    "                temperature=0.05,\n",
+    "                do_sample=True,\n",
+    "                top_p=0.8,\n",
+    "                pad_token_id=tokenizer.pad_token_id if tokenizer.pad_token_id else tokenizer.eos_token_id\n",
+    "            )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PARSE RESPONSE (DEEPSEEK STYLE) ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Mistral response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/mistral24_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/mistral24_local_query_index.txt\"\n",
+    "\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"Mistral Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_mistral_local(prompt)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d7212e75-0ff6-45a0-8695-c4a3d3e02818",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import transformers\n",
+    "print(f\"Transformers version: {transformers.__version__}\")\n",
+    "\n",
+    "# Check if Mistral3 is available\n",
+    "try:\n",
+    "    from transformers import Mistral3ForCausalLM\n",
+    "    print(\"✅ Mistral3 is available\")\n",
+    "except ImportError:\n",
+    "    print(\"❌ Mistral3 not available in this transformers version\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a6ab032e-246e-4c4e-9776-ff0bfbf6fd9c",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4__Figure_8_Deepfake_victims-checkpoint.ipynb

@@ -0,0 +1,668 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "06763fde",
+   "metadata": {},
+   "source": [
+    "# LLM annotation of Deepfake adapters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b773045a",
+   "metadata": {},
+   "source": [
+    "## Step 01 Data cleaning NER using spaCy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "234a55e5",
+   "metadata": {},
+   "source": [
+    "#### Here we clean leetspeak and architectre specifiers from the names as a preprossesing step for the named entity recognition NER below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f177df11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import spacy\n",
+    "import re\n",
+    "import torch\n",
+    "from pathlib import Path\n",
+    "import unicodedata\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "a2383f32",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "poi_models_dir = current_dir.parent / \"data/CSV/model_subsets/POI_models.csv\"  ### POI models dataset\n",
+    "output = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"  ### Output file"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "66fc691f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauhp/anaconda3/envs/latm/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "nlp = spacy.load(\"en_core_web_sm\")  # or another model of your choice"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "8348c4a4",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Done! Saved to NER_poi_step_01.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "def preprocess_name(name):\n",
+    "    name = str(name)\n",
+    "\n",
+    "    # Normalize unicode characters (e.g., fancy fonts)\n",
+    "    name = unicodedata.normalize(\"NFKD\", name)\n",
+    "\n",
+    "    # Lowercase everything\n",
+    "    name = name.lower()\n",
+    "\n",
+    "    # Remove special keywords and patterns\n",
+    "    junk_words = [\n",
+    "        'jav', 'jp', 'lora', 'locon', 'lycoris', 'requested', 'japanese', 'model',\n",
+    "        'flux', 'flux1.d', 'pony', 'realistic'\n",
+    "    ]\n",
+    "    for word in junk_words:\n",
+    "        name = re.sub(rf'\\b{re.escape(word)}\\b', '', name, flags=re.IGNORECASE)\n",
+    "\n",
+    "    # Remove versions like v1, v2.0, etc.\n",
+    "    name = re.sub(r'v\\.?\\d+(\\.\\d+)?', '', name)\n",
+    "\n",
+    "    # Remove 'not' followed by a word\n",
+    "    name = re.sub(r'\\bnot\\s+\\w+', '', name)\n",
+    "\n",
+    "    # Replace underscores and pipes with spaces\n",
+    "    name = re.sub(r'[_|]', ' ', name)\n",
+    "\n",
+    "    # Remove parentheses and content within\n",
+    "    name = re.sub(r'\\(.*?\\)', '', name)\n",
+    "\n",
+    "    # Remove excess whitespace\n",
+    "    name = re.sub(r'\\s+', ' ', name).strip()\n",
+    "\n",
+    "    return name\n",
+    "\n",
+    "# -------------------------\n",
+    "# Fallback Extractor\n",
+    "# -------------------------\n",
+    "def fallback_extract(text):\n",
+    "    words = text.split()\n",
+    "    capitalized = [w for w in words if w and w[0].isalpha()]\n",
+    "    if len(capitalized) >= 2:\n",
+    "        return \" \".join(capitalized[:2])\n",
+    "    elif capitalized:\n",
+    "        return capitalized[0]\n",
+    "    return None\n",
+    "\n",
+    "# -------------------------\n",
+    "# Full Extraction Logic\n",
+    "# -------------------------\n",
+    "def extract_real_name(raw_name):\n",
+    "    cleaned = preprocess_name(raw_name)\n",
+    "    doc = nlp(cleaned)\n",
+    "    persons = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    if persons:\n",
+    "        return persons[0]\n",
+    "    return fallback_extract(cleaned)\n",
+    "\n",
+    "# -------------------------\n",
+    "# Load Data and Apply\n",
+    "# -------------------------\n",
+    "df = pd.read_csv(poi_models_dir)  # Or your own path\n",
+    "\n",
+    "# Apply the full extractor\n",
+    "texts = df['name'].astype(str).tolist()\n",
+    "docs = list(nlp.pipe([preprocess_name(t) for t in texts], batch_size=32))\n",
+    "\n",
+    "def extract_from_doc(doc, raw_text):\n",
+    "    persons = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    if persons:\n",
+    "        return persons[0]\n",
+    "    return fallback_extract(preprocess_name(raw_text))\n",
+    "\n",
+    "df['real_name'] = [extract_from_doc(doc, raw_text) for doc, raw_text in zip(docs, texts)]\n",
+    "\n",
+    "# Save the result\n",
+    "df.to_csv(output, index=False)\n",
+    "print(\"Done! Saved to NER_poi_step_01.csv\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a40b17fe",
+   "metadata": {},
+   "source": [
+    "## Step 2: compare country and profession with lists"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "414954ed",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "          real_name likely_country likely_nationality likely_profession\n",
+      "0     ronnie alonte                                               model\n",
+      "1  zh elena kamperi                                     twitch streamer\n",
+      "2     sofia vergara                                               model\n",
+      "3          安然 anran                                           celebrity\n",
+      "4       zoe kravitz                                           celebrity\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "inputNER = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_01.csv\"\n",
+    "outfile = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(inputNER)\n",
+    "countries_df = pd.read_csv(countries)\n",
+    "professions_df = pd.read_csv(professions)\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag → (country, nationality) mapping\n",
+    "tag_to_country_nationality = {}\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Add variations to the mapping\n",
+    "    tag_to_country_nationality[country_lc] = (country, \"\")\n",
+    "    tag_to_country_nationality[nationality_lc] = (\"\", nationality)\n",
+    "    tag_to_country_nationality[country_lc.replace(\" \", \"\")] = (country, \"\")\n",
+    "    tag_to_country_nationality[nationality_lc.replace(\" \", \"\")] = (\"\", nationality)\n",
+    "\n",
+    "    for part in country_lc.split():\n",
+    "        tag_to_country_nationality[part] = (country, \"\")\n",
+    "    for part in nationality_lc.split():\n",
+    "        tag_to_country_nationality[part] = (\"\", nationality)\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality\n",
+    "# Step 3: Infer likely_country and likely_nationality\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality = tag_to_country_nationality[cleaned]\n",
+    "            # Special case: skip if country is \"Isle of Man\"\n",
+    "            if country == \"Isle of Man\":\n",
+    "                country = \"\"\n",
+    "            return pd.Series([country, nationality])\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag → profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(outfile, index=False)\n",
+    "\n",
+    "# Optional: Preview\n",
+    "print(poi_df[[\"real_name\", \"likely_country\", \"likely_nationality\", \"likely_profession\"]].head())\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "054f230b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install transformers torch\n",
+    "#!python -m spacy download en_core_web_trf\n",
+    "#pip install openai"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "59185461",
+   "metadata": {},
+   "source": [
+    "## Step 3: Query Deepseek-v3 with NAME and HINTS"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "504b970f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#!pip install openpyxl"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "7c209115",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Row 1/4...\n",
+      "Saved up to row 1\n",
+      "Row 2/4...\n",
+      "Saved up to row 2\n",
+      "Row 3/4...\n",
+      "Saved up to row 3\n",
+      "Row 4/4...\n",
+      "Saved up to row 4\n",
+      "All done! Files: /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.csv /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/data/CSV/Deepseek_annotated_POI.xlsx\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import openai\n",
+    "import time\n",
+    "import os\n",
+    "from pathlib import Path\n",
+    "from openai import OpenAI  # Add this import\n",
+    "\n",
+    "# === PATHS & CONFIG ===\n",
+    "current_dir = Path.cwd()\n",
+    "inputCSV = current_dir.parent / \"data/CSV/model_subsets/NER_poi_step_02.csv\"\n",
+    "api_key_file = current_dir.parent / \"misc/credentials/deepseek_api_key.txt\" #store your API key under misc/credentials/deepseek_api_key.txt\n",
+    "\n",
+    "# Output both CSV and Excel for compatibility\n",
+    "OUTPUT_CSV = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.csv\"\n",
+    "OUTPUT_XLSX = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.xlsx\"\n",
+    "INDEX_FILE = current_dir.parent / \"misc/deepseek_query_index.txt\"\n",
+    "SAVE_INTERVAL = 1  # Save every N rows\n",
+    "START_ROW = 1      # Row index to start from (0-based)\n",
+    "END_ROW = 5        # Row index to end (exclusive)\n",
+    "\n",
+    "# === LOAD API KEY & CLIENT ===\n",
+    "with open(api_key_file) as f:\n",
+    "    api_key = f.read().strip()\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=api_key,\n",
+    "    base_url=\"https://api.deepseek.com/v1\"\n",
+    ")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(inputCSV)\n",
+    "df = df.iloc[START_ROW:END_ROW].reset_index(drop=True)\n",
+    "\n",
+    "# === PREPARE PROMPTS ===\n",
+    "def create_prompt(row):\n",
+    "    name = row['real_name'] if pd.notna(row['real_name']) else row['name']\n",
+    "    hints = []\n",
+    "    for col in ('likely_profession','likely_nationality','likely_country'):\n",
+    "        if pd.notna(row.get(col, None)):\n",
+    "            hints.append(row[col])\n",
+    "    if not hints:\n",
+    "        tags = [row[f'tag_{i}'] for i in range(1,8) if pd.notna(row.get(f'tag_{i}', None))]\n",
+    "        hints.extend(tags[:3])\n",
+    "    hint_text = \", \".join(hints[:5])\n",
+    "    return (\n",
+    "        f\"Given '{name}' ({hint_text}), provide:\\n\"\n",
+    "        \"1. Full legal name (Western order if non-latin script)\\n\"\n",
+    "        \"2. Any stage names/aliases (comma separated)\\n\"\n",
+    "        \"3. Gender\\n\"\n",
+    "        \"4. Top 3 most specific, factual professions (use industry-standard terms, no euphemisms)\\n\"\n",
+    "        \"5. Primary country associated\\n\"\n",
+    "        \"Use 'Unknown' when uncertain or you encounter a fictional character or place. \"\n",
+    "        \"For entertainment fields, specify sub-genres when known (kpop, adult industry, etc.).\"\n",
+    "    )\n",
+    "\n",
+    "prompts = df.apply(create_prompt, axis=1).tolist()\n",
+    "\n",
+    "# === CHECK FOR EXISTING OUTPUT ===\n",
+    "if os.path.exists(INDEX_FILE):\n",
+    "    with open(INDEX_FILE, 'r') as f:\n",
+    "        current_index = int(f.read().strip())\n",
+    "else:\n",
+    "    current_index = 0\n",
+    "\n",
+    "results = []\n",
+    "if os.path.exists(OUTPUT_XLSX):\n",
+    "    existing = pd.read_excel(OUTPUT_XLSX)\n",
+    "    if {'full_name','aliases','gender','profession_llm','country'}.issubset(existing.columns):\n",
+    "        results = existing[['full_name','aliases','gender','profession_llm','country']].values.tolist()\n",
+    "\n",
+    "# === QUERY & PARSE ===\n",
+    "def query_deepseek(prompt):\n",
+    "    try:\n",
+    "        resp = client.chat.completions.create(\n",
+    "            model=\"deepseek-chat\",\n",
+    "            messages=[\n",
+    "                {\"role\":\"system\",\"content\":\"You extract key data on a person; respond with exactly 5 numbered lines.\"},\n",
+    "                {\"role\":\"user\",\"content\":prompt}\n",
+    "            ],\n",
+    "            temperature=0.05, top_p=0.8\n",
+    "        )\n",
+    "        return resp.choices[0].message.content.strip()\n",
+    "    except Exception as e:\n",
+    "        print(\"API error:\", e)\n",
+    "        return \"\"\n",
+    "\n",
+    "def parse_response(resp):\n",
+    "    lines = [l.strip() for l in resp.split('\\n') if l.strip()]\n",
+    "    out = [\"Unknown\"]*5\n",
+    "    for l in lines:\n",
+    "        if l.startswith('1.'): out[0] = l[2:].strip()\n",
+    "        elif l.startswith('2.'): out[1] = l[2:].strip()\n",
+    "        elif l.startswith('3.'): out[2] = l[2:].strip()\n",
+    "        elif l.startswith('4.'): out[3] = l[2:].strip()\n",
+    "        elif l.startswith('5.'): out[4] = l[2:].strip()\n",
+    "    return out\n",
+    "\n",
+    "# === PROCESS & SAVE ===\n",
+    "for i in range(current_index, len(df)):\n",
+    "    print(f\"Row {i+1}/{len(df)}...\")\n",
+    "    data = parse_response(query_deepseek(prompts[i]))\n",
+    "    if i < len(results): results[i] = data\n",
+    "    else: results.append(data)\n",
+    "    current_index = i+1\n",
+    "\n",
+    "    if current_index % SAVE_INTERVAL == 0 or current_index == len(df):\n",
+    "        out_df = df.iloc[:current_index].copy()\n",
+    "        out_df[['full_name','aliases','gender','profession_llm','country']] = pd.DataFrame(results[:current_index])\n",
+    "        # CSV\n",
+    "        out_df.to_csv(OUTPUT_CSV, index=False)\n",
+    "        # Excel\n",
+    "        out_df.to_excel(OUTPUT_XLSX, index=False)\n",
+    "        with open(INDEX_FILE, 'w') as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(\"Saved up to row\", current_index)\n",
+    "    time.sleep(1)\n",
+    "\n",
+    "print(\"All done! Files:\", OUTPUT_CSV, OUTPUT_XLSX)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7910e574",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9d377005",
+   "metadata": {},
+   "source": [
+    "# Aggregate by individual names"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d2e75354",
+   "metadata": {},
+   "source": [
+    " ##### e.g. Emma Watson [model1, model2, model3] etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "747c3a2f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "profession_map = current_dir.parent / \"misc/lists/mapped_professions.csv\"\n",
+    "\n",
+    "poi_df = current_dir.parent / \"data/CSV/Deepseek_annotated_POI.csv\"\n",
+    "\n",
+    "output = current_dir.parent / \"data/CSV/Deepseek_annotated_POI_aggregated.csv\"\n",
+    "\n",
+    "countries_csv = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "countries_df = pd.read_csv(countries_csv)\n",
+    "\n",
+    "# Extract valid country names (strip whitespace)\n",
+    "valid_countries = set(countries_df['en_short_name'].str.strip())\n",
+    "\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv(poi_df)  # Update path if needed\n",
+    "\n",
+    "# Step 1: Group by 'full_name' and aggregate required information\n",
+    "grouped_df = df.groupby('full_name').agg(\n",
+    "    No_of_models=('id', 'count'),\n",
+    "    modelIDs=('id', lambda x: list(x)),\n",
+    "    combinedDownloadCount=('downloadCount', 'sum')\n",
+    ").reset_index()\n",
+    "\n",
+    "# Step 2: Keep representative info for each person\n",
+    "# Keep representative info (including aliases)\n",
+    "additional_columns = df.groupby('full_name').agg(\n",
+    "    country=('country', 'first'),\n",
+    "    profession_llm=('profession_llm', 'first'),\n",
+    "    gender=('gender', 'first'),\n",
+    "    aliases=('aliases', 'first')  # ✅ Add this line\n",
+    ").reset_index()\n",
+    "\n",
+    "\n",
+    "\n",
+    "def standardize_country(country):\n",
+    "    if not isinstance(country, str):\n",
+    "        return \"Unknown\"\n",
+    "\n",
+    "    country_clean = country.strip()\n",
+    "    lowered = country_clean.lower()\n",
+    "\n",
+    "    # Handle fictional or fantasy countries\n",
+    "    fictional_keywords = [\"fictional\", \"westeros\", \"asgard\", \"middle-earth\", \"naboo\", \"middle earth\", \"latveria\"]\n",
+    "    if any(keyword in lowered for keyword in fictional_keywords):\n",
+    "        return \"Unknown\"\n",
+    "\n",
+    "    # Handle known region-based adjustments\n",
+    "    if \"macau\" in lowered:\n",
+    "        return \"Macau\"\n",
+    "    elif \"hong kong\" in lowered:\n",
+    "        return \"Hong Kong\"\n",
+    "    elif \"taiwan\" in lowered:\n",
+    "        return \"Taiwan\"\n",
+    "\n",
+    "    # Normalize complex or alternate country names\n",
+    "    lowered = lowered.replace(\"United Kingdom of Great Britain and Northern Ireland\", \"united kingdom\")\n",
+    "    lowered = lowered.replace(\"england\", \"united kingdom\")\n",
+    "    lowered = lowered.replace(\"united states of america\", \"united states\")\n",
+    "\n",
+    "    # Remove anything in brackets and after commas\n",
+    "    country_clean = re.sub(r\"\\(.*?\\)\", \"\", country_clean)\n",
+    "    country_clean = country_clean.split(',')[0].strip().lower()\n",
+    "\n",
+    "    # Manual overrides\n",
+    "    replacements = {\n",
+    "        \"united kingdom\": \"UK\",\n",
+    "        \"united kingdom of great britain and northern ireland\": \"UK\",\n",
+    "        \"french southern territories\": \"Other\",\n",
+    "        \"united states\": \"US\",\n",
+    "        \"united states of america\": \"US\",\n",
+    "        \"turkey\": \"Türkiye\",\n",
+    "        \"czech republic\": \"Czechia\"\n",
+    "    }\n",
+    "\n",
+    "    if country_clean in replacements:\n",
+    "        return replacements[country_clean]\n",
+    "\n",
+    "    # Final check against valid country list (case-insensitive)\n",
+    "    for valid in valid_countries:\n",
+    "        if country_clean == valid.lower():\n",
+    "            return valid\n",
+    "\n",
+    "    return \"Unknown\"\n",
+    "\n",
+    "\n",
+    "# Updated function to fully remove anything in brackets (complete or not)\n",
+    "def get_profession_short(profession):\n",
+    "    if isinstance(profession, str):\n",
+    "        # Get first part before comma\n",
+    "        first_prof = profession.split(',')[0].strip()\n",
+    "        # Remove all bracketed content, even malformed\n",
+    "        first_prof = re.sub(r\"[\\[].∗?[\\[].*?[\\]]\", \"\", first_prof)  # removes properly closed\n",
+    "        first_prof = re.sub(r\"[\\(\\[].*\", \"\", first_prof)         # removes malformed\n",
+    "        cleaned = first_prof.strip()\n",
+    "        # Normalize 'Actress' to 'Actor'\n",
+    "        if cleaned.lower() == \"actress\":\n",
+    "            return \"Actor\"\n",
+    "        return cleaned\n",
+    "    return None\n",
+    "\n",
+    "# Load your mapping file\n",
+    "mapping_df = pd.read_csv(profession_map, on_bad_lines='skip')  # or 'warn'\n",
+    "\n",
+    "\n",
+    "# Ensure the mapping columns are named correctly\n",
+    "# (Assuming columns are: 'profession_llm' or 'profession_short', and 'category' or 'mapped_profession')\n",
+    "# Adjust these as needed\n",
+    "mapping_df.columns = [col.strip().lower() for col in mapping_df.columns]\n",
+    "\n",
+    "# Rename for clarity and consistency\n",
+    "if 'profession_llm' in mapping_df.columns:\n",
+    "    mapping_df = mapping_df.rename(columns={'profession_llm': 'profession_short'})\n",
+    "if 'category' in mapping_df.columns:\n",
+    "    mapping_df = mapping_df.rename(columns={'category': 'mapped_profession'})\n",
+    "\n",
+    "# Merge the mapped profession into final_df\n",
+    "#final_df = final_df.merge(mapping_df[['profession_short', 'mapped_profession']], on='profession_short', how='left')\n",
+    "\n",
+    "\n",
+    "additional_columns = df.groupby('full_name').agg(\n",
+    "    country=('country', 'first'),\n",
+    "    profession_llm=('profession_llm', 'first'),\n",
+    "    gender=('gender', 'first'),\n",
+    "    aliases=('aliases', 'first')  # <-- Added this line\n",
+    ").reset_index()\n",
+    "\n",
+    "\n",
+    "# Step 3: Merge the aggregated info with the representative info\n",
+    "final_df = pd.merge(grouped_df, additional_columns, on='full_name', how='left')\n",
+    "\n",
+    "# Step 4: Clean and transform columns\n",
+    "final_df['profession_short'] = final_df['profession_llm'].apply(get_profession_short)\n",
+    "final_df['country'] = final_df['country'].apply(standardize_country)\n",
+    "\n",
+    "# Step 5: Merge with profession mapping\n",
+    "final_df = final_df.merge(mapping_df[['profession_short', 'mapped_profession']], on='profession_short', how='left')\n",
+    "\n",
+    "# Optional: Save the result to a CSV file\n",
+    "final_df.to_csv(output, index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "704e5246",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4__Figure_8b_sunburst_profession-checkpoint.ipynb

@@ -0,0 +1,123 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prepare *.json for Figure 8b"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Sunburst data saved to sunburst_data.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from collections import defaultdict\n",
+    "import json\n",
+    "from pathlib import Path\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "sunburst_path = current_dir.parent / \"public/json/sunburst_countries_A.json\"\n",
+    "\n",
+    "\n",
+    "aggregated_poi = current_dir.parent / \"data/CSV/Deepseek_annotated_POI_aggregated.csv\"\n",
+    "\n",
+    "df = pd.read_csv(aggregated_poi)\n",
+    "df['country_cleaned'] = df['country'].apply(lambda x: x if x not in ['Unknown', '', None] else 'Other')\n",
+    "\n",
+    "# Now get top countries excluding what was forced into 'Other'\n",
+    "top_countries = df['country_cleaned'].value_counts().nlargest(15).index.tolist()\n",
+    "\n",
+    "# Final limited country column\n",
+    "df['country_limited'] = df['country_cleaned'].apply(lambda x: x if x in top_countries else 'Other')\n",
+    "\n",
+    "# ---- Step 2: Limit to top 7 professions and combine Unknown and Sports Professional with Other ----\n",
+    "top_professions = df['mapped_profession'].value_counts().nlargest(7).index.tolist()\n",
+    "\n",
+    "# Explicitly remove 'Unknown' and 'Sports Professional' even if they are in the top 7\n",
+    "top_professions = [p for p in top_professions if p not in ['Unknown', 'Sports Professional']]\n",
+    "\n",
+    "# Normalize 'Unknown' and '\n",
+    "# Sports Professional' into 'Other'\n",
+    "# Normalize 'Unknown' and 'Sports Professional' into 'Other'\n",
+    "df['profession_limited'] = df['mapped_profession'].apply(\n",
+    "    lambda x: 'Other' if x in ['Unknown', 'Sports Professional'] or x not in top_professions else x\n",
+    ")\n",
+    "\n",
+    "\n",
+    "\n",
+    "# ---- Step 3: Group by the limited country and profession ----\n",
+    "sunburst_data = df.groupby(['country_limited', 'profession_limited']).size().reset_index(name='count')\n",
+    "\n",
+    "# ---- Step 4: Create a nested structure for D3.js ----\n",
+    "sunburst_dict = {\"name\": \"root\", \"children\": []}\n",
+    "country_map = defaultdict(list)\n",
+    "\n",
+    "for _, row in sunburst_data.iterrows():\n",
+    "    country = row['country_limited']\n",
+    "    profession = row['profession_limited']\n",
+    "    count = int(row['count'])\n",
+    "    country_map[country].append({\"name\": profession, \"value\": count})\n",
+    "\n",
+    "# For each country, sort the profession list so that \"Other\" appears at the end\n",
+    "for country, professions in country_map.items():\n",
+    "    professions_sorted = sorted(professions, key=lambda d: (d[\"name\"] == \"Other\", d[\"name\"]))\n",
+    "    country_map[country] = professions_sorted\n",
+    "\n",
+    "for country, professions in country_map.items():\n",
+    "    sunburst_dict[\"children\"].append({\"name\": country, \"children\": professions})\n",
+    "\n",
+    "# ---- Step 5: Save to a JSON file ----\n",
+    "with open(sunburst_path, \"w\", encoding='utf-8') as f:\n",
+    "    json.dump(sunburst_dict, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "\n",
+    "print(\"✅ Sunburst data saved to sunburst_data.json\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the resulting *.json is the input for Figure_8.html"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3-4_compare-models-checkpoint.ipynb

@@ -0,0 +1,237 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6cbaef9d-3058-4a59-a8ee-32fcc2062ed6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Jupyter Notebook Cell for Country Name Standardization\n",
+    "# Copy and paste this entire cell into your Jupyter notebook\n",
+    "\n",
+    "import pandas as pd\n",
+    "import os\n",
+    "import re\n",
+    "\n",
+    "# Define country standardization mappings\n",
+    "COUNTRY_MAPPINGS = {\n",
+    "    # USA variations\n",
+    "    'united states': 'USA',\n",
+    "    'united states of america': 'USA',\n",
+    "    'america': 'USA',\n",
+    "    'us': 'USA',\n",
+    "    'u.s.': 'USA',\n",
+    "    'u.s.a.': 'USA',\n",
+    "    'states': 'USA',\n",
+    "    \n",
+    "    # UK variations\n",
+    "    'united kingdom': 'UK',\n",
+    "    'england': 'UK',\n",
+    "    'britain': 'UK',\n",
+    "    'great britain': 'UK',\n",
+    "    'uk': 'UK',\n",
+    "    'u.k.': 'UK',\n",
+    "    \n",
+    "    # Turkey -> Türkiye\n",
+    "    'turkey': 'Türkiye',\n",
+    "    \n",
+    "    # Czech Republic -> Czechia\n",
+    "    'czech republic': 'Czechia',\n",
+    "    'czechoslovakia': 'Czechia',\n",
+    "    'czechoslowakia': 'Czechia',  # Common misspelling\n",
+    "    \n",
+    "    # USSR -> Russia\n",
+    "    'ussr': 'Russia',\n",
+    "    'udssr': 'Russia',\n",
+    "    'soviet union': 'Russia',\n",
+    "    \n",
+    "    # Korea variations (maintain distinction between North and South)\n",
+    "    'korea': 'South Korea',  # If just \"Korea\", assume South Korea\n",
+    "    'south korea': 'South Korea',\n",
+    "    'republic of korea': 'South Korea',\n",
+    "    'north korea': 'North Korea',\n",
+    "    'dprk': 'North Korea',\n",
+    "    \n",
+    "    # China variations\n",
+    "    'china': 'China',\n",
+    "    'people\\'s republic of china': 'China',\n",
+    "    'prc': 'China',\n",
+    "    'mainland china': 'China',\n",
+    "    \n",
+    "    # Common standardizations\n",
+    "    'holland': 'Netherlands',\n",
+    "    'the netherlands': 'Netherlands',\n",
+    "    'deutschland': 'Germany',\n",
+    "    'nippon': 'Japan',\n",
+    "    'espana': 'Spain',\n",
+    "    'españa': 'Spain',\n",
+    "    \n",
+    "    # Keep these as-is but included for completeness\n",
+    "    'usa': 'USA',\n",
+    "    'russia': 'Russia',\n",
+    "}\n",
+    "\n",
+    "def standardize_country(country_value):\n",
+    "    \"\"\"\n",
+    "    Standardize a single country name based on the mapping.\n",
+    "    \"\"\"\n",
+    "    if pd.isna(country_value):\n",
+    "        return country_value\n",
+    "    \n",
+    "    # Convert to string and strip whitespace\n",
+    "    country_str = str(country_value).strip()\n",
+    "    \n",
+    "    # Return if empty or 'Unknown'\n",
+    "    if not country_str or country_str.lower() == 'unknown':\n",
+    "        return country_str\n",
+    "    \n",
+    "    # Convert to lowercase for matching\n",
+    "    country_lower = country_str.lower()\n",
+    "    \n",
+    "    # Check if it matches any of our mappings\n",
+    "    for pattern, replacement in COUNTRY_MAPPINGS.items():\n",
+    "        if country_lower == pattern:\n",
+    "            return replacement\n",
+    "    \n",
+    "    # If no exact match found, return original with proper capitalization\n",
+    "    # This preserves countries not in our mapping\n",
+    "    return country_str\n",
+    "\n",
+    "def process_csv_file(input_file, output_file):\n",
+    "    \"\"\"\n",
+    "    Process a CSV file to standardize country names.\n",
+    "    \"\"\"\n",
+    "    print(f\"Processing: {input_file}\")\n",
+    "    \n",
+    "    # For mistral.csv which has no header, we need special handling\n",
+    "    if 'mistral' in input_file.lower():\n",
+    "        # Read without header\n",
+    "        df = pd.read_csv(input_file, header=None)\n",
+    "        \n",
+    "        # Check if the last column contains country data\n",
+    "        # Based on the structure, country should be in the last column\n",
+    "        last_col = df.columns[-1]\n",
+    "        \n",
+    "        # Apply standardization to the last column\n",
+    "        df[last_col] = df[last_col].apply(standardize_country)\n",
+    "        \n",
+    "        print(f\"  - Standardized column {last_col} (assumed to be country)\")\n",
+    "        print(f\"  - Sample values after standardization: {df[last_col].dropna().head(10).tolist()}\")\n",
+    "    else:\n",
+    "        # Normal CSV with header\n",
+    "        df = pd.read_csv(input_file)\n",
+    "        \n",
+    "        # Check if 'country' column exists\n",
+    "        if 'country' in df.columns:\n",
+    "            # Apply standardization\n",
+    "            df['country'] = df['country'].apply(standardize_country)\n",
+    "            \n",
+    "            print(f\"  - Found and standardized 'country' column\")\n",
+    "            print(f\"  - Unique countries after standardization: {sorted(df['country'].dropna().unique())}\")\n",
+    "        else:\n",
+    "            print(f\"  - Warning: No 'country' column found in {input_file}\")\n",
+    "    \n",
+    "    # Save the processed file\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    print(f\"  - Saved to: {output_file}\\n\")\n",
+    "    \n",
+    "    return df\n",
+    "\n",
+    "# ============================================================\n",
+    "# MAIN EXECUTION - Adjust paths as needed for your environment\n",
+    "# ============================================================\n",
+    "\n",
+    "# Input files - adjust these paths to match your file locations\n",
+    "input_files = [\n",
+    "    'gemma.csv',\n",
+    "    'mistral.csv',\n",
+    "    'qwen.csv'\n",
+    "]\n",
+    "\n",
+    "# Create a results dictionary to store processed dataframes\n",
+    "results = {}\n",
+    "\n",
+    "for filename in input_files:\n",
+    "    # Adjust these paths based on where your files are located\n",
+    "    # For example, if files are in current directory, use: input_path = filename\n",
+    "    input_path = filename  # Modify this based on your file location\n",
+    "    \n",
+    "    # Create output filename\n",
+    "    base_name = filename.replace('.csv', '')\n",
+    "    output_path = f'{base_name}_standardized_country.csv'\n",
+    "    \n",
+    "    # Check if input file exists\n",
+    "    if os.path.exists(input_path):\n",
+    "        df = process_csv_file(input_path, output_path)\n",
+    "        results[base_name] = df\n",
+    "    else:\n",
+    "        print(f\"Warning: {input_path} not found! Please check the file path.\\n\")\n",
+    "\n",
+    "# Display summary statistics\n",
+    "print(\"=\" * 60)\n",
+    "print(\"SUMMARY OF COUNTRY STANDARDIZATION\")\n",
+    "print(\"=\" * 60)\n",
+    "\n",
+    "for name, df in results.items():\n",
+    "    print(f\"\\n{name.upper()}:\")\n",
+    "    print(f\"  Total rows: {len(df)}\")\n",
+    "    \n",
+    "    # Find the country column\n",
+    "    if name == 'mistral':\n",
+    "        # For mistral, assume last column is country\n",
+    "        country_col = df.columns[-1]\n",
+    "    else:\n",
+    "        country_col = 'country' if 'country' in df.columns else None\n",
+    "    \n",
+    "    if country_col is not None:\n",
+    "        country_counts = df[country_col].value_counts()\n",
+    "        print(f\"  Top 10 countries:\")\n",
+    "        for country, count in country_counts.head(10).items():\n",
+    "            print(f\"    - {country}: {count}\")\n",
+    "\n",
+    "print(\"\\n\" + \"=\" * 60)\n",
+    "print(\"Processing complete!\")\n",
+    "print(\"Standardized files have been created with '_standardized_country.csv' suffix\")\n",
+    "print(\"=\" * 60)\n",
+    "\n",
+    "# Optional: Display before/after comparison for a sample\n",
+    "print(\"\\n\" + \"=\" * 60)\n",
+    "print(\"EXAMPLE TRANSFORMATIONS APPLIED:\")\n",
+    "print(\"=\" * 60)\n",
+    "print(\"• 'United States' → 'USA'\")\n",
+    "print(\"• 'United States of America' → 'USA'\")\n",
+    "print(\"• 'States' → 'USA'\")\n",
+    "print(\"• 'England' → 'UK'\")\n",
+    "print(\"• 'United Kingdom' → 'UK'\")\n",
+    "print(\"• 'Britain' → 'UK'\")\n",
+    "print(\"• 'Turkey' → 'Türkiye'\")\n",
+    "print(\"• 'Czech Republic' → 'Czechia'\")\n",
+    "print(\"• 'Korea' → 'South Korea'\")\n",
+    "print(\"• 'UDSSR' → 'Russia'\")\n",
+    "print(\"=\" * 60)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-3_Figure_5_co-occurence_promotional_tags-checkpoint.ipynb

@@ -0,0 +1,314 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create *.json for figure 5 (Co-occurence network of Tags)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: america\n",
+      "  ✅ Saved to /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/public/json/tags_america.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import re\n",
+    "import os\n",
+    "\n",
+    "from pathlib import Path\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = current_dir.parent / \"data/CSV/Models/Civi_models.csv\"\n",
+    "output_dir = current_dir.parent / \"public/json/\"\n",
+    "#target_terms = [\"asian\", \"indian\", \"man\", \"woman\", \"german\", \"korean\", \"american\", \"russian\", \"style\", \"japanese\", \"chinese\"]  # Add any tags you want to process\n",
+    "#target_terms = [\"character\", \"instagram\", \"youtuber\", \"actor\", \"actress\", \"celebrity\", \"vtuber\", \"kpop\"]  # Add any tags you want to process\n",
+    "target_terms = [\"america\"]  # Add any tags you want to process\n",
+    "min_connections = 1   # minimum number of link connections per node\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === MAIN LOOP ===\n",
+    "for target_term in target_terms:\n",
+    "    print(f\"Processing: {target_term}\")\n",
+    "    \n",
+    "    pattern = re.compile(rf'\\b{re.escape(target_term)}\\b', flags=re.IGNORECASE)\n",
+    "    df_filtered = df_tags[df_tags.apply(\n",
+    "        lambda row: row.astype(str).apply(lambda x: bool(pattern.search(x))).any(),\n",
+    "        axis=1\n",
+    "    )]\n",
+    "\n",
+    "    # Skip if no data matches\n",
+    "    if df_filtered.empty:\n",
+    "        print(f\"  ⚠️ No matches for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === COUNT INDIVIDUAL TAGS ===\n",
+    "    all_tags = df_filtered.values.flatten()\n",
+    "    all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "    tag_counts = Counter(all_tags)\n",
+    "\n",
+    "    # === CO-OCCURRENCE ===\n",
+    "    co_occurrences = defaultdict(int)\n",
+    "    for tags in df_filtered.itertuples(index=False, name=None):\n",
+    "        tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "        for tag1, tag2 in combinations(tags, 2):\n",
+    "            co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "    edges = [(list(pair)[0], list(pair)[1], weight) for pair, weight in co_occurrences.items()]\n",
+    "\n",
+    "    # === FILTER BY CONNECTIONS ===\n",
+    "    connected_tags = Counter()\n",
+    "    for tag1, tag2, _ in edges:\n",
+    "        connected_tags[tag1] += 1\n",
+    "        connected_tags[tag2] += 1\n",
+    "\n",
+    "    nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts if connected_tags[tag] >= min_connections]\n",
+    "    valid_ids = set(node[\"id\"] for node in nodes)\n",
+    "    links = [{\"source\": tag1, \"target\": tag2, \"value\": weight}\n",
+    "             for tag1, tag2, weight in edges\n",
+    "             if tag1 in valid_ids and tag2 in valid_ids]\n",
+    "\n",
+    "    if not nodes or not links:\n",
+    "        print(f\"  ⚠️ Not enough connections for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === EXPORT ===\n",
+    "    d3_data = {\"nodes\": nodes, \"links\": links}\n",
+    "    safe_term = re.sub(r'\\W+', '_', target_term.lower())\n",
+    "    output_file = os.path.join(output_dir, f\"tags_{safe_term}.json\")\n",
+    "    \n",
+    "    with open(output_file, \"w\") as f:\n",
+    "        json.dump(d3_data, f, indent=4)\n",
+    "    \n",
+    "    print(f\"  ✅ Saved to {output_file}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Different Countries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_68582/2797381217.py:15: DtypeWarning: Columns (50,51,54,55,56,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,458) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  df = pd.read_csv(file_path)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: united states\n",
+      "  ⚠️ No matches for 'united states', skipping.\n",
+      "Processing: korea\n",
+      "  ✅ Saved to data/json/tags_korea_poi.json\n",
+      "Processing: uk\n",
+      "  ✅ Saved to data/json/tags_uk_poi.json\n",
+      "Processing: russia\n",
+      "  ✅ Saved to data/json/tags_russia_poi.json\n",
+      "Processing: china\n",
+      "  ✅ Saved to data/json/tags_china_poi.json\n",
+      "Processing: canada\n",
+      "  ✅ Saved to data/json/tags_canada_poi.json\n",
+      "Processing: India\n",
+      "  ✅ Saved to data/json/tags_india_poi.json\n",
+      "Processing: germany\n",
+      "  ✅ Saved to data/json/tags_germany_poi.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import re\n",
+    "import os\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = \"data/model_subsets/all_models_poi_true.csv\"\n",
+    "output_dir = \"data/json/\"\n",
+    "target_terms = [\"united states\", \"korea\", \"uk\", \"russia\", \"china\", \"canada\", \"India\", \"germany\"] # Add any tags you want to process\n",
+    "min_connections = 1  # minimum number of link connections per node\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === MAIN LOOP ===\n",
+    "for target_term in target_terms:\n",
+    "    print(f\"Processing: {target_term}\")\n",
+    "    \n",
+    "    pattern = re.compile(rf'\\b{re.escape(target_term)}\\b', flags=re.IGNORECASE)\n",
+    "    df_filtered = df_tags[df_tags.apply(\n",
+    "        lambda row: row.astype(str).apply(lambda x: bool(pattern.search(x))).any(),\n",
+    "        axis=1\n",
+    "    )]\n",
+    "\n",
+    "    # Skip if no data matches\n",
+    "    if df_filtered.empty:\n",
+    "        print(f\"  ⚠️ No matches for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === COUNT INDIVIDUAL TAGS ===\n",
+    "    all_tags = df_filtered.values.flatten()\n",
+    "    all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "    tag_counts = Counter(all_tags)\n",
+    "\n",
+    "    # === CO-OCCURRENCE ===\n",
+    "    co_occurrences = defaultdict(int)\n",
+    "    for tags in df_filtered.itertuples(index=False, name=None):\n",
+    "        tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "        for tag1, tag2 in combinations(tags, 2):\n",
+    "            co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "    edges = [(list(pair)[0], list(pair)[1], weight) for pair, weight in co_occurrences.items()]\n",
+    "\n",
+    "    # === FILTER BY CONNECTIONS ===\n",
+    "    connected_tags = Counter()\n",
+    "    for tag1, tag2, _ in edges:\n",
+    "        connected_tags[tag1] += 1\n",
+    "        connected_tags[tag2] += 1\n",
+    "\n",
+    "    nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts if connected_tags[tag] >= min_connections]\n",
+    "    valid_ids = set(node[\"id\"] for node in nodes)\n",
+    "    links = [{\"source\": tag1, \"target\": tag2, \"value\": weight}\n",
+    "             for tag1, tag2, weight in edges\n",
+    "             if tag1 in valid_ids and tag2 in valid_ids]\n",
+    "\n",
+    "    if not nodes or not links:\n",
+    "        print(f\"  ⚠️ Not enough connections for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === EXPORT ===\n",
+    "    d3_data = {\"nodes\": nodes, \"links\": links}\n",
+    "    safe_term = re.sub(r'\\W+', '_', target_term.lower())\n",
+    "    output_file = os.path.join(output_dir, f\"tags_{safe_term}_poi.json\")\n",
+    "    \n",
+    "    with open(output_file, \"w\") as f:\n",
+    "        json.dump(d3_data, f, indent=4)\n",
+    "    \n",
+    "    print(f\"  ✅ Saved to {output_file}\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_79893/1420003123.py:13: DtypeWarning: Columns (50,51,54,55,56,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,458) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  df = pd.read_csv(file_path)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Exported 60330 nodes and 16921 links to public/json/nodes_all.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import os\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = \"data/model_subsets/all_models_poi_false.csv\"\n",
+    "output_file = \"public/json/nodes_all.json\"\n",
+    "min_link_threshold = 10  # Only keep edges with co-occurrence >= this\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === COUNT INDIVIDUAL TAGS ===\n",
+    "all_tags = df_tags.values.flatten()\n",
+    "all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "tag_counts = Counter(all_tags)\n",
+    "\n",
+    "# === CO-OCCURRENCE ===\n",
+    "co_occurrences = defaultdict(int)\n",
+    "for tags in df_tags.itertuples(index=False, name=None):\n",
+    "    tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "    for tag1, tag2 in combinations(tags, 2):\n",
+    "        co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "# === Build Edges (Filtered by co-occurrence threshold)\n",
+    "edges = [\n",
+    "    {\"source\": list(pair)[0], \"target\": list(pair)[1], \"value\": weight}\n",
+    "    for pair, weight in co_occurrences.items()\n",
+    "    if weight >= min_link_threshold\n",
+    "]\n",
+    "\n",
+    "# === Build Nodes (All tags that appear, regardless of links)\n",
+    "nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts]\n",
+    "\n",
+    "# === EXPORT ===\n",
+    "d3_data = {\"nodes\": nodes, \"links\": edges}\n",
+    "\n",
+    "os.makedirs(os.path.dirname(output_file), exist_ok=True)\n",
+    "with open(output_file, \"w\") as f:\n",
+    "    json.dump(d3_data, f, indent=4)\n",
+    "\n",
+    "print(f\"✅ Exported {len(nodes)} nodes and {len(edges)} links to {output_file}\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/.ipynb_checkpoints/Section_2-4_Figure_9_ectract_LoRA_metadata_v2-checkpoint.ipynb

@@ -0,0 +1,400 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f36422c8",
+   "metadata": {},
+   "source": [
+    "# LoRA metadata"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "8a2feb6e",
+   "metadata": {},
+   "source": [
+    "LoRA Metadata Processing Workflow\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Load CSV File        │ --> │ Read adapter metadata CSV file.                   │\n",
+    "│ Read Model Versions  │     │ Extract model version IDs and relevant data.      │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Download Adapter     │ --> │ Use stored download URLs to fetch adapter files   │\n",
+    "│ Files Using API      │     │ using rotating API keys.                          │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Parse Metadata       │ --> │ Extract safetensors metadata, such as training    │\n",
+    "│ from SafeTensor      │     │ images, model type, and architecture.             │\n",
+    "│ Files                │     │                                                   │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Store Parsed         │ --> │ Save extracted metadata into structured JSON      │\n",
+    "│ Metadata as JSON     │     │ files for later analysis.                         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Process JSON Files   │ --> │ Read saved JSON metadata, extract relevant        │\n",
+    "│ for Consolidation    │     │ details, and filter necessary attributes.         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Extract Training     │ --> │ Identify most frequent training tags, architectures│\n",
+    "│ Tags & Model Info    │     │ and systems used for model creation.               │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Save Consolidated    │ --> │ Store all processed metadata in a structured CSV  │\n",
+    "│ Metadata to CSV      │     │ format for final analysis.                        ���\n",
+    "└──────────────────────┘     └───────────────────────────────────────────────────┘\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "efc9939d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import json\n",
+    "import csv\n",
+    "import struct\n",
+    "import requests\n",
+    "from pathlib import Path\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "from pathlib import Path\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.font_manager import FontProperties\n",
+    "from matplotlib import font_manager\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "\n",
+    "# Define the current directory and important file paths\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# Define frequently used directories\n",
+    "\n",
+    "data_dir = current_dir.parent / 'data/csv/adapters.csv'\n",
+    "fonts_dir = current_dir.parent / 'misc/assets/fonts'\n",
+    "plots_dir = current_dir.parent / 'results/plots'\n",
+    "raw_data_dir = current_dir.parent / 'data/adapter_metadata/lora'  ### location of the LoRA metadata (JSON)\n",
+    "temp_dir = current_dir.parent / 'data/raw/adapters_safetensors'\n",
+    "misc_dir = current_dir.parent / 'misc'\n",
+    "\n",
+    "# File paths\n",
+    "adapters_csv = current_dir.parent / 'data/csv/adapters.csv'\n",
+    "output_json_dir = raw_data_dir\n",
+    "api_keys_file = misc_dir / 'credentials/civit.txt'\n",
+    "\n",
+    "# Ensure directories exist\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "\n",
+    "# Load fonts into Matplotlib\n",
+    "for font_path in font_paths:\n",
+    "    font_manager.fontManager.addfont(font_path)\n",
+    "\n",
+    "# Set default font family for plots\n",
+    "plt.rcParams['font.family'] = ['Noto Sans JP', 'Noto Sans SC', 'sans-serif']\n",
+    "\n",
+    "print('Paths and fonts initialized successfully.')\n",
+    "\n",
+    "print('Paths initialized successfully.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87a58593",
+   "metadata": {},
+   "source": [
+    "## Step 2: Download LoRA and extract *.safetensors metadata\n",
+    "This script downloads LoRA adapters from the filtered Civiverse-Models dataset and extracts the metadata found within the *.safetensors' data structure"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "abd3a0bc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import sys\n",
+    "import csv\n",
+    "import json\n",
+    "import struct\n",
+    "import time\n",
+    "import requests\n",
+    "import signal\n",
+    "import contextlib\n",
+    "from pathlib import Path\n",
+    "import re\n",
+    "\n",
+    "# === Paste your API keys here ===\n",
+    "API_KEYS = [\n",
+    "    \"399c06ea6d1b7349556a115376ec346b\",  #DISCORD\n",
+    "    \"213be9d373130f86e394c6fea4d75162\", #ASDD 1\n",
+    "    \"4f180c0c56334b74394b467c5e5b8201\", #ASDD 2\n",
+    "    \"bdfba7ac53290f66bc76130f25b74336\", #BSDD \n",
+    "    \"43294f4a27b388624a896db5a65f445a\"\n",
+    "]\n",
+    "if not API_KEYS or any(not isinstance(k, str) or not k.strip() for k in API_KEYS):\n",
+    "    raise ValueError(\"Please paste at least one valid API key into API_KEYS.\")\n",
+    "\n",
+    "# === Config (adjust paths as needed) ===\n",
+    "current_dir = Path.cwd()\n",
+    "output_json_dir = current_dir.parent / \"data/adapter_metadata/lora\"        # where JSON outputs go\n",
+    "temp_dir = current_dir.parent / \"data/raw/adapters_safetensors\"                            # where downloads go\n",
+    "csv_path = current_dir.parent / \"data/csv/adapters_poi_false_sfw.csv\"\n",
+    "\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "# === API key state ===\n",
+    "current_key_index = 0\n",
+    "\n",
+    "\n",
+    "\n",
+    "def safe_filename(name: str, max_length: int = 100) -> str:\n",
+    "    # Replace unsafe chars\n",
+    "    sanitized = re.sub(r'[^a-zA-Z0-9_\\-]', '_', name)\n",
+    "    # Truncate if too long\n",
+    "    if len(sanitized) > max_length:\n",
+    "        sanitized = sanitized[:max_length]\n",
+    "    return sanitized\n",
+    "\n",
+    "\n",
+    "def get_headers():\n",
+    "    global current_key_index\n",
+    "    return {\n",
+    "        \"Accept\": \"application/json\",\n",
+    "        \"Authorization\": f\"Bearer {API_KEYS[current_key_index].strip()}\"\n",
+    "    }\n",
+    "\n",
+    "def rotate_api_key():\n",
+    "    global current_key_index\n",
+    "    if current_key_index < len(API_KEYS) - 1:\n",
+    "        current_key_index += 1\n",
+    "        print(f\"🔁 Rotated to API key #{current_key_index + 1}\")\n",
+    "    else:\n",
+    "        raise Exception(\"All API keys have been exhausted.\")\n",
+    "\n",
+    "# === Utilities ===\n",
+    "def save_json(data, filename):\n",
+    "    with open(filename, 'w', encoding=\"utf-8\") as f:\n",
+    "        json.dump(data, f, indent=4, ensure_ascii=False)\n",
+    "\n",
+    "def parse_safetensors(file_path):\n",
+    "    # Minimal, tolerant metadata reader; returns {} on failure.\n",
+    "    try:\n",
+    "        with open(file_path, 'rb') as f:\n",
+    "            file_data = f.read()\n",
+    "        # Many safetensors use 8-byte header length; this code follows your original logic\n",
+    "        # (4-byte) but keeps the 8-byte skip. Keep if it's working in your dataset.\n",
+    "        metadata_size = struct.unpack('<I', file_data[:4])[0]\n",
+    "        metadata_bytes = file_data[8:8 + metadata_size]\n",
+    "        metadata_str = metadata_bytes.decode('utf-8', errors='replace')\n",
+    "        metadata = json.loads(metadata_str)\n",
+    "        return metadata.get('__metadata__', {})\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error parsing safetensors file: {e}\")\n",
+    "        return {}\n",
+    "\n",
+    "# === Timeout context ===\n",
+    "class TimeoutException(Exception):\n",
+    "    pass\n",
+    "\n",
+    "@contextlib.contextmanager\n",
+    "def time_limit(seconds):\n",
+    "    def signal_handler(signum, frame):\n",
+    "        raise TimeoutException(f\"Timed out after {seconds} seconds\")\n",
+    "    # Note: SIGALRM works on Unix-like OS; on Windows this will be a no-op.\n",
+    "    try:\n",
+    "        signal.signal(signal.SIGALRM, signal_handler)\n",
+    "        signal.alarm(seconds)\n",
+    "    except Exception:\n",
+    "        # Fallback: no hard alarm on non-Unix systems\n",
+    "        pass\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        try:\n",
+    "            signal.alarm(0)\n",
+    "        except Exception:\n",
+    "            pass\n",
+    "\n",
+    "# === Download with timeout, retries, backoff, and key rotation ===\n",
+    "def download_file(url, output_folder, timeout=30, overall_timeout=120, max_retries=3):\n",
+    "    filename = url.split(\"/\")[-1]\n",
+    "    output_path = os.path.join(output_folder, filename)\n",
+    "\n",
+    "    global current_key_index\n",
+    "    retries = 0\n",
+    "    backoff = 2\n",
+    "\n",
+    "    while current_key_index < len(API_KEYS):\n",
+    "        try:\n",
+    "            with time_limit(overall_timeout):  # global cap per download\n",
+    "                #print(f\"➡️  GET {url} using key #{current_key_index + 1}\")\n",
+    "                resp = requests.get(\n",
+    "                    url,\n",
+    "                    headers=get_headers(),\n",
+    "                    stream=True,\n",
+    "                    timeout=(10, timeout),  # (connect timeout, per-chunk read timeout)\n",
+    "                )\n",
+    "\n",
+    "                # Auth errors → rotate key\n",
+    "                if resp.status_code in (401, 403):\n",
+    "                    print(f\"❌ Auth {resp.status_code} with key #{current_key_index + 1}. Rotating.\")\n",
+    "                    rotate_api_key()\n",
+    "                    retries = 0\n",
+    "                    backoff = 2\n",
+    "                    continue\n",
+    "\n",
+    "                # Not found → bubble up as FileNotFoundError (do not rotate)\n",
+    "                if resp.status_code == 404:\n",
+    "                    raise FileNotFoundError(f\"Model not found at {url}\")\n",
+    "\n",
+    "                # Rate limit → either rotate or wait/backoff\n",
+    "                if resp.status_code == 429:\n",
+    "                    print(\"⏳ Rate limited (429).\", end=\" \")\n",
+    "                    if current_key_index < len(API_KEYS) - 1:\n",
+    "                        print(\"Rotating key.\")\n",
+    "                        rotate_api_key()\n",
+    "                        retries = 0\n",
+    "                        backoff = 2\n",
+    "                        continue\n",
+    "                    else:\n",
+    "                        print(f\"Waiting {backoff}s (no other keys).\")\n",
+    "                        time.sleep(backoff)\n",
+    "                        backoff = min(backoff * 2, 60)\n",
+    "                        continue\n",
+    "\n",
+    "                # Other HTTP errors → raise to RequestException path\n",
+    "                resp.raise_for_status()\n",
+    "\n",
+    "                # Save file\n",
+    "                with open(output_path, 'wb') as fh:\n",
+    "                    for chunk in resp.iter_content(chunk_size=8192):\n",
+    "                        if chunk:\n",
+    "                            fh.write(chunk)\n",
+    "\n",
+    "                return output_path, filename\n",
+    "\n",
+    "        except TimeoutException as e:\n",
+    "            # Hard overall timeout → propagate\n",
+    "            raise e\n",
+    "        except requests.exceptions.RequestException as e:\n",
+    "            # Network-ish errors: retry same key with backoff up to max_retries\n",
+    "            retries += 1\n",
+    "            if retries <= max_retries:\n",
+    "                print(f\"🌐 Network error (try {retries}/{max_retries}) with key #{current_key_index + 1}: {e}\")\n",
+    "                time.sleep(backoff)\n",
+    "                backoff = min(backoff * 2, 60)\n",
+    "                continue\n",
+    "            else:\n",
+    "                raise Exception(f\"Failed to download {url} after {max_retries} retries: {e}\")\n",
+    "\n",
+    "    # If we exit the loop, we truly ran out\n",
+    "    raise Exception(\"All API keys have been exhausted or failed.\")\n",
+    "\n",
+    "# === Main processing ===\n",
+    "def process_csv(csv_path):\n",
+    "    with open(csv_path, newline='', encoding='utf-8') as csvfile:\n",
+    "        reader = csv.DictReader(csvfile)\n",
+    "        for index, row in enumerate(reader):\n",
+    "            # Collect up to 20 version IDs; use the most recent\n",
+    "            version_ids = []\n",
+    "            for i in range(1, 21):\n",
+    "                k = f'version_id_{i}'\n",
+    "                if k in row and row[k]:\n",
+    "                    try:\n",
+    "                        version_ids.append(int(float(row[k])))\n",
+    "                    except ValueError:\n",
+    "                        print(f\"Invalid version_id value '{row[k]}' in row: {row}\")\n",
+    "\n",
+    "            if not version_ids:\n",
+    "                print(f\"No valid version IDs found in row: {row}\")\n",
+    "                continue\n",
+    "\n",
+    "            most_recent_version_id = str(max(version_ids))\n",
+    "            name = row.get('name', 'unknown')\n",
+    "            sanitized_name = safe_filename(name, max_length=100)\n",
+    "            new_json_file = os.path.join(\n",
+    "                output_json_dir,\n",
+    "                f\"{index:08d}_{most_recent_version_id}_{sanitized_name}.json\"\n",
+    "            )\n",
+    "\n",
+    "            # Skip if JSON already exists\n",
+    "            if os.path.exists(new_json_file):\n",
+    "                #print(f\"↩️  Skipping versionID {most_recent_version_id} (JSON already exists)\")\n",
+    "                continue\n",
+    "\n",
+    "            try:\n",
+    "                adapter_file, fname = download_file(\n",
+    "                    row['downloadUrl'], str(temp_dir),\n",
+    "                    timeout=30, overall_timeout=180\n",
+    "                )\n",
+    "                metadata = parse_safetensors(adapter_file)\n",
+    "\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                new_json_data = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": metadata,\n",
+    "                    \"versionID\": most_recent_version_id\n",
+    "                }\n",
+    "                save_json(new_json_data, new_json_file)\n",
+    "                #print(f\"✅ Created JSON for versionID {most_recent_version_id} with file {fname}\")\n",
+    "\n",
+    "            except FileNotFoundError as e:\n",
+    "                print(f\"⚠️ {e} — saving empty metadata.\")\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                empty_json = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": {},\n",
+    "                    \"versionID\": most_recent_version_id,\n",
+    "                    \"error\": \"Model not found (404)\"\n",
+    "                }\n",
+    "                save_json(empty_json, new_json_file)\n",
+    "            except Exception as e:\n",
+    "                print(f\"⚠️ Error processing versionID {most_recent_version_id}: {e}\")\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                empty_json = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": {},\n",
+    "                    \"versionID\": most_recent_version_id,\n",
+    "                    \"error\": str(e)\n",
+    "                }\n",
+    "                save_json(empty_json, new_json_file)\n",
+    "                print(f\"💾 Saved empty JSON for versionID {most_recent_version_id} due to failure.\")\n",
+    "\n",
+    "# === Run ===\n",
+    "if __name__ == \"__main__\":\n",
+    "    process_csv(csv_path)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/0_Scraping_image_metadata.ipynb

@@ -0,0 +1,1345 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1111ea95-d385-49b9-a4d9-ef886ace5c7a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T11:24:25.566747Z",
+     "iopub.status.busy": "2025-02-06T11:24:25.566066Z",
+     "iopub.status.idle": "2025-02-06T11:24:25.571748Z",
+     "shell.execute_reply": "2025-02-06T11:24:25.571305Z",
+     "shell.execute_reply.started": "2025-02-06T11:24:25.566705Z"
+    }
+   },
+   "source": [
+    "# 0 Scraping Metadata and Dataset consolidation\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6632505a-e7ca-4463-9ffc-e36fad42235f",
+   "metadata": {},
+   "source": [
+    "## IMAGES\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e3388bac-bb71-40bc-a693-9ac7a2d5f32c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T10:08:22.229784Z",
+     "iopub.status.busy": "2025-02-06T10:08:22.229287Z",
+     "iopub.status.idle": "2025-02-06T10:08:22.232210Z",
+     "shell.execute_reply": "2025-02-06T10:08:22.231793Z",
+     "shell.execute_reply.started": "2025-02-06T10:08:22.229766Z"
+    }
+   },
+   "source": [
+    "### Step 1: Image metadata scraping, sorting and CSV consolidation"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "8b5d623a-a647-42ae-9951-efa99048cdf3",
+   "metadata": {},
+   "source": [
+    "Dataset creation workflow\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Scrape Data from API │ --> │ Use CivitAI API with a timestamp-based cursor to  │\n",
+    "│ Paginate Using Cursor│     │ scrape image metadata and save in JSON batches.   │\n",
+    "│ Save in JSON Format  │     └───────────────────────────────────────────────────┘\n",
+    "└─────────┬────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Locate JSON Files    │ --> │ Find all JSON files saved in the directory.       │\n",
+    "│ in Directory         │     │                                                   │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Read Each JSON File  │ --> │ Parse JSON files and extract metadata, reactions, │\n",
+    "│ Parse and Extract    │     │ and resource details.                             │\n",
+    "│ Metadata, Reactions, │     │                                                   │\n",
+    "│ and Resources        │     │                                                   │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Sort Items by        │ --> │ Sort the extracted items chronologically using    │\n",
+    "│ createdAt Timestamp  │     │ their createdAt timestamp.                        │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Write Extracted Data │ --> │ Save the processed and sorted data into a         │\n",
+    "│ to a Consolidated    │     │ consolidated CSV file with structured columns.    │\n",
+    "│ CSV File             │     │                                                   │\n",
+    "└────────��─────────────┘     └───────────────────────────────────────────────────┘\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f8decb63-43f5-4731-823d-94632eee7618",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:51.115763Z",
+     "iopub.status.busy": "2025-02-08T19:37:51.114573Z",
+     "iopub.status.idle": "2025-02-08T19:37:51.170027Z",
+     "shell.execute_reply": "2025-02-08T19:37:51.169401Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:51.115738Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import csv\n",
+    "import requests\n",
+    "from datetime import datetime\n",
+    "import time\n",
+    "from pathlib import Path\n",
+    "import hashlib\n",
+    "import pandas as pd\n",
+    "import sys\n",
+    "from datetime import datetime, timedelta\n",
+    "import shutil"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4b2426c3-96a0-468e-b6dc-78dea9c3e92b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:51.809027Z",
+     "iopub.status.busy": "2025-02-08T19:37:51.808835Z",
+     "iopub.status.idle": "2025-02-08T19:37:51.812922Z",
+     "shell.execute_reply": "2025-02-08T19:37:51.812429Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:51.809009Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "dd0daa43-d988-4ac5-b9c7-2f9c65078325",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-07T20:59:30.413939Z",
+     "iopub.status.busy": "2025-02-07T20:59:30.412861Z",
+     "iopub.status.idle": "2025-02-07T20:59:30.418660Z",
+     "shell.execute_reply": "2025-02-07T20:59:30.418054Z",
+     "shell.execute_reply.started": "2025-02-07T20:59:30.413918Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# Define the input timestamp in ISO 8601 format\n",
+    "input_timestamp = \"2025-03-24T12:59:03.335Z\"  # point in time from when you want to obtain metadata (you can copy the timestamp from the last *.json batch obtained to get the data of longer timespans)\n",
+    "\n",
+    "# Function to convert an ISO 8601 date string to a Unix timestamp in milliseconds with a 2-hour offset \n",
+    "def iso_to_timestamp(iso_str):\n",
+    "    # Parse the ISO 8601 date string (including milliseconds and 'Z' indicating UTC)\n",
+    "    date = datetime.strptime(iso_str, \"%Y-%m-%dT%H:%M:%S.%fZ\") \n",
+    "    # Add 2 hours to the parsed date\n",
+    "    adjusted_date = date + timedelta(hours=2)\n",
+    "    # Convert to Unix timestamp (in milliseconds)\n",
+    "    timestamp = int(adjusted_date.timestamp() * 1000)\n",
+    "    return timestamp\n",
+    "\n",
+    "# Convert the input timestamp to an initial cursor\n",
+    "initial_cursor = iso_to_timestamp(input_timestamp)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "20b4bc72-80c4-4af6-bc76-aeecfa65f9dc",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-07T20:59:31.134620Z",
+     "iopub.status.busy": "2025-02-07T20:59:31.133548Z",
+     "iopub.status.idle": "2025-02-07T20:59:31.138081Z",
+     "shell.execute_reply": "2025-02-07T20:59:31.137555Z",
+     "shell.execute_reply.started": "2025-02-07T20:59:31.134599Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "data_raw = current_dir.parent / f\"data/raw/image_metadata/001/{input_timestamp.replace(':', '').replace('T', '_').replace('.', '_').replace('Z', '')}\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f69c921d-ca57-4746-8124-a0fa4fce80ff",
+   "metadata": {},
+   "source": [
+    "#### The *.json files will be saved in the following structure:"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "d731001e-32a4-4894-a2fe-835ea69214fc",
+   "metadata": {},
+   "source": [
+    "data/\n",
+    "└── raw/\n",
+    "    └── image_metadata\n",
+    "           └── 001/\n",
+    "                └── YYYYMMDD_HHMMSS/                 # Folder named based on the input timestamp\n",
+    "                    ├── most_recent_1.json           # Sequential JSON files\n",
+    "                    ├── most_recent_2.json\n",
+    "                    ├── ...\n",
+    "                    ├── most_recent_50000.json\n",
+    "                    └── YYYYMMDD_HHMMSS_session_TIMESTAMP/  # New folder after API limit is reached\n",
+    "                        ├── most_recent_50001.json\n",
+    "                        ├── most_recent_50002.json\n",
+    "                        ├── ...\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "4cb3c005-bf70-43fc-afe8-93bf30872fb4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:51:57.402732Z",
+     "iopub.status.busy": "2025-02-06T16:51:57.402532Z",
+     "iopub.status.idle": "2025-02-06T16:52:00.855604Z",
+     "shell.execute_reply": "2025-02-06T16:52:00.854678Z",
+     "shell.execute_reply.started": "2025-02-06T16:51:57.402712Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "max_images = 50000 # some huge number now redundant because CivitAI API currently limits free api usage to 50'000\n",
+    "\n",
+    "def get_image_metadata():\n",
+    "    base_url = \"https://civitai.com/api/v1/images\"\n",
+    "    headers = {\n",
+    "        \"Accept\": \"application/json\",\n",
+    "        \"Authorization\": \"Bearer APITOKEN\"  # Replace with your actual API token\n",
+    "    }\n",
+    "    params = {\n",
+    "        \"sort\": \"Most Reactions\",\n",
+    "        \"nsfw\": \"X\",\n",
+    "        \"cursor\": f\"0|{initial_cursor}\"\n",
+    "    }\n",
+    "\n",
+    "    # Use pathlib to create the base directory for saving files\n",
+    "\n",
+    "    file_counter = 0\n",
+    "\n",
+    "    # Create a folder based on the input timestamp\n",
+    "    data_raw.mkdir(parents=True, exist_ok=True)\n",
+    "    sub_directory_path = data_raw\n",
+    "\n",
+    "    retry_delay = 300  # 300 seconds / 5 minutes\n",
+    "    retries_without_cursor = 0  # Track consecutive retries without a new cursor\n",
+    "\n",
+    "    while True:\n",
+    "        response = requests.get(base_url, headers=headers, params=params)\n",
+    "        if response.status_code == 200:\n",
+    "            data = response.json()\n",
+    "            items = data.get('items', [])\n",
+    "            if not items:\n",
+    "                print(\"No more data available.\")\n",
+    "                retries_without_cursor += 1\n",
+    "                if retries_without_cursor > 5:  # Allow up to 5 retries before stopping\n",
+    "                    print(\"Reached the end of the data after multiple retries.\")\n",
+    "                    break\n",
+    "                time.sleep(retry_delay)\n",
+    "                continue\n",
+    "\n",
+    "            retries_without_cursor = 0  # Reset if we get data\n",
+    "\n",
+    "            next_cursor = data.get('metadata', {}).get('nextCursor')\n",
+    "            if next_cursor:\n",
+    "                # Increment the cursor by 50 (e.g., \"0|1722470401000\" -> \"50|1722470401000\")\n",
+    "                cursor_value = int(params['cursor'].split(\"|\")[0])\n",
+    "                new_cursor_value = cursor_value + 50\n",
+    "                params['cursor'] = f\"{new_cursor_value}|{params['cursor'].split('|')[1]}\"\n",
+    "            else:\n",
+    "                print(\"No new cursor returned, stopping.\")\n",
+    "                break\n",
+    "\n",
+    "            file_counter += 1\n",
+    "            if file_counter % max_images == 0:\n",
+    "                time_stamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
+    "                sub_directory_path = data_raw / f\"{input_timestamp.replace(':', '').replace('T', '_').replace('.', '_').replace('Z', '')}_session_{time_stamp}\"\n",
+    "                sub_directory_path.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "            file_path = sub_directory_path / f'most_recent_{file_counter}.json'\n",
+    "            with open(file_path, 'w', encoding='utf-8') as file:\n",
+    "                json.dump(data, file, indent=4)\n",
+    "\n",
+    "        elif response.status_code == 502:\n",
+    "            print(f\"Received HTTP 502. Retrying in {retry_delay // 60} minutes.\")\n",
+    "            time.sleep(retry_delay)  # Wait for 5 minutes before retrying\n",
+    "        else:\n",
+    "            print(f\"Failed to fetch data: HTTP {response.status_code}\")\n",
+    "            break\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "238044f1-544f-4821-a91e-57386d464f9e",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:13:44.748157Z",
+     "iopub.status.busy": "2025-02-06T16:13:44.747986Z",
+     "iopub.status.idle": "2025-02-06T16:13:44.752230Z",
+     "shell.execute_reply": "2025-02-06T16:13:44.751694Z",
+     "shell.execute_reply.started": "2025-02-06T16:13:44.748140Z"
+    }
+   },
+   "source": [
+    "uncomment this line to scrape image metadata:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7c89cb68-983b-47df-a028-e02d7ca0829d",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:52:00.857210Z",
+     "iopub.status.busy": "2025-02-06T16:52:00.857015Z",
+     "iopub.status.idle": "2025-02-06T16:52:04.901096Z",
+     "shell.execute_reply": "2025-02-06T16:52:04.900159Z",
+     "shell.execute_reply.started": "2025-02-06T16:52:00.857191Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "get_image_metadata()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "06f5f27e-7278-44c9-ae34-9a2020cac2f2",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "648aee1b-9e01-4400-a6a8-3361c97c377a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T11:33:11.524990Z",
+     "iopub.status.busy": "2025-02-06T11:33:11.523287Z",
+     "iopub.status.idle": "2025-02-06T11:33:11.531385Z",
+     "shell.execute_reply": "2025-02-06T11:33:11.530884Z",
+     "shell.execute_reply.started": "2025-02-06T11:33:11.524893Z"
+    }
+   },
+   "source": [
+    "### Step 02 Chronological Sorting"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "b73d7ea5-50e0-485d-8bc5-a4736f31c789",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:52:04.902663Z",
+     "iopub.status.busy": "2025-02-06T16:52:04.902478Z",
+     "iopub.status.idle": "2025-02-06T16:52:08.491068Z",
+     "shell.execute_reply": "2025-02-06T16:52:08.490105Z",
+     "shell.execute_reply.started": "2025-02-06T16:52:04.902646Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def organize_files(source_dir, target_dir, max_items_per_file=100):\n",
+    "    print(f\"Starting to organize files from {source_dir} to {target_dir}\")\n",
+    "    item_buffer = []\n",
+    "    file_count = 0\n",
+    "\n",
+    "    # Walk through all files in the source directory\n",
+    "    for root, dirs, files in os.walk(source_dir):\n",
+    "        if '.ipynb_checkpoints' in root:\n",
+    "            continue  # Skip .ipynb_checkpoints directories\n",
+    "        print(f\"Checking directory: {root}\")\n",
+    "        for filename in files:\n",
+    "            if filename.lower().endswith('.json'):\n",
+    "                file_path = os.path.join(root, filename)\n",
+    "                try:\n",
+    "                    with open(file_path, 'r') as file:\n",
+    "                        data = json.load(file)\n",
+    "                        items = data.get('items', [])  # Get the list of items\n",
+    "                        for item in items:\n",
+    "                            item_buffer.append(item)\n",
+    "                            # Write out the buffer if it has reached the maximum size\n",
+    "                            if len(item_buffer) >= max_items_per_file:\n",
+    "                                write_items(item_buffer[:max_items_per_file], target_dir)\n",
+    "                                item_buffer = item_buffer[max_items_per_file:]\n",
+    "                                file_count += 1\n",
+    "                except json.JSONDecodeError:\n",
+    "                    print(f\"Error decoding JSON from file {file_path}\")\n",
+    "                except Exception as e:\n",
+    "                    print(f\"An error occurred with file {file_path}: {e}\")\n",
+    "\n",
+    "    # Write any remaining items in the buffer\n",
+    "    if item_buffer:\n",
+    "        write_items(item_buffer, target_dir)\n",
+    "        file_count += 1\n",
+    "\n",
+    "    #print(f\"Processed {file_count} files.\")\n",
+    "\n",
+    "def write_items(items, target_dir):\n",
+    "    # Use the createdAt from the first item to determine the directory\n",
+    "    created_at = items[0].get('createdAt')\n",
+    "    if created_at:\n",
+    "        date_obj = datetime.fromisoformat(created_at.rstrip(\"Z\"))\n",
+    "        new_dir = os.path.join(target_dir, f\"{date_obj.year}\", f\"{date_obj.year}-{date_obj.month:02d}\", f\"{date_obj.year}-{date_obj.month:02d}-{date_obj.day:02d}\")\n",
+    "        os.makedirs(new_dir, exist_ok=True)\n",
+    "        new_file_path = os.path.join(new_dir, f\"batch_{date_obj.strftime('%Y%m%dT%H%M%S')}.json\")\n",
+    "        with open(new_file_path, 'w', encoding='utf-8') as new_file:\n",
+    "            json.dump(items, new_file, indent=4)\n",
+    "        #print(f\"Wrote {len(items)} items to {new_file_path}\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "07cebb8a-2955-4432-a0d1-7ee94975a34b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:52:08.492517Z",
+     "iopub.status.busy": "2025-02-06T16:52:08.492328Z",
+     "iopub.status.idle": "2025-02-06T16:53:13.648165Z",
+     "shell.execute_reply": "2025-02-06T16:53:13.647511Z",
+     "shell.execute_reply.started": "2025-02-06T16:52:08.492499Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Starting to organize files from /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/image_metadata/001/2025-02-01_000000_000 to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/sorted/image_metadata\n",
+      "Checking directory: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/image_metadata/001/2025-02-01_000000_000\n"
+     ]
+    }
+   ],
+   "source": [
+    "data_sorted = current_dir.parent / 'data/sorted/image_metadata'\n",
+    "\n",
+    "origin = data_raw\n",
+    "target = data_sorted\n",
+    "\n",
+    "organize_files( origin, target)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9255fd24-54ee-4235-bc44-a0246e2fd927",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64639464-6dc4-4997-a805-3f9f41f305c7",
+   "metadata": {},
+   "source": [
+    "### Step 3 CSV consolidation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "add5f97b-562a-4af5-a8f2-635433a8158f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:03:15.512133Z",
+     "iopub.status.busy": "2025-02-06T12:03:15.511399Z",
+     "iopub.status.idle": "2025-02-06T12:03:15.516297Z",
+     "shell.execute_reply": "2025-02-06T12:03:15.515815Z",
+     "shell.execute_reply.started": "2025-02-06T12:03:15.512114Z"
+    }
+   },
+   "source": [
+    "#### this script walks through the directories in **image-metadata/** and creates the dataset as **.csv** \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0f8c83aa-c143-4fac-b890-a8d4249067a0",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:02:29.925370Z",
+     "iopub.status.busy": "2025-02-06T12:02:29.923935Z",
+     "iopub.status.idle": "2025-02-06T12:02:31.421575Z",
+     "shell.execute_reply": "2025-02-06T12:02:31.420575Z",
+     "shell.execute_reply.started": "2025-02-06T12:02:29.925332Z"
+    }
+   },
+   "source": [
+    "\n",
+    "### Dataset Columns\n",
+    "\n",
+    "| **Column Name**       | **Description**                                                                                |\n",
+    "|------------------------|-----------------------------------------------------------------------------------------------|\n",
+    "| `createdAt`           | Timestamp when the item was created.                                                          |\n",
+    "| `url`                 | URL associated with the item.                                                                 |\n",
+    "| `positivePrompt`      | Positive prompts used in the generation process.                                              |\n",
+    "| `negativePrompt`      | Negative prompts used in the generation process.                                              |\n",
+    "| `nsfw`                | Indicates whether the item is NSFW (Not Safe for Work).                                       |\n",
+    "| `nsfwLevel`           | Level of NSFW content (e.g., Soft, Mature).                we only considered SFW!!           |\n",
+    "| `browsingLevel`       | Browsing level required to access the content.                                                |\n",
+    "| `statsSummary`        | All social reactions summed up: cryCount, likeCount, heartCount, CommentCount                 |\n",
+    "| `commentCount`        | Number of comments received.                                                                  |\n",
+    "| `username`            | Username of the creator of the item.                                                          |\n",
+    "| `Model`               | Model used to generate the content.                                                           |\n",
+    "| `Meta`                | Simplified meta details, including size, seed, steps, sampler, and version.                   |\n",
+    "| `VAE`                 | Variational Autoencoder (VAE) used, if any.                                                   |\n",
+    "| `resourceIDs`| Array of up to six resources (LoRA etc.), including name, type, and weight.                            |\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "f8db24f9-6678-4e7d-a3b7-453f33ad0f71",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-07T20:59:38.328530Z",
+     "iopub.status.busy": "2025-02-07T20:59:38.328049Z",
+     "iopub.status.idle": "2025-02-07T20:59:38.342970Z",
+     "shell.execute_reply": "2025-02-07T20:59:38.342400Z",
+     "shell.execute_reply.started": "2025-02-07T20:59:38.328508Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def find_json_files(directory):\n",
+    "    \"\"\"Walk through the directory and its subdirectories to find all JSON files.\"\"\"\n",
+    "    json_files = []\n",
+    "    for root, _, files in os.walk(directory):\n",
+    "        for file in files:\n",
+    "            if file.endswith('.json'):\n",
+    "                json_files.append(os.path.join(root, file))\n",
+    "    return json_files\n",
+    "\n",
+    "def hash_username(username):\n",
+    "    \"\"\"Convert a username into a unique hash.\"\"\"\n",
+    "    return hashlib.sha256(username.encode('utf-8')).hexdigest()[:16]  # Use first 16 characters for brevity\n",
+    "\n",
+    "def extract_resource_ids(meta, resources):\n",
+    "    \"\"\"Extract resource IDs from metadata and resources.\"\"\"\n",
+    "    resource_ids = set()  # Use a set to avoid duplicates\n",
+    "    hash_ids = set()  # Separate set to track hashes specifically\n",
+    "\n",
+    "    # Extract CivitAI model IDs\n",
+    "    if \"civitaiResources\" in meta:\n",
+    "        for resource in meta[\"civitaiResources\"]:\n",
+    "            if \"modelVersionId\" in resource:\n",
+    "                resource_ids.add(str(resource[\"modelVersionId\"]))  # Ensure all IDs are strings\n",
+    "\n",
+    "    # Extract model hashes\n",
+    "    if \"Model hash\" in meta:\n",
+    "        hash_ids.add(meta[\"Model hash\"])\n",
+    "    if \"base_model_hash\" in meta:\n",
+    "        hash_ids.add(meta[\"base_model_hash\"])\n",
+    "    if \"models\" in meta and isinstance(meta[\"models\"], list):\n",
+    "        hash_ids.update(str(model) for model in meta[\"models\"])\n",
+    "\n",
+    "    # Extract identifiers from resources list\n",
+    "    for resource in resources:\n",
+    "        if isinstance(resource, dict):  # Ensure resource is a dictionary\n",
+    "            resource_hash = resource.get(\"hash\")\n",
+    "            if resource_hash:\n",
+    "                hash_ids.add(str(resource_hash))  # Add to hash set\n",
+    "            else:\n",
+    "                name = resource.get(\"name\")\n",
+    "                if name:\n",
+    "                    resource_ids.add(str(name))\n",
+    "\n",
+    "    # Remove any named IDs if their hash exists\n",
+    "    resource_ids = {rid for rid in resource_ids if rid not in hash_ids}\n",
+    "\n",
+    "    # Combine hashes and remaining resource IDs\n",
+    "    final_ids = list(hash_ids) + list(resource_ids)\n",
+    "    return final_ids\n",
+    "\n",
+    "def format_resource_ids(resource_ids):\n",
+    "    \"\"\"Format resource IDs in a standardized list format.\"\"\"\n",
+    "    formatted_ids = [str(rid).strip() for rid in resource_ids if rid]\n",
+    "    return f\"[{', '.join(formatted_ids)}]\"\n",
+    "\n",
+    "def truncate_prompt(prompt, max_tokens=77):\n",
+    "    \"\"\"Truncate a prompt to a maximum number of tokens.\"\"\"\n",
+    "    return ' '.join(prompt.split()[:max_tokens]) if prompt else \"\"\n",
+    "\n",
+    "def write_to_csv(json_files, output_csv):\n",
+    "    \"\"\"Read JSON files, extract data, and write to a CSV file.\"\"\"\n",
+    "    with open(output_csv, 'w', newline='', encoding='utf-8') as csvfile:\n",
+    "        fieldnames = [\n",
+    "            'id', 'createdAt', 'url', 'positivePrompt', 'negativePrompt', 'nsfw',\n",
+    "            'browsingLevel', 'statsSummary', 'usernameHash', 'Model', 'cfgScale',\n",
+    "            'sampler', 'Size', 'seed', 'VAE', 'generationSystem', 'resourceIDs'\n",
+    "        ]\n",
+    "        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
+    "        writer.writeheader()\n",
+    "\n",
+    "        for json_file in json_files:\n",
+    "            with open(json_file, 'r', encoding='utf-8') as file:\n",
+    "                try:\n",
+    "                    data = json.load(file)\n",
+    "                    for item in data:\n",
+    "                        meta = item.get('meta', {}) or {}\n",
+    "                        stats = item.get('stats', {}) or {}\n",
+    "                        resources = meta.get('resources', []) if isinstance(meta, dict) else []\n",
+    "\n",
+    "                        # Anonymize username\n",
+    "                        username = item.get('username', '')\n",
+    "                        username_hash = hash_username(username) if username else ''\n",
+    "\n",
+    "                        # Extract resource IDs\n",
+    "                        resource_ids = extract_resource_ids(meta, resources)\n",
+    "                        formatted_resource_ids = format_resource_ids(resource_ids)\n",
+    "\n",
+    "                        # Summarize stats counts\n",
+    "                        stats_summary = sum(stats.get(key, 0) for key in stats)\n",
+    "\n",
+    "                        # Truncate prompts\n",
+    "                        positive_prompt = truncate_prompt(meta.get('prompt', ''))\n",
+    "                        negative_prompt = truncate_prompt(meta.get('negativePrompt', ''))\n",
+    "\n",
+    "                        # Extract metadata fields\n",
+    "                        cfg_scale = meta.get('cfgScale', 'N/A')\n",
+    "                        sampler = meta.get('sampler', 'N/A')\n",
+    "                        size = meta.get('Size', 'N/A')\n",
+    "                        seed = meta.get('seed', 'N/A')\n",
+    "                        vae = meta.get('VAE', 'N/A')\n",
+    "                        generation_system = \"CivitAI\" if \"civitaiResources\" in meta else \"Undetermined\"\n",
+    "\n",
+    "                        # Process a single row\n",
+    "                        row = {\n",
+    "                            'id': item.get('id', ''),\n",
+    "                            'createdAt': item.get('createdAt', ''),\n",
+    "                            'url': item.get('url', ''),\n",
+    "                            'positivePrompt': positive_prompt,\n",
+    "                            'negativePrompt': negative_prompt,\n",
+    "                            'nsfw': item.get('nsfw', False),\n",
+    "                            'browsingLevel': item.get('browsingLevel', 'N/A'),\n",
+    "                            'statsSummary': stats_summary,\n",
+    "                            'usernameHash': username_hash,\n",
+    "                            'Model': meta.get('Model', ''),\n",
+    "                            'cfgScale': cfg_scale,\n",
+    "                            'sampler': sampler,\n",
+    "                            'Size': size,\n",
+    "                            'seed': seed,\n",
+    "                            'VAE': vae,\n",
+    "                            'generationSystem': generation_system,\n",
+    "                            'resourceIDs': formatted_resource_ids\n",
+    "                        }\n",
+    "\n",
+    "                        writer.writerow(row)\n",
+    "                except (json.JSONDecodeError, KeyError, TypeError) as e:\n",
+    "                    print(f\"Error processing file {json_file}: {e}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1c51189e-5eb2-40ef-a5dd-807cd9cded5f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:49:11.129158Z",
+     "iopub.status.busy": "2025-02-06T12:49:11.128195Z",
+     "iopub.status.idle": "2025-02-06T12:49:11.132440Z",
+     "shell.execute_reply": "2025-02-06T12:49:11.131779Z",
+     "shell.execute_reply.started": "2025-02-06T12:49:11.129132Z"
+    }
+   },
+   "source": [
+    "### Save as 'Civiverse' dataset in data/CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "07c97ff9-7ce0-43b7-a156-f5fb79c49cd9",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-07T20:59:40.218975Z",
+     "iopub.status.busy": "2025-02-07T20:59:40.218224Z",
+     "iopub.status.idle": "2025-02-07T21:04:11.221872Z",
+     "shell.execute_reply": "2025-02-07T21:04:11.220974Z",
+     "shell.execute_reply.started": "2025-02-07T20:59:40.218926Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Error processing file /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/sorted/image_metadata/2024/2024-07/2024-07-29/batch_20240729T192800.json: Expecting value: line 1 column 1 (char 0)\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "raw = current_dir.parent / 'data/sorted/image_metadata/'\n",
+    "csv_output = current_dir.parent / 'data/CSV/Civiverse.csv'\n",
+    "\n",
+    "csv_output.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "json_files = find_json_files(raw)\n",
+    "write_to_csv(json_files, csv_output)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b3e64b07-98d1-480d-b4e1-f624a17db848",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T13:50:08.120671Z",
+     "iopub.status.busy": "2025-02-06T13:50:08.119949Z",
+     "iopub.status.idle": "2025-02-06T13:50:08.124761Z",
+     "shell.execute_reply": "2025-02-06T13:50:08.124298Z",
+     "shell.execute_reply.started": "2025-02-06T13:50:08.120632Z"
+    }
+   },
+   "source": [
+    "### Optional step 4: Create sampled subset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "b531b8cd-4653-451d-a6d5-386a2721827c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:55:01.916121Z",
+     "iopub.status.busy": "2025-02-06T16:55:01.915489Z",
+     "iopub.status.idle": "2025-02-06T16:55:01.920043Z",
+     "shell.execute_reply": "2025-02-06T16:55:01.919655Z",
+     "shell.execute_reply.started": "2025-02-06T16:55:01.916103Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "input_file = current_dir.parent / 'data/CSV/Civiverse.csv'\n",
+    "output_file = current_dir.parent / 'data/CSV/image_subsets/Civiverse-mini.csv'\n",
+    "output_file.parent.mkdir(parents=True, exist_ok=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "b82db8b3-310c-49e4-9d7a-aa34662f9556",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:55:01.920861Z",
+     "iopub.status.busy": "2025-02-06T16:55:01.920586Z",
+     "iopub.status.idle": "2025-02-06T16:55:06.790164Z",
+     "shell.execute_reply": "2025-02-06T16:55:06.789503Z",
+     "shell.execute_reply.started": "2025-02-06T16:55:01.920844Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/sctmp/lauwag/ipykernel_1043676/3764041865.py:4: DtypeWarning: Columns (13) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  df = pd.read_csv(input_file)\n",
+      "/sctmp/lauwag/ipykernel_1043676/3764041865.py:10: SettingWithCopyWarning: \n",
+      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
+      "Try using .loc[row_indexer,col_indexer] = value instead\n",
+      "\n",
+      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
+      "  df_sample['createdAt'] = pd.to_datetime(df_sample['createdAt'], errors='coerce')\n"
+     ]
+    }
+   ],
+   "source": [
+    "\n",
+    "# Define the sampling rate\n",
+    "n = 100\n",
+    "# Read the CSV file\n",
+    "df = pd.read_csv(input_file)\n",
+    "\n",
+    "# Sample every 15th row\n",
+    "df_sample = df.iloc[::n, :]\n",
+    "\n",
+    "# Convert the 'createdAt' column to datetime, inferring mixed formats\n",
+    "df_sample['createdAt'] = pd.to_datetime(df_sample['createdAt'], errors='coerce')\n",
+    "\n",
+    "# Sort the sampled data by the 'createdAt' column\n",
+    "df_sample_sorted = df_sample.sort_values(by='createdAt')\n",
+    "\n",
+    "# Save the sorted sample DataFrame to a new CSV file\n",
+    "df_sample_sorted.to_csv(output_file, index=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d915b7d6-937a-4fa9-9edc-ded67e3606f5",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "---\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11647bb7-5ce9-414a-8486-5bdce8d9cfea",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:49:43.126762Z",
+     "iopub.status.busy": "2025-02-06T12:49:43.125797Z",
+     "iopub.status.idle": "2025-02-06T12:49:43.129759Z",
+     "shell.execute_reply": "2025-02-06T12:49:43.129176Z",
+     "shell.execute_reply.started": "2025-02-06T12:49:43.126736Z"
+    }
+   },
+   "source": [
+    "## MODELS"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5c9cb5e3-7cea-4574-9319-f3cd89354b1f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T13:23:38.017078Z",
+     "iopub.status.busy": "2025-02-06T13:23:38.016639Z",
+     "iopub.status.idle": "2025-02-06T13:23:38.019993Z",
+     "shell.execute_reply": "2025-02-06T13:23:38.019549Z",
+     "shell.execute_reply.started": "2025-02-06T13:23:38.017053Z"
+    }
+   },
+   "source": [
+    "### Step 1: Scrape model metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "83e12b9f-5ae1-407d-9754-5979d837f787",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T14:06:04.857874Z",
+     "iopub.status.busy": "2025-02-06T14:06:04.857500Z",
+     "iopub.status.idle": "2025-02-06T14:06:04.860438Z",
+     "shell.execute_reply": "2025-02-06T14:06:04.860030Z",
+     "shell.execute_reply.started": "2025-02-06T14:06:04.857856Z"
+    }
+   },
+   "source": [
+    "#### the resulting files will appear in data/raw/model_metadata as *.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "41ee14e0-fb78-4f91-aba1-13faf05af7d8",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:56.687832Z",
+     "iopub.status.busy": "2025-02-08T19:37:56.687251Z",
+     "iopub.status.idle": "2025-02-08T19:37:56.696572Z",
+     "shell.execute_reply": "2025-02-08T19:37:56.696059Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:56.687809Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import datetime\n",
+    "\n",
+    "def load_api_keys():\n",
+    "    \"\"\"Load API keys from a text file, one per line.\"\"\"\n",
+    "    if not os.path.exists(key_karussell):\n",
+    "        raise FileNotFoundError(f\"API key file '{API_KEYS_FILE}' not found!\")\n",
+    "    \n",
+    "    with open(key_karussell, 'r') as file:\n",
+    "        keys = [line.strip() for line in file if line.strip()]\n",
+    "    \n",
+    "    if not keys:\n",
+    "        raise ValueError(\"No API keys found in the file!\")\n",
+    "    \n",
+    "    return keys\n",
+    "\n",
+    "def get_model_metadata():\n",
+    "    base_url = \"https://civitai.com/api/v1/models\"\n",
+    "    params = {\"sort\": \"Newest\", \"nsfw\": True}\n",
+    "\n",
+    "    # Load API keys\n",
+    "    api_keys = load_api_keys()\n",
+    "    key_index = 0  # Start with the first key\n",
+    "\n",
+    "    page_counter = 0\n",
+    "    max_pages = 300000000  # Adjust as needed\n",
+    "    os.makedirs(directory_path, exist_ok=True)\n",
+    "\n",
+    "    while True:\n",
+    "        if page_counter >= max_pages:\n",
+    "            print(f\"Reached the limit of {max_pages} pages.\")\n",
+    "            break\n",
+    "\n",
+    "        headers = {\n",
+    "            \"Accept\": \"application/json\",\n",
+    "            \"Authorization\": f\"Bearer {api_keys[key_index]}\"\n",
+    "        }\n",
+    "\n",
+    "        response = requests.get(base_url, headers=headers, params=params)\n",
+    "\n",
+    "        if response.status_code == 200:\n",
+    "            data = response.json()\n",
+    "            page_counter += 1\n",
+    "\n",
+    "            # Add timestamp\n",
+    "            formatted_timestamp = datetime.datetime.now().strftime(\"data obtained on the %d.%m.%Y at %H:%M CEST\")\n",
+    "\n",
+    "            data['timestamp'] = formatted_timestamp\n",
+    "\n",
+    "            # Save data to file\n",
+    "            file_path = os.path.join(directory_path, f'newest_models_{page_counter}.json')\n",
+    "            with open(file_path, 'w', encoding='utf-8') as file:\n",
+    "                json.dump(data, file, indent=4)\n",
+    "\n",
+    "            # Check for nextCursor\n",
+    "            next_cursor = data.get('metadata', {}).get('nextCursor')\n",
+    "            if not next_cursor:\n",
+    "                print(\"No more data available.\")\n",
+    "                break\n",
+    "            else:\n",
+    "                params['cursor'] = next_cursor\n",
+    "        \n",
+    "        elif response.status_code in (401, 403):  # Unauthorized or Forbidden\n",
+    "            print(f\"API Key {key_index + 1} failed with status {response.status_code}. Trying next key...\")\n",
+    "            key_index += 1\n",
+    "\n",
+    "            if key_index >= len(api_keys):\n",
+    "                print(\"All API keys failed. Exiting.\")\n",
+    "                break  # Stop if all keys fail\n",
+    "        \n",
+    "        else:\n",
+    "            print(f\"Failed to fetch data: HTTP {response.status_code}\")\n",
+    "            break  # Stop on other errors\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "0b574bc0-8bc5-48fd-9ced-8723d07e0d0e",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:57.395918Z",
+     "iopub.status.busy": "2025-02-08T19:37:57.395733Z",
+     "iopub.status.idle": "2025-02-08T19:37:57.399679Z",
+     "shell.execute_reply": "2025-02-08T19:37:57.399183Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:57.395902Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "key_karussell = current_dir.parent / 'misc/api_keys.txt'\n",
+    "directory_path = current_dir.parent / 'data/raw/model_metadata/'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f43ce23-5986-4b67-be2d-453831af9a6e",
+   "metadata": {},
+   "source": [
+    "uncomment this to get model metadata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "3f95b4ba-5742-4268-b2e8-de9145faf495",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:58.117386Z",
+     "iopub.status.busy": "2025-02-08T19:37:58.117158Z",
+     "iopub.status.idle": "2025-02-08T19:37:58.121162Z",
+     "shell.execute_reply": "2025-02-08T19:37:58.120580Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:58.117369Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "#get_model_metadata()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1ed5f44e-612e-4b6d-8974-904fb3e058d6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:52:40.162173Z",
+     "iopub.status.busy": "2025-02-06T12:52:40.159989Z",
+     "iopub.status.idle": "2025-02-06T12:52:40.170634Z",
+     "shell.execute_reply": "2025-02-06T12:52:40.169945Z",
+     "shell.execute_reply.started": "2025-02-06T12:52:40.162124Z"
+    }
+   },
+   "source": [
+    "### Step 2 Consolidate Model-dataset CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "464d82f5-c24e-4b53-9b50-682fa4bf3430",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:59.052245Z",
+     "iopub.status.busy": "2025-02-08T19:37:59.051645Z",
+     "iopub.status.idle": "2025-02-08T19:37:59.056017Z",
+     "shell.execute_reply": "2025-02-08T19:37:59.055598Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:59.052224Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "## path thingy\n",
+    "try: #scripts\n",
+    "    current_dir = Path(__file__).resolve().parent\n",
+    "except NameError:\n",
+    "    # jupyter\n",
+    "    current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "14f3db4d-ef93-4629-8a27-971adb49248b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:59.443457Z",
+     "iopub.status.busy": "2025-02-08T19:37:59.443236Z",
+     "iopub.status.idle": "2025-02-08T19:37:59.890103Z",
+     "shell.execute_reply": "2025-02-08T19:37:59.889571Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:59.443439Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_2.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_1.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_4.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_6.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_7.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_5.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_3.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import pandas as pd\n",
+    "import hashlib\n",
+    "\n",
+    "\n",
+    "json_path = current_dir.parent / 'data/raw/model_metadata/'\n",
+    "\n",
+    "# Initialize a list to hold all the processed records\n",
+    "data_records = []\n",
+    "\n",
+    "def hash_username(username):\n",
+    "    \"\"\"Convert a username into a unique hash.\"\"\"\n",
+    "    return hashlib.sha256(username.encode('utf-8')).hexdigest()[:16]  # Use first 16 characters for brevity\n",
+    "\n",
+    "# Loop through each file in the directory\n",
+    "for filename in os.listdir(json_path):\n",
+    "    # Construct the full path to the JSON file\n",
+    "    file_path = os.path.join(json_path, filename)\n",
+    "\n",
+    "    # Only process files with .json extension\n",
+    "    if filename.endswith('.json') and os.path.isfile(file_path):\n",
+    "        print(f\"Processing file: {file_path}\")\n",
+    "\n",
+    "        # Open and load the JSON file\n",
+    "        with open(file_path, 'r') as file:\n",
+    "            try:\n",
+    "                data = json.load(file)\n",
+    "            except json.JSONDecodeError as e:\n",
+    "                print(f\"Error decoding JSON in file {filename}: {e}\")\n",
+    "                continue\n",
+    "\n",
+    "            # Determine how the items are structured within the dictionary\n",
+    "            if 'items' in data:\n",
+    "                items = data['items']\n",
+    "            elif 'data' in data:\n",
+    "                items = data['data']\n",
+    "            else:\n",
+    "                print(f\"No known item list key found in {filename}\")\n",
+    "                continue\n",
+    "\n",
+    "            for item in items:\n",
+    "                if not isinstance(item, dict):\n",
+    "                    print(f\"Expected a dictionary for each item, but found: {type(item)} in {filename}\")\n",
+    "                    continue\n",
+    "\n",
+    "                # Extract required information\n",
+    "                model_versions = item.get('modelVersions', [])\n",
+    "                download_url = model_versions[0]['files'][0]['downloadUrl'] if model_versions and 'files' in model_versions[0] and model_versions[0]['files'] else ''\n",
+    "                auto_hashes = model_versions[0]['files'][0]['hashes'] if model_versions and 'files' in model_versions[0] and model_versions[0]['files'] else {}\n",
+    "\n",
+    "                # Collect up to 20 modelVersion IDs\n",
+    "                model_version_ids = [mv.get('id', '') for mv in model_versions[:20]]\n",
+    "\n",
+    "                # Get preview images: first and latest\n",
+    "                first_image_url = model_versions[0]['images'][0]['url'] if model_versions and 'images' in model_versions[0] and model_versions[0]['images'] else ''\n",
+    "                latest_index = len(model_versions) - 1\n",
+    "                latest_image_url = model_versions[latest_index]['images'][0]['url'] if model_versions and 'images' in model_versions[latest_index] and model_versions[latest_index]['images'] else ''\n",
+    "\n",
+    "                # Hash the username\n",
+    "                username = item.get('creator', {}).get('username', '')\n",
+    "                username_hash = hash_username(username) if username else ''\n",
+    "\n",
+    "                # Extract additional fields\n",
+    "                record = {\n",
+    "                    'id': item.get('id', ''),\n",
+    "                    'name': item.get('name', ''),\n",
+    "                    'type': item.get('type', ''),\n",
+    "                    'baseModel': model_versions[0].get('baseModel', '') if model_versions else '',\n",
+    "                    'downloadCount': item.get('stats', {}).get('downloadCount', 0),\n",
+    "                    'nsfwLevel': item.get('nsfwLevel', 0),\n",
+    "                    'modelVersions': len(model_versions),\n",
+    "                    'publishedAt': model_versions[0].get('publishedAt', '') if model_versions else '',\n",
+    "                    'usernameHash': username_hash,\n",
+    "                    'downloadUrl': download_url,\n",
+    "                    'firstImageUrl': first_image_url,\n",
+    "                    'latestImageUrl': latest_image_url,\n",
+    "                    'poi': item.get('poi', False),\n",
+    "                    'AutoV1': auto_hashes.get('AutoV1', ''),\n",
+    "                    'AutoV2': auto_hashes.get('AutoV2', ''),\n",
+    "                    'AutoV3': auto_hashes.get('AutoV3', ''),\n",
+    "                    'SHA256': auto_hashes.get('SHA256', ''),\n",
+    "                    'CRC32': auto_hashes.get('CRC32', ''),\n",
+    "                    'BLAKE3': auto_hashes.get('BLAKE3', ''),\n",
+    "                    'previewImage': latest_image_url\n",
+    "                }\n",
+    "\n",
+    "                # Add version IDs to the record\n",
+    "                for i in range(20):  # Collect up to 20 version IDs\n",
+    "                    version_key = f'version_id_{i+1}'\n",
+    "                    record[version_key] = model_version_ids[i] if len(model_version_ids) > i else ''\n",
+    "\n",
+    "                # Add tags\n",
+    "                tags = item.get('tags', [])\n",
+    "                for i in range(7):  # To retrieve up to tag 6\n",
+    "                    tag_key = f'tag_{i+1}'\n",
+    "                    record[tag_key] = tags[i] if len(tags) > i else ''\n",
+    "\n",
+    "                data_records.append(record)\n",
+    "\n",
+    "# Create a DataFrame and sort by download count\n",
+    "df = pd.DataFrame(data_records)\n",
+    "df_sorted = df.sort_values(by='downloadCount', ascending=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d6f822f-754f-4feb-9ed6-6f868421c454",
+   "metadata": {},
+   "source": [
+    "### Save model-data to CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "f861cf7b-5eb3-46ad-ab2d-9e2fdf2169b4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:38:02.047421Z",
+     "iopub.status.busy": "2025-02-08T19:38:02.046761Z",
+     "iopub.status.idle": "2025-02-08T19:38:02.193381Z",
+     "shell.execute_reply": "2025-02-08T19:38:02.192886Z",
+     "shell.execute_reply.started": "2025-02-08T19:38:02.047397Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Data has been saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/Civiverse-Models.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_csv = current_dir.parent / 'data/CSV/Civiverse-Models.csv'\n",
+    "output_csv.parent.mkdir(parents=True, exist_ok=True)\n",
+    "df_sorted.to_csv(output_csv, index=False)\n",
+    "print(f\"Data has been saved to {output_csv}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9b948933-2a2f-42b0-9083-2d81586ae3f2",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T13:42:32.447860Z",
+     "iopub.status.busy": "2025-02-06T13:42:32.446832Z",
+     "iopub.status.idle": "2025-02-06T13:42:32.453827Z",
+     "shell.execute_reply": "2025-02-06T13:42:32.453271Z",
+     "shell.execute_reply.started": "2025-02-06T13:42:32.447821Z"
+    }
+   },
+   "source": [
+    "### Step 3 Create Subsets: Checkpoint only, POI True, POI False"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "ae894e26-4984-40e6-80a6-54fb6b61c873",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:38:03.089160Z",
+     "iopub.status.busy": "2025-02-08T19:38:03.088490Z",
+     "iopub.status.idle": "2025-02-08T19:38:03.093067Z",
+     "shell.execute_reply": "2025-02-08T19:38:03.092634Z",
+     "shell.execute_reply.started": "2025-02-08T19:38:03.089138Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "file_path = current_dir.parent / 'data/CSV/Civiverse-Models.csv'  # Update this with your actual file path\n",
+    "(current_dir.parent / 'data/CSV/model_subsets').mkdir(parents=True, exist_ok=True)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "88438f88-c723-423e-a4e5-e07f59096b72",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:41:39.279495Z",
+     "iopub.status.busy": "2025-02-08T19:41:39.279039Z",
+     "iopub.status.idle": "2025-02-08T19:41:39.674445Z",
+     "shell.execute_reply": "2025-02-08T19:41:39.673981Z",
+     "shell.execute_reply.started": "2025-02-08T19:41:39.279476Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Files saved successfully!\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "# Load the dataset\n",
+    "\n",
+    "data = pd.read_csv(file_path)\n",
+    "\n",
+    "# Version 1: Only 'poi' true models\n",
+    "poi_true_models = data[data['poi'] == True]\n",
+    "\n",
+    "# Version 2: Only types lora, dora, locon, textual inversion\n",
+    "specific_types = ['LORA', 'DORA', 'LOCON', 'textualInversion']\n",
+    "adapters = data[data['type'].isin(specific_types)]\n",
+    "\n",
+    "# Version 3: Only type checkpoint\n",
+    "checkpoint_models = data[data['type'] == 'Checkpoint']\n",
+    "\n",
+    "# Version 4: All models apart from 'poi' true\n",
+    "non_poi_models = data[data['poi'] != True]\n",
+    "\n",
+    "# Version 5: All models apart from 'poi' true and with nsfwLevel below 13\n",
+    "non_poi_low_nsfw_models = data[(data['poi'] != True) & (data['nsfwLevel'] < 13)]\n",
+    "\n",
+    "# Save the versions as separate CSV files\n",
+    "poi_true_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_adapters_poi_true.csv', index=False)\n",
+    "adapters.to_csv(current_dir.parent / 'data/CSV/adapters.csv', index=False)\n",
+    "checkpoint_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_checkpoint_only.csv', index=False)\n",
+    "non_poi_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_adapters_poi_false.csv', index=False)\n",
+    "\n",
+    "print(\"Files saved successfully!\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "49e35088-8b2e-4189-83e1-3098d55dcad2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import os\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv('data/all_models_with_tags.csv')\n",
+    "\n",
+    "# Filter for rows where poi is True\n",
+    "filtered_df = df[df['poi'] == True]\n",
+    "os.makedirs('data/model_subsets', exist_ok=True)\n",
+    "\n",
+    "# Save the filtered DataFrame to a new CSV file\n",
+    "filtered_df.to_csv('data/model_subsets/all_models_poi.csv', index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "06c15f2c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import os\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv('data/all_models_with_tags.csv')\n",
+    "\n",
+    "# Filter for rows where poi is True\n",
+    "filtered_df = df[df['poi'] == False]\n",
+    "os.makedirs('data/model_subsets', exist_ok=True)\n",
+    "\n",
+    "# Save the filtered DataFrame to a new CSV file\n",
+    "filtered_df.to_csv('data/model_subsets/all_models_poi_false.csv', index=False)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/0_Scraping_model_metadata.ipynb

@@ -0,0 +1,643 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1111ea95-d385-49b9-a4d9-ef886ace5c7a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T11:24:25.566747Z",
+     "iopub.status.busy": "2025-02-06T11:24:25.566066Z",
+     "iopub.status.idle": "2025-02-06T11:24:25.571748Z",
+     "shell.execute_reply": "2025-02-06T11:24:25.571305Z",
+     "shell.execute_reply.started": "2025-02-06T11:24:25.566705Z"
+    }
+   },
+   "source": [
+    "# 0 Scraping Metadata and Dataset consolidation\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6632505a-e7ca-4463-9ffc-e36fad42235f",
+   "metadata": {},
+   "source": [
+    "## IMAGES\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e3388bac-bb71-40bc-a693-9ac7a2d5f32c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T10:08:22.229784Z",
+     "iopub.status.busy": "2025-02-06T10:08:22.229287Z",
+     "iopub.status.idle": "2025-02-06T10:08:22.232210Z",
+     "shell.execute_reply": "2025-02-06T10:08:22.231793Z",
+     "shell.execute_reply.started": "2025-02-06T10:08:22.229766Z"
+    }
+   },
+   "source": [
+    "### Step 1: Image metadata scraping, sorting and CSV consolidation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "f8decb63-43f5-4731-823d-94632eee7618",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:51.115763Z",
+     "iopub.status.busy": "2025-02-08T19:37:51.114573Z",
+     "iopub.status.idle": "2025-02-08T19:37:51.170027Z",
+     "shell.execute_reply": "2025-02-08T19:37:51.169401Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:51.115738Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import csv\n",
+    "import requests\n",
+    "from datetime import datetime\n",
+    "import time\n",
+    "from pathlib import Path\n",
+    "import hashlib\n",
+    "import pandas as pd\n",
+    "import sys\n",
+    "from datetime import datetime, timedelta\n",
+    "import shutil"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4b2426c3-96a0-468e-b6dc-78dea9c3e92b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:51.809027Z",
+     "iopub.status.busy": "2025-02-08T19:37:51.808835Z",
+     "iopub.status.idle": "2025-02-08T19:37:51.812922Z",
+     "shell.execute_reply": "2025-02-08T19:37:51.812429Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:51.809009Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11647bb7-5ce9-414a-8486-5bdce8d9cfea",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:49:43.126762Z",
+     "iopub.status.busy": "2025-02-06T12:49:43.125797Z",
+     "iopub.status.idle": "2025-02-06T12:49:43.129759Z",
+     "shell.execute_reply": "2025-02-06T12:49:43.129176Z",
+     "shell.execute_reply.started": "2025-02-06T12:49:43.126736Z"
+    }
+   },
+   "source": [
+    "## MODELS"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5c9cb5e3-7cea-4574-9319-f3cd89354b1f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T13:23:38.017078Z",
+     "iopub.status.busy": "2025-02-06T13:23:38.016639Z",
+     "iopub.status.idle": "2025-02-06T13:23:38.019993Z",
+     "shell.execute_reply": "2025-02-06T13:23:38.019549Z",
+     "shell.execute_reply.started": "2025-02-06T13:23:38.017053Z"
+    }
+   },
+   "source": [
+    "### Step 1: Scrape model metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "83e12b9f-5ae1-407d-9754-5979d837f787",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T14:06:04.857874Z",
+     "iopub.status.busy": "2025-02-06T14:06:04.857500Z",
+     "iopub.status.idle": "2025-02-06T14:06:04.860438Z",
+     "shell.execute_reply": "2025-02-06T14:06:04.860030Z",
+     "shell.execute_reply.started": "2025-02-06T14:06:04.857856Z"
+    }
+   },
+   "source": [
+    "#### the resulting files will appear in data/raw/model_metadata as *.json"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "5db9c00e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "key_karussell = current_dir.parent / 'misc/credentials/civitai_api_keys.txt'\n",
+    "directory_path = current_dir.parent / 'data/raw/model_metadata/'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "41ee14e0-fb78-4f91-aba1-13faf05af7d8",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:56.687832Z",
+     "iopub.status.busy": "2025-02-08T19:37:56.687251Z",
+     "iopub.status.idle": "2025-02-08T19:37:56.696572Z",
+     "shell.execute_reply": "2025-02-08T19:37:56.696059Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:56.687809Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import datetime\n",
+    "\n",
+    "def load_api_keys():\n",
+    "    \"\"\"Load API keys from a text file, one per line.\"\"\"\n",
+    "    if not os.path.exists(key_karussell):\n",
+    "        raise FileNotFoundError(f\"API key file '{API_KEYS_FILE}' not found!\")\n",
+    "    \n",
+    "    with open(key_karussell, 'r') as file:\n",
+    "        keys = [line.strip() for line in file if line.strip()]\n",
+    "    \n",
+    "    if not keys:\n",
+    "        raise ValueError(\"No API keys found in the file!\")\n",
+    "    \n",
+    "    return keys\n",
+    "\n",
+    "def get_model_metadata():\n",
+    "    base_url = \"https://civitai.com/api/v1/models\"\n",
+    "    params = {\"sort\": \"Newest\", \"nsfw\": True}\n",
+    "\n",
+    "    # Load API keys\n",
+    "    api_keys = load_api_keys()\n",
+    "    key_index = 0  # Start with the first key\n",
+    "\n",
+    "    page_counter = 0\n",
+    "    max_pages = 300000000  # Adjust as needed\n",
+    "    os.makedirs(directory_path, exist_ok=True)\n",
+    "\n",
+    "    while True:\n",
+    "        if page_counter >= max_pages:\n",
+    "            print(f\"Reached the limit of {max_pages} pages.\")\n",
+    "            break\n",
+    "\n",
+    "        headers = {\n",
+    "            \"Accept\": \"application/json\",\n",
+    "            \"Authorization\": f\"Bearer {api_keys[key_index]}\"\n",
+    "        }\n",
+    "\n",
+    "        response = requests.get(base_url, headers=headers, params=params)\n",
+    "\n",
+    "        if response.status_code == 200:\n",
+    "            data = response.json()\n",
+    "            page_counter += 1\n",
+    "\n",
+    "            # Add timestamp\n",
+    "            formatted_timestamp = datetime.datetime.now().strftime(\"data obtained on the %d.%m.%Y at %H:%M CEST\")\n",
+    "\n",
+    "            data['timestamp'] = formatted_timestamp\n",
+    "\n",
+    "            # Save data to file\n",
+    "            file_path = os.path.join(directory_path, f'newest_models_{page_counter}.json')\n",
+    "            with open(file_path, 'w', encoding='utf-8') as file:\n",
+    "                json.dump(data, file, indent=4)\n",
+    "\n",
+    "            # Check for nextCursor\n",
+    "            next_cursor = data.get('metadata', {}).get('nextCursor')\n",
+    "            if not next_cursor:\n",
+    "                print(\"No more data available.\")\n",
+    "                break\n",
+    "            else:\n",
+    "                params['cursor'] = next_cursor\n",
+    "        \n",
+    "        elif response.status_code in (401, 403):  # Unauthorized or Forbidden\n",
+    "            print(f\"API Key {key_index + 1} failed with status {response.status_code}. Trying next key...\")\n",
+    "            key_index += 1\n",
+    "\n",
+    "            if key_index >= len(api_keys):\n",
+    "                print(\"All API keys failed. Exiting.\")\n",
+    "                break  # Stop if all keys fail\n",
+    "        \n",
+    "        else:\n",
+    "            print(f\"Failed to fetch data: HTTP {response.status_code}\")\n",
+    "            break  # Stop on other errors\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f43ce23-5986-4b67-be2d-453831af9a6e",
+   "metadata": {},
+   "source": [
+    "uncomment this to get model metadata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "3f95b4ba-5742-4268-b2e8-de9145faf495",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:58.117386Z",
+     "iopub.status.busy": "2025-02-08T19:37:58.117158Z",
+     "iopub.status.idle": "2025-02-08T19:37:58.121162Z",
+     "shell.execute_reply": "2025-02-08T19:37:58.120580Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:58.117369Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Failed to fetch data: HTTP 500\n"
+     ]
+    }
+   ],
+   "source": [
+    "get_model_metadata()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1ed5f44e-612e-4b6d-8974-904fb3e058d6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T12:52:40.162173Z",
+     "iopub.status.busy": "2025-02-06T12:52:40.159989Z",
+     "iopub.status.idle": "2025-02-06T12:52:40.170634Z",
+     "shell.execute_reply": "2025-02-06T12:52:40.169945Z",
+     "shell.execute_reply.started": "2025-02-06T12:52:40.162124Z"
+    }
+   },
+   "source": [
+    "### Step 2 Consolidate Model-dataset CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "464d82f5-c24e-4b53-9b50-682fa4bf3430",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:59.052245Z",
+     "iopub.status.busy": "2025-02-08T19:37:59.051645Z",
+     "iopub.status.idle": "2025-02-08T19:37:59.056017Z",
+     "shell.execute_reply": "2025-02-08T19:37:59.055598Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:59.052224Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "## path thingy\n",
+    "try: #scripts\n",
+    "    current_dir = Path(__file__).resolve().parent\n",
+    "except NameError:\n",
+    "    # jupyter\n",
+    "    current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "14f3db4d-ef93-4629-8a27-971adb49248b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:37:59.443457Z",
+     "iopub.status.busy": "2025-02-08T19:37:59.443236Z",
+     "iopub.status.idle": "2025-02-08T19:37:59.890103Z",
+     "shell.execute_reply": "2025-02-08T19:37:59.889571Z",
+     "shell.execute_reply.started": "2025-02-08T19:37:59.443439Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_2.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_1.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_4.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_6.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_7.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_5.json\n",
+      "Processing file: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/model_metadata/newest_models_3.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import pandas as pd\n",
+    "import hashlib\n",
+    "from pathlib import Path\n",
+    "from datetime import datetime, timezone\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "def hash_username(username):\n",
+    "    return hashlib.sha256(username.encode('utf-8')).hexdigest()[:16]\n",
+    "\n",
+    "def parse_date(date_str):\n",
+    "    try:\n",
+    "        return datetime.fromisoformat(date_str.replace('Z', '+00:00'))\n",
+    "    except Exception:\n",
+    "        return datetime.min.replace(tzinfo=timezone.utc)  # Make it timezone-aware\n",
+    "\n",
+    "def get_latest_model_version(model_versions):\n",
+    "    return max(model_versions, key=lambda mv: parse_date(mv.get('publishedAt', '')))\n",
+    "\n",
+    "def process_directory_recursively(root_dir):\n",
+    "    root_path = Path(root_dir)\n",
+    "    seen = {}  # id -> (publishedAt, record)\n",
+    "    data_records = []\n",
+    "\n",
+    "    for json_file in root_path.rglob('*.json'):\n",
+    "        if not json_file.is_file():\n",
+    "            continue\n",
+    "\n",
+    "        #print(f\"Processing file: {json_file}\")\n",
+    "        try:\n",
+    "            with open(json_file, 'r', encoding='utf-8') as f:\n",
+    "                data = json.load(f)\n",
+    "        except Exception as e:\n",
+    "            print(f\"Failed to load {json_file}: {e}\")\n",
+    "            continue\n",
+    "\n",
+    "        items = data.get('items') or data.get('data') or []\n",
+    "        for item in items:\n",
+    "            if not isinstance(item, dict):\n",
+    "                continue\n",
+    "\n",
+    "            model_id = item.get('id')\n",
+    "            model_versions = item.get('modelVersions', [])\n",
+    "            if not model_versions:\n",
+    "                continue\n",
+    "\n",
+    "            latest_version = get_latest_model_version(model_versions)\n",
+    "            published_at = latest_version.get('publishedAt', '')\n",
+    "            current_dt = parse_date(published_at)\n",
+    "\n",
+    "            if model_id in seen and current_dt <= seen[model_id][0]:\n",
+    "                continue\n",
+    "            seen[model_id] = (current_dt, item)\n",
+    "\n",
+    "    for model_id, (_, item) in seen.items():\n",
+    "        model_versions = item.get('modelVersions', [])\n",
+    "        latest_version = get_latest_model_version(model_versions)\n",
+    "        version_ids = [mv.get('id', '') for mv in model_versions[:20]]\n",
+    "\n",
+    "        files = latest_version.get('files', [])\n",
+    "        auto_hashes = files[0].get('hashes', {}) if files else {}\n",
+    "        images = latest_version.get('images', [])\n",
+    "        first_image_url = images[0]['url'] if images else ''\n",
+    "        latest_image_url = images[-1]['url'] if images else ''\n",
+    "\n",
+    "        username = item.get('creator', {}).get('username', '')\n",
+    "        record = {\n",
+    "            'id': item.get('id', ''),\n",
+    "            'name': item.get('name', ''),\n",
+    "            'type': item.get('type', ''),\n",
+    "            'baseModel': latest_version.get('baseModel', ''),\n",
+    "            'downloadCount': item.get('stats', {}).get('downloadCount', 0),\n",
+    "            'nsfwLevel': item.get('nsfwLevel', 0),\n",
+    "            'modelVersions': len(model_versions),\n",
+    "            'publishedAt': latest_version.get('publishedAt', ''),\n",
+    "            'usernameHash': hash_username(username) if username else '',\n",
+    "            'downloadUrl': latest_version.get('downloadUrl', ''),\n",
+    "            'firstImageUrl': first_image_url,\n",
+    "            'latestImageUrl': latest_image_url,\n",
+    "            'poi': item.get('poi', False),\n",
+    "            'AutoV1': auto_hashes.get('AutoV1', ''),\n",
+    "            'AutoV2': auto_hashes.get('AutoV2', ''),\n",
+    "            'AutoV3': auto_hashes.get('AutoV3', ''),\n",
+    "            'SHA256': auto_hashes.get('SHA256', ''),\n",
+    "            'CRC32': auto_hashes.get('CRC32', ''),\n",
+    "            'BLAKE3': auto_hashes.get('BLAKE3', ''),\n",
+    "            'previewImage': latest_image_url\n",
+    "        }\n",
+    "\n",
+    "        for i in range(20):\n",
+    "            record[f'version_id_{i+1}'] = version_ids[i] if i < len(version_ids) else ''\n",
+    "\n",
+    "        tags = item.get('tags', [])\n",
+    "        for i in range(7):\n",
+    "            record[f'tag_{i+1}'] = tags[i] if i < len(tags) else ''\n",
+    "\n",
+    "        data_records.append(record)\n",
+    "\n",
+    "    return pd.DataFrame(data_records)\n",
+    "\n",
+    "# Usage Example\n",
+    "root_directory = current_dir.parent / 'data/raw/model_metadata/'\n",
+    "df = process_directory_recursively(root_directory)\n",
+    "df_sorted = df.sort_values(by='downloadCount', ascending=False)\n",
+    "\n",
+    "# Optionally save\n",
+    "# df_sorted.to_csv('combined_metadata.csv', index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d6f822f-754f-4feb-9ed6-6f868421c454",
+   "metadata": {},
+   "source": [
+    "### Save model-data to CSV"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f861cf7b-5eb3-46ad-ab2d-9e2fdf2169b4",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:38:02.047421Z",
+     "iopub.status.busy": "2025-02-08T19:38:02.046761Z",
+     "iopub.status.idle": "2025-02-08T19:38:02.193381Z",
+     "shell.execute_reply": "2025-02-08T19:38:02.192886Z",
+     "shell.execute_reply.started": "2025-02-08T19:38:02.047397Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Data has been saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/Civiverse-Models.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_csv = current_dir.parent / 'data/CSV/Civiverse-Models_2025.csv'\n",
+    "output_csv.parent.mkdir(parents=True, exist_ok=True)\n",
+    "df_sorted.to_csv(output_csv, index=False)\n",
+    "print(f\"Data has been saved to {output_csv}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9b948933-2a2f-42b0-9083-2d81586ae3f2",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T13:42:32.447860Z",
+     "iopub.status.busy": "2025-02-06T13:42:32.446832Z",
+     "iopub.status.idle": "2025-02-06T13:42:32.453827Z",
+     "shell.execute_reply": "2025-02-06T13:42:32.453271Z",
+     "shell.execute_reply.started": "2025-02-06T13:42:32.447821Z"
+    }
+   },
+   "source": [
+    "### Step 3 Create Subsets: Checkpoint only, POI True, POI False"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "ae894e26-4984-40e6-80a6-54fb6b61c873",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:38:03.089160Z",
+     "iopub.status.busy": "2025-02-08T19:38:03.088490Z",
+     "iopub.status.idle": "2025-02-08T19:38:03.093067Z",
+     "shell.execute_reply": "2025-02-08T19:38:03.092634Z",
+     "shell.execute_reply.started": "2025-02-08T19:38:03.089138Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "file_path = current_dir.parent / 'data/CSV/Civiverse-Models.csv'  # Update this with your actual file path\n",
+    "(current_dir.parent / 'data/CSV/model_subsets').mkdir(parents=True, exist_ok=True)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "88438f88-c723-423e-a4e5-e07f59096b72",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T19:41:39.279495Z",
+     "iopub.status.busy": "2025-02-08T19:41:39.279039Z",
+     "iopub.status.idle": "2025-02-08T19:41:39.674445Z",
+     "shell.execute_reply": "2025-02-08T19:41:39.673981Z",
+     "shell.execute_reply.started": "2025-02-08T19:41:39.279476Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Files saved successfully!\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "# Load the dataset\n",
+    "\n",
+    "data = pd.read_csv(file_path)\n",
+    "\n",
+    "# Version 1: Only 'poi' true models\n",
+    "poi_true_models = data[data['poi'] == True]\n",
+    "\n",
+    "# Version 2: Only types lora, dora, locon, textual inversion\n",
+    "specific_types = ['LORA', 'DORA', 'LOCON', 'textualInversion']\n",
+    "adapters = data[data['type'].isin(specific_types)]\n",
+    "\n",
+    "# Version 3: Only type checkpoint\n",
+    "checkpoint_models = data[data['type'] == 'Checkpoint']\n",
+    "\n",
+    "# Version 4: All models apart from 'poi' true\n",
+    "non_poi_models = data[data['poi'] != True]\n",
+    "\n",
+    "# Version 5: All models apart from 'poi' true and with nsfwLevel below 13\n",
+    "non_poi_low_nsfw_models = data[(data['poi'] != True) & (data['nsfwLevel'] < 13)]\n",
+    "\n",
+    "# Save the versions as separate CSV files\n",
+    "poi_true_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_adapters_poi_true.csv', index=False)\n",
+    "adapters.to_csv(current_dir.parent / 'data/CSV/adapters.csv', index=False)\n",
+    "checkpoint_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_checkpoint_only.csv', index=False)\n",
+    "non_poi_models.to_csv(current_dir.parent / 'data/CSV/model_subsets/Civiverse_adapters_poi_false.csv', index=False)\n",
+    "\n",
+    "print(\"Files saved successfully!\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "49e35088-8b2e-4189-83e1-3098d55dcad2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import os\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv('data/all_models_with_tags.csv')\n",
+    "\n",
+    "# Filter for rows where poi is True\n",
+    "filtered_df = df[df['poi'] == True]\n",
+    "os.makedirs('data/model_subsets', exist_ok=True)\n",
+    "\n",
+    "# Save the filtered DataFrame to a new CSV file\n",
+    "filtered_df.to_csv('data/model_subsets/all_models_poi.csv', index=False)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "06c15f2c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import os\n",
+    "\n",
+    "# Load the dataset\n",
+    "df = pd.read_csv('data/all_models_with_tags.csv')\n",
+    "\n",
+    "# Filter for rows where poi is True\n",
+    "filtered_df = df[df['poi'] == False]\n",
+    "os.makedirs('data/model_subsets', exist_ok=True)\n",
+    "\n",
+    "# Save the filtered DataFrame to a new CSV file\n",
+    "filtered_df.to_csv('data/model_subsets/all_models_poi_false.csv', index=False)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_1_Figure_1_image_grid.ipynb

@@ -0,0 +1,417 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "e8752a29-06b6-49a2-a724-a7b1c8f3d23b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T14:16:50.014780Z",
+     "iopub.status.busy": "2025-02-06T14:16:50.013936Z",
+     "iopub.status.idle": "2025-02-06T14:16:50.018030Z",
+     "shell.execute_reply": "2025-02-06T14:16:50.017506Z",
+     "shell.execute_reply.started": "2025-02-06T14:16:50.014757Z"
+    }
+   },
+   "source": [
+    "# Section 2. Download Images and create image Grid"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f81ff2d-7763-4ac5-a9e3-d9a74541e143",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:47:42.502304Z",
+     "iopub.status.busy": "2025-02-06T16:47:42.501200Z",
+     "iopub.status.idle": "2025-02-06T16:47:42.746600Z",
+     "shell.execute_reply": "2025-02-06T16:47:42.745736Z",
+     "shell.execute_reply.started": "2025-02-06T16:47:42.502269Z"
+    }
+   },
+   "source": [
+    "This script downloads images and places them under data/sorted/YYYY/YYYY-MM/YYYY-MM-DD. A grid is created that blurs all NSFW true images and assigns a color overlay based on the color-coding of the BrowsingLevel"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d6d7d826-353c-4c1b-9d69-fa67f07e5245",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T16:17:00.985310Z",
+     "iopub.status.busy": "2025-02-06T16:17:00.984540Z",
+     "iopub.status.idle": "2025-02-06T16:17:01.124348Z",
+     "shell.execute_reply": "2025-02-06T16:17:01.123740Z",
+     "shell.execute_reply.started": "2025-02-06T16:17:00.985282Z"
+    }
+   },
+   "source": [
+    "![Alt text](../plots/grid2x24.png)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "6e5227f5-925c-4155-8015-53045794b986",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:48.202108Z",
+     "iopub.status.busy": "2025-02-08T21:58:48.201629Z",
+     "iopub.status.idle": "2025-02-08T21:58:48.485464Z",
+     "shell.execute_reply": "2025-02-08T21:58:48.484783Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:48.202077Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from PIL import Image, ImageFilter, ImageDraw\n",
+    "import os\n",
+    "from pathlib import Path\n",
+    "import json\n",
+    "import matplotlib.colors as mcolors\n",
+    "import os\n",
+    "import json\n",
+    "import requests\n",
+    "from datetime import datetime\n",
+    "import argparse"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "25cb523d-6d22-4d13-84f1-f9e38534d08e",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:49.153040Z",
+     "iopub.status.busy": "2025-02-08T21:58:49.152592Z",
+     "iopub.status.idle": "2025-02-08T21:58:49.156982Z",
+     "shell.execute_reply": "2025-02-08T21:58:49.156561Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:49.153017Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "037ecae3-a060-4d90-beaf-040dfb018696",
+   "metadata": {},
+   "source": [
+    "## Step 1: Download images "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "948809b0-e78a-4821-86d0-fe4eb156265a",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:50.811934Z",
+     "iopub.status.busy": "2025-02-08T21:58:50.811459Z",
+     "iopub.status.idle": "2025-02-08T21:58:50.823112Z",
+     "shell.execute_reply": "2025-02-08T21:58:50.822600Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:50.811911Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import json\n",
+    "import requests\n",
+    "from datetime import datetime\n",
+    "from PIL import Image\n",
+    "\n",
+    "def download_and_save_data(input_dir, output_dir):\n",
+    "    print(f\"Scanning directory: {input_dir}\")\n",
+    "    found_json = False\n",
+    "    for root, dirs, files in os.walk(input_dir):\n",
+    "        for file in files:\n",
+    "            if file.endswith('.json'):\n",
+    "                found_json = True\n",
+    "                file_path = os.path.join(root, file)\n",
+    "                print(f\"Processing JSON file: {file_path}\")\n",
+    "                try:\n",
+    "                    with open(file_path, 'r', encoding='utf-8') as json_file:\n",
+    "                        items = json.load(json_file)\n",
+    "                        for item in items:\n",
+    "                            if isinstance(item, dict):\n",
+    "                                process_item(item, root, output_dir, input_dir)\n",
+    "                except json.JSONDecodeError:\n",
+    "                    print(f\"Error decoding JSON from file {file_path}\")\n",
+    "                except Exception as e:\n",
+    "                    print(f\"An error occurred with file {file_path}: {e}\")\n",
+    "    if not found_json:\n",
+    "        print(\"No JSON files found in the directory.\")\n",
+    "\n",
+    "def process_item(item, root, output_dir, input_dir):\n",
+    "    if 'createdAt' in item and 'url' in item:\n",
+    "        created_at = datetime.strptime(item['createdAt'], \"%Y-%m-%dT%H:%M:%S.%fZ\")\n",
+    "        time_str = created_at.strftime(\"%Y%m%dT%H%M%S%f\")  # Include microseconds in the filename\n",
+    "        relative_path = os.path.relpath(root, input_dir)\n",
+    "        save_path = os.path.join(output_dir, relative_path)\n",
+    "        os.makedirs(save_path, exist_ok=True)\n",
+    "        image_path = os.path.join(save_path, f\"{time_str}.jpeg\")\n",
+    "        download_image(item['url'], image_path)\n",
+    "        resize_image(image_path, 512)  # Resize if necessary\n",
+    "        save_prompt(item.get('meta', {}).get('prompt', 'No prompt available'), os.path.join(save_path, f\"{time_str}_positive.txt\"))\n",
+    "        save_prompt(item.get('meta', {}).get('negativePrompt', 'No negative prompt available'), os.path.join(save_path, f\"{time_str}_negative.txt\"))\n",
+    "        save_json(item, os.path.join(save_path, f\"{time_str}.json\"))\n",
+    "    else:\n",
+    "        print(\"Item missing 'createdAt' or 'url', skipping...\")\n",
+    "\n",
+    "def download_image(url, path):\n",
+    "    try:\n",
+    "        response = requests.get(url)\n",
+    "        if response.status_code == 200:\n",
+    "            with open(path, 'wb') as f:\n",
+    "                f.write(response.content)\n",
+    "    except requests.RequestException as e:\n",
+    "        print(f\"Request failed for {url}: {e}\")\n",
+    "\n",
+    "def resize_image(image_path, max_size):\n",
+    "    with Image.open(image_path) as img:\n",
+    "        if img.width > max_size or img.height > max_size:\n",
+    "            img.thumbnail((max_size, max_size))\n",
+    "            img.save(image_path)\n",
+    "\n",
+    "def save_prompt(prompt, path):\n",
+    "    with open(path, 'w') as f:\n",
+    "        f.write(prompt)\n",
+    "\n",
+    "def save_json(data, path):\n",
+    "    with open(path, 'w', encoding='utf-8') as f:\n",
+    "        json.dump(data, f, indent=4)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "133e143d-1bb5-4145-8d59-323570bb6e95",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:51.566234Z",
+     "iopub.status.busy": "2025-02-08T21:58:51.565711Z",
+     "iopub.status.idle": "2025-02-08T21:58:51.568973Z",
+     "shell.execute_reply": "2025-02-08T21:58:51.568474Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:51.566214Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "input_dir = current_dir.parent / 'data/sorted/image_metadata/'\n",
+    "images = current_dir.parent / 'data/sorted/images'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e11c4ad5-c040-42b7-a1b9-a0e21b33ce20",
+   "metadata": {},
+   "source": [
+    "uncomment this to download the images, otherwise proceed with grid creation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "88a6ca8e-51ee-4d27-ba0c-5f026d5750df",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:52.986899Z",
+     "iopub.status.busy": "2025-02-08T21:58:52.986456Z",
+     "iopub.status.idle": "2025-02-08T21:58:52.989508Z",
+     "shell.execute_reply": "2025-02-08T21:58:52.988892Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:52.986878Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Scanning directory: /home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/sorted/image_metadata\n",
+      "No JSON files found in the directory.\n"
+     ]
+    }
+   ],
+   "source": [
+    "download_and_save_data(input_dir, images)  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f247091b-7b45-4cd4-a6ca-3d4dea414e19",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:58:54.127935Z",
+     "iopub.status.busy": "2025-02-08T21:58:54.127626Z",
+     "iopub.status.idle": "2025-02-08T21:58:54.254290Z",
+     "shell.execute_reply": "2025-02-08T21:58:54.253243Z",
+     "shell.execute_reply.started": "2025-02-08T21:58:54.127914Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Grid image saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/plots/grid10x120.png\n"
+     ]
+    }
+   ],
+   "source": [
+    "import random\n",
+    "\n",
+    "# For creating the figure grid\n",
+    "images = []\n",
+    "all_valid_images = []\n",
+    "\n",
+    "\n",
+    "directory = current_dir.parent / 'data/sorted/images/'\n",
+    "#directory = '/home/lauwag/shares/laura_wagner/Civitai_page_analysis/Civitai_dataset/dataset/chronological/full/prompts-images/2024/2024-05/2024-05-31/'\n",
+    "output = current_dir.parent / 'plots/grid10x120.png'\n",
+    "\n",
+    "cell_size = 64  # Resize to 32x32 pixels\n",
+    "grid_size = (10, 120)\n",
+    "\n",
+    "\n",
+    "def apply_pixelation(img, level):\n",
+    "    \"\"\" Apply pixelation based on the level. \"\"\"\n",
+    "    if level > 1:\n",
+    "        pixel_size = 8  # Adjust this value for more or less pixelation\n",
+    "        img_small = img.resize(\n",
+    "            (img.width // pixel_size, img.height // pixel_size), Image.NEAREST\n",
+    "        )\n",
+    "        img_pixelated = img_small.resize(img.size, Image.NEAREST)\n",
+    "        return img_pixelated\n",
+    "    return img\n",
+    "\n",
+    "def apply_blur(img, level):\n",
+    "    \"\"\" Apply Gaussian blur based on the level of browsing. \"\"\"\n",
+    "    if level > 1:\n",
+    "        return img.filter(ImageFilter.GaussianBlur(radius=1.75))  # Adjust radius as needed\n",
+    "    return img\n",
+    "\n",
+    "def apply_border_and_overlay(img, color):\n",
+    "    \"\"\" Apply a colored border and a slight color overlay. \"\"\"\n",
+    "    # Convert Matplotlib color name to RGB\n",
+    "    rgb_color = tuple(int(x * 255) for x in mcolors.to_rgb(color))\n",
+    "    # Create a border\n",
+    "    border_size = 0  # Adjust border size as needed\n",
+    "    border_img = Image.new('RGB', (img.width + 2 * border_size, img.height + 2 * border_size), rgb_color)\n",
+    "    border_img.paste(img, (border_size, border_size))\n",
+    "    # Create an overlay\n",
+    "    overlay = Image.new('RGBA', border_img.size, (*rgb_color, 0))  # Semi-transparent overlay\n",
+    "    #overlay = Image.new('RGBA', border_img.size, (*rgb_color, 128))  # Semi-transparent overlay\n",
+    "    final_img = Image.alpha_composite(border_img.convert('RGBA'), overlay)\n",
+    "    return final_img.convert('RGB')\n",
+    "\n",
+    "def process_image(image_path, json_path, cell_size):\n",
+    "    \"\"\" Process each image: resize, crop, blur, and add border and overlay based on JSON metadata. \"\"\"\n",
+    "    with open(json_path, 'r') as f:\n",
+    "        metadata = json.load(f)\n",
+    "    browsing_level = metadata.get('browsingLevel', 1)\n",
+    "\n",
+    "    with Image.open(image_path) as img:\n",
+    "        img = resize_and_crop_image(img, cell_size)\n",
+    "        img = apply_pixelation(img, browsing_level)\n",
+    "\n",
+    "        color_map = {\n",
+    "            2: 'rosybrown',  # Matplotlib color name\n",
+    "            4: 'coral',\n",
+    "            8: 'red',\n",
+    "            16: 'magenta'\n",
+    "        }\n",
+    "        if browsing_level in color_map:\n",
+    "            img = apply_border_and_overlay(img, color_map[browsing_level])\n",
+    "\n",
+    "        return img\n",
+    "\n",
+    "def resize_and_crop_image(img, output_size):\n",
+    "    \"\"\" Resize and crop the image to a square of the specified size. \"\"\"\n",
+    "    ratio = min(img.width / output_size, img.height / output_size)\n",
+    "    new_size = (int(img.width / ratio), int(img.height / ratio))\n",
+    "    img = img.resize(new_size, Image.Resampling.LANCZOS)\n",
+    "    left = (img.width - output_size) // 2\n",
+    "    top = (img.height - output_size) // 2\n",
+    "    right = left + output_size\n",
+    "    bottom = top + output_size\n",
+    "    return img.crop((left, top, right, bottom))\n",
+    "\n",
+    "# Example usage and the rest of your script remains unchanged\n",
+    "\n",
+    "# Example usage\n",
+    "  # 4x4 grid\n",
+    "\n",
+    "file_types = ('png', 'jpg', 'jpeg')  # Define acceptable image file types\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "# First, collect all valid image paths\n",
+    "for root, dirs, files in os.walk(directory):\n",
+    "    for file in files:\n",
+    "        if file.lower().endswith(file_types):\n",
+    "            image_path = os.path.join(root, file)\n",
+    "            json_path = image_path.rsplit('.', 1)[0] + '.json'\n",
+    "            if os.path.exists(json_path):\n",
+    "                all_valid_images.append((image_path, json_path))\n",
+    "\n",
+    "# Randomly sample from valid images\n",
+    "num_needed = grid_size[0] * grid_size[1]\n",
+    "if len(all_valid_images) >= num_needed:\n",
+    "    random.seed(42)  # For reproducibility\n",
+    "    sampled_images = random.sample(all_valid_images, num_needed)\n",
+    "    \n",
+    "    for image_path, json_path in sampled_images:\n",
+    "        img = process_image(image_path, json_path, cell_size)\n",
+    "        images.append(img)\n",
+    "else:\n",
+    "    print(f\"Warning: Only {len(all_valid_images)} valid images found, need {num_needed}\")\n",
+    "\n",
+    "\n",
+    "# Create the grid image\n",
+    "grid_img = Image.new('RGB', (grid_size[1] * cell_size, grid_size[0] * cell_size))\n",
+    "for index, img in enumerate(images):\n",
+    "    x = (index % grid_size[1]) * cell_size\n",
+    "    y = (index // grid_size[1]) * cell_size\n",
+    "    grid_img.paste(img, (x, y))\n",
+    "\n",
+    "grid_img.save(output)\n",
+    "print(f\"Grid image saved to {output}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "28bba299-c5ef-449b-a7fa-1afdc5e26262",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-2-2_Figure_4_Demographic_patterns_in_gen_images.ipynb

@@ -0,0 +1,1795 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "d8161f06-4fd2-436d-9e59-3f68b5a67f2c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:30:16.974712Z",
+     "iopub.status.busy": "2025-02-06T18:30:16.974296Z",
+     "iopub.status.idle": "2025-02-06T18:30:16.976909Z",
+     "shell.execute_reply": "2025-02-06T18:30:16.976526Z",
+     "shell.execute_reply.started": "2025-02-06T18:30:16.974692Z"
+    }
+   },
+   "source": [
+    "# Section 6.2:  Age and Gender Estimation using MiVOLO"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64aaedec-ef56-4a62-b61e-12de2675a1ae",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T19:52:51.171282Z",
+     "iopub.status.busy": "2025-02-06T19:52:51.170711Z",
+     "iopub.status.idle": "2025-02-06T19:52:55.405039Z",
+     "shell.execute_reply": "2025-02-06T19:52:55.404308Z",
+     "shell.execute_reply.started": "2025-02-06T19:52:51.171245Z"
+    }
+   },
+   "source": [
+    "![Alt text](../plots/mivolo.svg)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4293f307-44fd-455e-90fe-6e6928be9af5",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:59:21.970807Z",
+     "iopub.status.busy": "2025-02-08T21:59:21.969931Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.724295Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.723583Z",
+     "shell.execute_reply.started": "2025-02-08T21:59:21.970784Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    }
+   ],
+   "source": [
+    "import csv\n",
+    "from pathlib import Path\n",
+    "import logging\n",
+    "import os\n",
+    "import pandas as pd\n",
+    "import requests\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "import cv2\n",
+    "from io import BytesIO\n",
+    "from PIL import Image, UnidentifiedImageError\n",
+    "from datetime import datetime, timedelta\n",
+    "from dateutil.relativedelta import relativedelta\n",
+    "from mivolo.predictor import Predictor\n",
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.patches as mpatches"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "63a54f5f-900c-48dd-8932-2632e56c5670",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.726069Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.725699Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.730626Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.730099Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.726050Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "mini = current_dir.parent / 'data/CSV/image_subsets/Civiverse-mini.csv'\n",
+    "mivolo_in = current_dir.parent / 'data/CSV/image_subsets/Civiverse-mini-by-month/'\n",
+    "(current_dir.parent /  'data/CSV/image_subsets/Civiverse-mini-by-month/').mkdir(parents=True, exist_ok=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1fdfb89a-6094-4382-8755-fae213221ea5",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.731540Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.731359Z",
+     "iopub.status.idle": "2025-02-08T22:00:09.825738Z",
+     "shell.execute_reply": "2025-02-08T22:00:09.825258Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.731524Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "def split_by_month(input_path, output_dir):\n",
+    "    # Load the dataset\n",
+    "    df = pd.read_csv(input_path)\n",
+    "    \n",
+    "    # Convert the 'createdAt' column to datetime\n",
+    "    df['createdAt'] = pd.to_datetime(df['createdAt'], errors='coerce')\n",
+    "    \n",
+    "    # Extract year and month\n",
+    "    df['year_month'] = df['createdAt'].dt.to_period('M')\n",
+    "    \n",
+    "    # Group the data by year and month and save each group as a CSV file\n",
+    "    unique_months = df['year_month'].unique()\n",
+    "\n",
+    "    for month in unique_months:\n",
+    "        # Filter data for the specific month\n",
+    "        df_month = df[df['year_month'] == month]\n",
+    "        \n",
+    "        # Define the file name based on the year and month\n",
+    "        file_name = f'{output_dir}/Civiverse-{month}.csv'\n",
+    "        \n",
+    "        # Save the file\n",
+    "        df_month.to_csv(file_name, index=False)\n",
+    "\n",
+    "    print(f\"Data has been split and saved to {output_dir}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "2c909d7c-7d16-4dc7-8364-7f1c0784414c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:09.827095Z",
+     "iopub.status.busy": "2025-02-08T22:00:09.826919Z",
+     "iopub.status.idle": "2025-02-08T22:00:10.479484Z",
+     "shell.execute_reply": "2025-02-08T22:00:10.478777Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:09.827079Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/sctmp/lauwag/ipykernel_1497673/1825509207.py:9: UserWarning: Converting to PeriodArray/Index representation will drop timezone information.\n",
+      "  df['year_month'] = df['createdAt'].dt.to_period('M')\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Data has been split and saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month\n"
+     ]
+    }
+   ],
+   "source": [
+    "split_by_month(mini, mivolo_in)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "4c543306-ffc8-4b9c-a3df-b49b2271caa9",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:10.480505Z",
+     "iopub.status.busy": "2025-02-08T22:00:10.480310Z",
+     "iopub.status.idle": "2025-02-08T22:00:10.483961Z",
+     "shell.execute_reply": "2025-02-08T22:00:10.483400Z",
+     "shell.execute_reply.started": "2025-02-08T22:00:10.480486Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "mivolo_out = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'\n",
+    "mivolo_out.mkdir(parents=True, exist_ok=True)  # Create the output directory if it doesn't exist"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ffb7dd23",
+   "metadata": {},
+   "source": [
+    "## MiVOLO gender and age inference"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "304ed12f-c7b6-4129-b24d-7ccc793a62c7",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T22:00:10.484802Z",
+     "iopub.status.busy": "2025-02-08T22:00:10.484639Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/ultralytics/nn/tasks.py:634: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  return torch.load(file, map_location=\"cpu\"), file  # load\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Model summary (fused): 268 layers, 68125494 parameters, 0 gradients, 257.4 GFLOPs\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[W208 23:00:15.738708520 NNPACK.cpp:61] Could not initialize NNPACK! Reason: Unsupported hardware.\n",
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/mivolo/model/mi_volo.py:33: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  state = torch.load(ckpt_path, map_location=\"cpu\")\n",
+      "INFO:MiVOLO:Model meta:\n",
+      "min_age: 1, max_age: 95, avg_age: 48.0, num_classes: 3, in_chans: 6, with_persons_model: True, disable_faces: False, use_persons: True, only_age: False, num_classes_gender: 2, input_size: 224, use_person_crops: True, use_face_crops: True\n",
+      "/home/lauwag/data/conda/envs/horde/lib/python3.12/site-packages/timm/models/_helpers.py:39: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
+      "  checkpoint = torch.load(checkpoint_path, map_location='cpu')\n",
+      "INFO:timm.models._helpers:Loaded state_dict from checkpoint '/shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/ext/MiVOLO/models/mivolo_imbd.pth.tar'\n",
+      "INFO:MiVOLO:Model mivolo_d1_224 created, param count: 27432414\n",
+      "INFO:timm.data.config:Data processing configuration for current model + dataset:\n",
+      "INFO:timm.data.config:\tinput_size: (3, 224, 224)\n",
+      "INFO:timm.data.config:\tinterpolation: bicubic\n",
+      "INFO:timm.data.config:\tmean: (0.485, 0.456, 0.406)\n",
+      "INFO:timm.data.config:\tstd: (0.229, 0.224, 0.225)\n",
+      "INFO:timm.data.config:\tcrop_pct: 0.96\n",
+      "INFO:timm.data.config:\tcrop_mode: center\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2022-11.csv\n",
+      "\n",
+      "0: 640x640 (no detections), 723.9ms\n",
+      "Speed: 12.1ms preprocess, 723.9ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2022-11.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2022-12.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-01.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-02.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-03.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-04.csv\n",
+      "File not found: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-05.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-06.csv\n",
+      "\n",
+      "0: 416x640 1 person, 455.1ms\n",
+      "Speed: 3.5ms preprocess, 455.1ms inference, 33.5ms postprocess per image at shape (1, 3, 416, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 32.89\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-06.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-07.csv\n",
+      "\n",
+      "0: 640x320 1 person, 395.7ms\n",
+      "Speed: 2.9ms preprocess, 395.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 320)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 33.49\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 478.5ms\n",
+      "Speed: 1.9ms preprocess, 478.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.81\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 2 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-07.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-08.csv\n",
+      "\n",
+      "0: 640x448 1 person, 478.0ms\n",
+      "Speed: 2.9ms preprocess, 478.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 40.62\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-08.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-09.csv\n",
+      "\n",
+      "0: 416x640 (no detections), 567.1ms\n",
+      "Speed: 2.4ms preprocess, 567.1ms inference, 0.4ms postprocess per image at shape (1, 3, 416, 640)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 320x640 (no detections), 393.6ms\n",
+      "Speed: 1.7ms preprocess, 393.6ms inference, 0.4ms postprocess per image at shape (1, 3, 320, 640)\n",
+      "\n",
+      "0: 640x640 (no detections), 711.9ms\n",
+      "Speed: 3.4ms preprocess, 711.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x640 (no detections), 699.8ms\n",
+      "Speed: 2.3ms preprocess, 699.8ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x576 1 person, 629.6ms\n",
+      "Speed: 2.4ms preprocess, 629.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 576)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 28.65\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 598.3ms\n",
+      "Speed: 2.1ms preprocess, 598.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.85\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/98848c97-3d1e-4b52-9967-aeeca354a30e/width=656/98848c97-3d1e-4b52-9967-aeeca354a30e.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00133740>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-09.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-10.csv\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/e6469288-b487-4a06-99c1-59e7ac22fa77/width=1024/e6469288-b487-4a06-99c1-59e7ac22fa77.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00ecbdd0>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 448x640 (no detections), 536.6ms\n",
+      "Speed: 10.1ms preprocess, 536.6ms inference, 0.4ms postprocess per image at shape (1, 3, 448, 640)\n",
+      "Processed and saved 2 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-10.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-11.csv\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 662.9ms\n",
+      "Speed: 2.6ms preprocess, 662.9ms inference, 0.9ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.0\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x384 1 person, 895.9ms\n",
+      "Speed: 2.0ms preprocess, 895.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 43.33\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 529.4ms\n",
+      "Speed: 2.6ms preprocess, 529.4ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 539.3ms\n",
+      "Speed: 2.8ms preprocess, 539.3ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 39.15\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 708.6ms\n",
+      "Speed: 2.5ms preprocess, 708.6ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 29.64\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/ad4b5481-0a58-4196-a3e8-fca2fe22a3cc/width=1080/ad4b5481-0a58-4196-a3e8-fca2fe22a3cc.mp4: cannot identify image file <_io.BytesIO object at 0x14cb010c24d0>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-11.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2023-12.csv\n",
+      "\n",
+      "0: 640x384 1 person, 1 face, 461.0ms\n",
+      "Speed: 2.4ms preprocess, 461.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 19.61\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 501.3ms\n",
+      "Speed: 3.1ms preprocess, 501.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.58\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/3004b5fa-af81-4de7-829d-1d809d70b878/width=512/3004b5fa-af81-4de7-829d-1d809d70b878.mp4: cannot identify image file <_io.BytesIO object at 0x14cb0020ad40>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 842.5ms\n",
+      "Speed: 4.5ms preprocess, 842.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x416 (no detections), 446.8ms\n",
+      "Speed: 2.5ms preprocess, 446.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "Processed and saved 5 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2023-12.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-01.csv\n",
+      "\n",
+      "0: 640x448 (no detections), 638.5ms\n",
+      "Speed: 2.3ms preprocess, 638.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x416 (no detections), 441.7ms\n",
+      "Speed: 2.5ms preprocess, 441.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "\n",
+      "0: 640x448 (no detections), 470.3ms\n",
+      "Speed: 2.3ms preprocess, 470.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 693.9ms\n",
+      "Speed: 2.5ms preprocess, 693.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 808.6ms\n",
+      "Speed: 3.2ms preprocess, 808.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 15.01\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x320 1 person, 1 face, 345.6ms\n",
+      "Speed: 2.0ms preprocess, 345.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 320)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.86\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 6 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-01.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-02.csv\n",
+      "\n",
+      "0: 640x384 1 person, 1 face, 387.8ms\n",
+      "Speed: 1.9ms preprocess, 387.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 384)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.31\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x480 1 person, 1 face, 540.4ms\n",
+      "Speed: 2.5ms preprocess, 540.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.47\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 1 person, 1 face, 713.1ms\n",
+      "Speed: 3.8ms preprocess, 713.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 17.85\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 778.8ms\n",
+      "Speed: 28.7ms preprocess, 778.8ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 528.2ms\n",
+      "Speed: 2.3ms preprocess, 528.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.63\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 518.4ms\n",
+      "Speed: 3.9ms preprocess, 518.4ms inference, 0.9ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 18.25\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 470.7ms\n",
+      "Speed: 2.5ms preprocess, 470.7ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.51\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x480 1 person, 1 face, 647.1ms\n",
+      "Speed: 2.4ms preprocess, 647.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 58.87\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 469.8ms\n",
+      "Speed: 2.6ms preprocess, 469.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 477.5ms\n",
+      "Speed: 2.3ms preprocess, 477.5ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 23.79\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 10 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-02.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-03.csv\n",
+      "\n",
+      "0: 640x448 1 face, 511.4ms\n",
+      "Speed: 2.5ms preprocess, 511.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 24.87\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x544 (no detections), 576.5ms\n",
+      "Speed: 2.9ms preprocess, 576.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 544)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 687.1ms\n",
+      "Speed: 9.9ms preprocess, 687.1ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.76\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 498.3ms\n",
+      "Speed: 2.3ms preprocess, 498.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 (no detections), 573.2ms\n",
+      "Speed: 3.0ms preprocess, 573.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "Processed and saved 5 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-03.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-04.csv\n",
+      "\n",
+      "0: 640x384 (no detections), 518.2ms\n",
+      "Speed: 2.7ms preprocess, 518.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x512 (no detections), 707.7ms\n",
+      "Speed: 3.6ms preprocess, 707.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x416 (no detections), 453.7ms\n",
+      "Speed: 2.4ms preprocess, 453.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 416)\n",
+      "\n",
+      "0: 640x384 (no detections), 391.0ms\n",
+      "Speed: 2.0ms preprocess, 391.0ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 449.8ms\n",
+      "Speed: 2.3ms preprocess, 449.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.39\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 618.4ms\n",
+      "Speed: 2.3ms preprocess, 618.4ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 631.0ms\n",
+      "Speed: 2.2ms preprocess, 631.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 24.05\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 1 person, 496.4ms\n",
+      "Speed: 2.6ms preprocess, 496.4ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.81\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 442.8ms\n",
+      "Speed: 2.3ms preprocess, 442.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 477.7ms\n",
+      "Speed: 2.4ms preprocess, 477.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.62\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 447.0ms\n",
+      "Speed: 2.2ms preprocess, 447.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 54.31\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 (no detections), 819.0ms\n",
+      "Speed: 3.6ms preprocess, 819.0ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 478.2ms\n",
+      "Speed: 1.8ms preprocess, 478.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.56\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 471.2ms\n",
+      "Speed: 2.7ms preprocess, 471.2ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.31\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 484.0ms\n",
+      "Speed: 2.2ms preprocess, 484.0ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x640 (no detections), 832.6ms\n",
+      "Speed: 3.0ms preprocess, 832.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 640)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 508.9ms\n",
+      "Speed: 2.5ms preprocess, 508.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 27.19\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6879c7b9-5cb3-42db-b409-30b4e2f71945/width=1080/6879c7b9-5cb3-42db-b409-30b4e2f71945.mp4: cannot identify image file <_io.BytesIO object at 0x14cb0020ad40>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 9 persons, 461.8ms\n",
+      "Speed: 2.4ms preprocess, 461.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([9, 3, 224, 224]), person_input: torch.Size([9, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 30.4\n",
+      "INFO:MiVOLO:\tgender: male [55%]\n",
+      "INFO:MiVOLO:\tage: 28.89\n",
+      "INFO:MiVOLO:\tgender: female [63%]\n",
+      "INFO:MiVOLO:\tage: 30.31\n",
+      "INFO:MiVOLO:\tgender: female [68%]\n",
+      "INFO:MiVOLO:\tage: 31.62\n",
+      "INFO:MiVOLO:\tgender: female [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 33.02\n",
+      "INFO:MiVOLO:\tgender: male [95%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n",
+      "INFO:MiVOLO:\tage: 35.17\n",
+      "INFO:MiVOLO:\tgender: male [53%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 19 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-04.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-05.csv\n",
+      "\n",
+      "0: 640x448 1 person, 455.5ms\n",
+      "Speed: 2.2ms preprocess, 455.5ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 37.57\n",
+      "INFO:MiVOLO:\tgender: male [95%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 438.7ms\n",
+      "Speed: 2.2ms preprocess, 438.7ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 15.62\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 444.8ms\n",
+      "Speed: 2.3ms preprocess, 444.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "ERROR:inference:Unidentified image error for URL https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/6032bd70-6d53-4007-9e89-e69d4748efb5/width=528/6032bd70-6d53-4007-9e89-e69d4748efb5.mp4: cannot identify image file <_io.BytesIO object at 0x14cb00ed0950>\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 453.9ms\n",
+      "Speed: 2.3ms preprocess, 453.9ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 475.0ms\n",
+      "Speed: 1.6ms preprocess, 475.0ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.5\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 447.6ms\n",
+      "Speed: 2.5ms preprocess, 447.6ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 23.46\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 (no detections), 528.5ms\n",
+      "Speed: 3.2ms preprocess, 528.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x448 1 person, 1 face, 449.8ms\n",
+      "Speed: 2.3ms preprocess, 449.8ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 29.32\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 1 face, 617.7ms\n",
+      "Speed: 2.4ms preprocess, 617.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.32\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 609.1ms\n",
+      "Speed: 2.3ms preprocess, 609.1ms inference, 0.5ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 436.2ms\n",
+      "Speed: 2.5ms preprocess, 436.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 585.6ms\n",
+      "Speed: 3.1ms preprocess, 585.6ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.5\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 457.3ms\n",
+      "Speed: 2.1ms preprocess, 457.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 25.19\n",
+      "INFO:MiVOLO:\tgender: male [81%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed and saved 14 images so far.\n",
+      "Processed and saved to: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/MiVOLO-results/2024-05.csv\n",
+      "Processing: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/image_subsets/Civiverse-mini-by-month/Civiverse-2024-06.csv\n",
+      "\n",
+      "0: 640x448 (no detections), 484.5ms\n",
+      "Speed: 2.8ms preprocess, 484.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "Processed and saved 1 images so far.\n",
+      "\n",
+      "0: 640x512 (no detections), 524.8ms\n",
+      "Speed: 2.9ms preprocess, 524.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 512)\n",
+      "\n",
+      "0: 640x480 1 person, 478.0ms\n",
+      "Speed: 2.6ms preprocess, 478.0ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 480)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 39.4\n",
+      "INFO:MiVOLO:\tgender: male [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x512 1 person, 1 face, 539.8ms\n",
+      "Speed: 2.6ms preprocess, 539.8ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 21.33\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 1 person, 2 faces, 446.7ms\n",
+      "Speed: 2.4ms preprocess, 446.7ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([2, 3, 224, 224]), person_input: torch.Size([2, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 20.65\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n",
+      "INFO:MiVOLO:\tage: 20.53\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x640 1 person, 1 face, 655.0ms\n",
+      "Speed: 3.3ms preprocess, 655.0ms inference, 0.8ms postprocess per image at shape (1, 3, 640, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 26.34\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x384 (no detections), 400.6ms\n",
+      "Speed: 2.1ms preprocess, 400.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 384)\n",
+      "\n",
+      "0: 640x448 1 person, 587.9ms\n",
+      "Speed: 2.2ms preprocess, 587.9ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 448)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 30.4\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 610.3ms\n",
+      "Speed: 2.3ms preprocess, 610.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 453.6ms\n",
+      "Speed: 2.3ms preprocess, 453.6ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x512 1 person, 1 face, 511.3ms\n",
+      "Speed: 2.8ms preprocess, 511.3ms inference, 0.7ms postprocess per image at shape (1, 3, 640, 512)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 34.28\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 441.2ms\n",
+      "Speed: 2.3ms preprocess, 441.2ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 586.3ms\n",
+      "Speed: 2.3ms preprocess, 586.3ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 640x448 (no detections), 437.5ms\n",
+      "Speed: 2.4ms preprocess, 437.5ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 448x640 1 person, 1 face, 437.4ms\n",
+      "Speed: 2.4ms preprocess, 437.4ms inference, 0.7ms postprocess per image at shape (1, 3, 448, 640)\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO:MiVOLO:faces_input: torch.Size([1, 3, 224, 224]), person_input: torch.Size([1, 3, 224, 224])\n",
+      "INFO:MiVOLO:\tage: 22.81\n",
+      "INFO:MiVOLO:\tgender: female [99%]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "0: 640x448 (no detections), 436.8ms\n",
+      "Speed: 2.6ms preprocess, 436.8ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n",
+      "0: 448x640 (no detections), 433.0ms\n",
+      "Speed: 1.9ms preprocess, 433.0ms inference, 0.4ms postprocess per image at shape (1, 3, 448, 640)\n",
+      "\n",
+      "0: 640x448 (no detections), 599.7ms\n",
+      "Speed: 2.5ms preprocess, 599.7ms inference, 0.4ms postprocess per image at shape (1, 3, 640, 448)\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Set up logging\n",
+    "detector_weights = current_dir.parent / 'ext/MiVOLO/models/yolov8x_person_face.pt'\n",
+    "checkpoint = current_dir.parent / 'ext/MiVOLO/models/mivolo_imbd.pth.tar'\n",
+    "\n",
+    "_logger = logging.getLogger(\"inference\")\n",
+    "logging.basicConfig(level=logging.INFO)\n",
+    "\n",
+    "# Placeholder configuration and predictor initialization for MiVOLO\n",
+    "class Config:\n",
+    "    def __init__(self, detector_weights, checkpoint, device, with_persons=True, disable_faces=False, draw=False):\n",
+    "        self.detector_weights = detector_weights\n",
+    "        self.checkpoint = checkpoint\n",
+    "        self.device = device\n",
+    "        self.with_persons = with_persons\n",
+    "        self.disable_faces = disable_faces\n",
+    "        self.draw = draw\n",
+    "\n",
+    "\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "config = Config(detector_weights=detector_weights, checkpoint=checkpoint, device=device)\n",
+    "predictor = Predictor(config, verbose=True)\n",
+    "\n",
+    "def download_image(url):\n",
+    "    try:\n",
+    "        response = requests.get(url)\n",
+    "        response.raise_for_status()\n",
+    "        return Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
+    "    except requests.RequestException as e:\n",
+    "        _logger.error(f\"Failed to download image from {url}: {e}\")\n",
+    "        return None\n",
+    "    except UnidentifiedImageError as e:\n",
+    "        _logger.error(f\"Unidentified image error for URL {url}: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "def process_images_with_progress(data, predictor, output_file, start_idx=0):\n",
+    "    results = []\n",
+    "    total_images = len(data)\n",
+    "\n",
+    "    for idx, row in data.iterrows():\n",
+    "        if idx < start_idx:\n",
+    "            continue\n",
+    "\n",
+    "        img_url = row[\"url\"]\n",
+    "        pil_image = download_image(img_url)\n",
+    "        if pil_image is None:\n",
+    "            continue\n",
+    "\n",
+    "        np_image = np.array(pil_image)\n",
+    "        np_image = cv2.cvtColor(np_image, cv2.COLOR_RGB2BGR)\n",
+    "        detected_objects, _ = predictor.recognize(np_image)\n",
+    "\n",
+    "        row_result = row.to_dict()  # Start with the original row's data\n",
+    "\n",
+    "        if detected_objects and detected_objects.ages:\n",
+    "            for i in range(len(detected_objects.ages)):\n",
+    "                age = detected_objects.ages[i]\n",
+    "                gender = detected_objects.genders[i]\n",
+    "                gender_confidence = detected_objects.gender_scores[i]\n",
+    "\n",
+    "                if gender_confidence >= 0.83:\n",
+    "                    detection = {\n",
+    "                        \"detection_type\": 'face' if i in detected_objects.face_to_person_map else 'person',\n",
+    "                        \"gender\": gender,\n",
+    "                        \"gender_confidence\": gender_confidence,\n",
+    "                        \"age\": age,\n",
+    "                        \"n_persons\": detected_objects.n_persons,\n",
+    "                        \"n_faces\": detected_objects.n_faces,\n",
+    "                        \"detected\": True\n",
+    "                    }\n",
+    "                else:\n",
+    "                    detection = {\n",
+    "                        \"detection_type\": \"N/A\",\n",
+    "                        \"gender\": \"N/A\",\n",
+    "                        \"gender_confidence\": 0,\n",
+    "                        \"age\": 0,\n",
+    "                        \"n_persons\": detected_objects.n_persons if detected_objects else 0,\n",
+    "                        \"n_faces\": detected_objects.n_faces if detected_objects else 0,\n",
+    "                        \"detected\": False\n",
+    "                    }\n",
+    "\n",
+    "                results.append({**row_result, **detection})\n",
+    "        else:\n",
+    "            detection = {\n",
+    "                \"detection_type\": \"N/A\",\n",
+    "                \"gender\": \"N/A\",\n",
+    "                \"gender_confidence\": 0,\n",
+    "                \"age\": 0,\n",
+    "                \"n_persons\": detected_objects.n_persons if detected_objects else 0,\n",
+    "                \"n_faces\": detected_objects.n_faces if detected_objects else 0,\n",
+    "                \"detected\": False\n",
+    "            }\n",
+    "            results.append({**row_result, **detection})\n",
+    "\n",
+    "        if idx % 100 == 0 or idx == total_images - 1:\n",
+    "            df = pd.DataFrame(results)\n",
+    "            if os.path.exists(output_file):\n",
+    "                df.to_csv(output_file, mode='a', header=False, index=False)\n",
+    "            else:\n",
+    "                df.to_csv(output_file, mode='w', header=True, index=False)\n",
+    "            results = []\n",
+    "            print(f\"Processed and saved {idx + 1} images so far.\")\n",
+    "\n",
+    "def generate_months(start, end):\n",
+    "    start_date = datetime.strptime(start, '%Y-%m')\n",
+    "    end_date = datetime.strptime(end, '%Y-%m')\n",
+    "    while start_date <= end_date:\n",
+    "        yield start_date.strftime('%Y-%m')\n",
+    "        start_date += relativedelta(months=1)  # Increment by calendar months\n",
+    "\n",
+    "\n",
+    "start_month = '2022-11'\n",
+    "end_month = '2024-12'\n",
+    "\n",
+    "for month in generate_months(start_month, end_month):\n",
+    "    input_file_path = mivolo_in / f'Civiverse-{month}.csv'\n",
+    "    output_file_path = mivolo_out / f'{month}.csv'\n",
+    "\n",
+    "    if input_file_path.exists():\n",
+    "        print(f\"Processing: {input_file_path}\")\n",
+    "\n",
+    "        data = pd.read_csv(input_file_path)\n",
+    "        start_index = 0\n",
+    "        process_images_with_progress(data, predictor, output_file_path, start_idx=start_index)\n",
+    "\n",
+    "        print(f\"Processed and saved to: {output_file_path}\")\n",
+    "    else:\n",
+    "        print(f\"File not found: {input_file_path}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "26aeeef7",
+   "metadata": {},
+   "source": [
+    "## Visualization code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "88ec896a-bf9b-4cc6-a787-c1343f8acb41",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import matplotlib.patches as mpatches\n",
+    "\n",
+    "input_dir = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'\n",
+    "plot_dir = current_dir.parent / 'plots/'\n",
+    "\n",
+    "\n",
+    "all_data = pd.DataFrame()\n",
+    "for file_path in input_dir.glob('*.csv'):  # Reads all CSV files in the folder\n",
+    "    #print(f\"Loading: {file_path}\")\n",
+    "    data = pd.read_csv(file_path)\n",
+    "    all_data = pd.concat([all_data, data], ignore_index=True)\n",
+    "\n",
+    "# Filter rows where detection_type equals \"person\"\n",
+    "person_data = all_data[all_data['detection_type'] == 'person']\n",
+    "\n",
+    "# Count unique images and categorize by persons detected\n",
+    "n_images = all_data['id'].nunique()\n",
+    "images_with_zero_persons = all_data[all_data['n_persons'] == 0]['id'].nunique()\n",
+    "images_with_one_or_more_persons = all_data[all_data['n_persons'] > 0]['id'].nunique()\n",
+    "\n",
+    "n_persons_detected = person_data['id'].nunique()  # Unique images with at least one detected person\n",
+    "total_persons_detected = person_data.shape[0]  # Total number of persons detected\n",
+    "\n",
+    "\n",
+    "\n",
+    "n_total_female = person_data[person_data['gender'] == 'female']['id'].nunique()\n",
+    "n_total_male = person_data[person_data['gender'] == 'male']['id'].nunique()\n",
+    "\n",
+    "# Filter the data further for non-missing age and gender\n",
+    "filtered_data = person_data.dropna(subset=['age', 'gender'])\n",
+    "\n",
+    "# Round ages for consistent plotting\n",
+    "filtered_data['rounded_age'] = np.round(filtered_data['age'] * 4) / 4\n",
+    "\n",
+    "# Map browsingLevel to colors\n",
+    "def get_browsing_color(browsing_level):\n",
+    "    color_mapping = {\n",
+    "        1: 'silver',\n",
+    "        2: 'rosybrown',\n",
+    "        4: 'coral',\n",
+    "        8: 'crimson',\n",
+    "        16: 'blueviolet'\n",
+    "    }\n",
+    "    return color_mapping.get(browsing_level, 'black')  # Default to black for unknown values\n",
+    "\n",
+    "filtered_data['color'] = filtered_data['browsingLevel'].apply(get_browsing_color)\n",
+    "\n",
+    "# Aggregate data for plotting\n",
+    "aggregated_data = (\n",
+    "    filtered_data.groupby(['rounded_age', 'gender', 'color'])\n",
+    "    .size()\n",
+    "    .unstack(fill_value=0)\n",
+    ")\n",
+    "\n",
+    "# Define NSFW colors\n",
+    "nsfw_colors = ['blueviolet', 'crimson', 'coral', 'rosybrown', 'silver']\n",
+    "\n",
+    "# Plotting function\n",
+    "def plot_gender_data(ax, data, gender_label):\n",
+    "    ages = data.index\n",
+    "    bottom = np.zeros(len(ages))\n",
+    "    \n",
+    "    for color in nsfw_colors:\n",
+    "        counts = data[color] if color in data.columns else np.zeros(len(ages))\n",
+    "        ax.bar(\n",
+    "            ages,\n",
+    "            counts,\n",
+    "            color=color,\n",
+    "            edgecolor=color,\n",
+    "            linewidth=1,\n",
+    "            width=0.2,\n",
+    "            bottom=bottom,\n",
+    "            alpha=0.5\n",
+    "        )\n",
+    "        bottom += counts\n",
+    "\n",
+    "    x_min = 5\n",
+    "    x_max = filtered_data['rounded_age'].max()\n",
+    "    ax.set_xticks(np.arange(x_min, x_max + 0.5, 5))\n",
+    "    ax.set_xticklabels([f'{int(age)}' for age in np.arange(x_min, x_max + 0.5, 5)], fontsize=12, fontweight='bold')\n",
+    "    ax.set_xticks(np.arange(x_min, x_max + 0.5, 0.5), minor=True)\n",
+    "\n",
+    "    y_min = 0\n",
+    "    y_max = bottom.max() + 100\n",
+    "    y_ticks = np.arange(y_min, y_max + 1, 100)  # Fine-grained steps of 100\n",
+    "    ax.set_yticks(y_ticks)\n",
+    "    ax.set_yticklabels([str(int(y)) for y in y_ticks], fontsize=12, fontweight='bold')\n",
+    "\n",
+    "    ax.grid(True, which='major', color='lightgrey', linestyle='-', linewidth=0.5)\n",
+    "    ax.grid(True, which='minor', color='lightgrey', linestyle=':', linewidth=0.5)\n",
+    "\n",
+    "    ax.spines['top'].set_visible(False)\n",
+    "    ax.spines['right'].set_visible(False)\n",
+    "    ax.spines['left'].set_visible(False)\n",
+    "    ax.spines['bottom'].set_visible(False)\n",
+    "    \n",
+    "    ax.set_xlabel('Age', fontsize=12, fontweight='bold')\n",
+    "    if gender_label == 'Female':\n",
+    "        ax.set_ylabel('Number of Subjects', fontsize=14, fontweight='bold')\n",
+    "    ax.set_title(f'{gender_label} Read', fontsize=14, fontweight='bold')\n",
+    "\n",
+    "# Set up the subplots\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(14, 6.5), sharey=True)\n",
+    "\n",
+    "plot_gender_data(axes[0], aggregated_data.xs('male', level='gender'), 'Male')\n",
+    "plot_gender_data(axes[1], aggregated_data.xs('female', level='gender'), 'Female')\n",
+    "\n",
+    "legend_patches = [\n",
+    "    mpatches.Patch(facecolor='blueviolet', edgecolor='blueviolet', linewidth=2, label='Level 16: XXX'),\n",
+    "    mpatches.Patch(facecolor='crimson', edgecolor='crimson', linewidth=2, label='Level 8: X'),\n",
+    "    mpatches.Patch(facecolor='coral', edgecolor='coral', linewidth=2, label='Level 4: Mature'),\n",
+    "    mpatches.Patch(facecolor='rosybrown', edgecolor='rosybrown', linewidth=2, label='Level 2: Soft'),\n",
+    "    mpatches.Patch(facecolor='silver', edgecolor='silver', linewidth=2, label='Level 1: SFW'),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'n images: {n_images}', alpha=0),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'Total persons detected: {total_persons_detected}', alpha=0),\n",
+    "    mpatches.Patch(facecolor='none', edgecolor='none', label=f'Unique images containing persons: {n_persons_detected}', alpha=0),\n",
+    "]\n",
+    "\n",
+    "axes[0].legend(handles=legend_patches, title=\"Browsing Levels\", loc='upper left', fontsize=12, title_fontsize=12, frameon=True)\n",
+    "plt.savefig(f'{plot_dir}/mivolo.svg', format='svg', bbox_inches='tight')\n",
+    "plt.tight_layout()\n",
+    "\n",
+    "# Count images with at least one person\n",
+    "images_with_one_or_more_persons = all_data[all_data['n_persons'] > 0]['id'].nunique()\n",
+    "\n",
+    "# Count unique images in `person_data`\n",
+    "n_persons_detected = person_data['id'].nunique()\n",
+    "\n",
+    "# Count total persons detected\n",
+    "total_persons = person_data.shape[0]  # This counts all detected persons\n",
+    "\n",
+    "# Display potential inconsistencies\n",
+    "print(f\"Total images: {n_images}\")\n",
+    "print(f\"Images with at least one person: {images_with_one_or_more_persons}\")\n",
+    "print(f\"Unique images in `person_data`: {n_persons_detected}\")\n",
+    "print(f\"Total number of persons detected: {total_persons}\")\n",
+    "\n",
+    "\n",
+    "\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "42b2a557-b8f4-4d0f-8907-98e3012a1b34",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T20:01:54.848400Z",
+     "iopub.status.busy": "2025-02-06T20:01:54.847713Z",
+     "iopub.status.idle": "2025-02-06T20:01:54.851533Z",
+     "shell.execute_reply": "2025-02-06T20:01:54.851102Z",
+     "shell.execute_reply.started": "2025-02-06T20:01:54.848376Z"
+    }
+   },
+   "source": [
+    "### Latex Table"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3e506c41-6497-4ece-99f3-73f09fe1129e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Define the directory containing CSV files\n",
+    "directory_path = current_dir.parent / 'data/CSV/image_subsets/MiVOLO-results/'  # Update this path with the actual directory path\n",
+    "\n",
+    "# Prepare data for LaTeX table\n",
+    "table_rows = []\n",
+    "\n",
+    "# Loop through each file in the directory\n",
+    "for file_name in os.listdir(directory_path):\n",
+    "    if file_name.endswith('.csv'):\n",
+    "        file_path = os.path.join(directory_path, file_name)\n",
+    "        print(f\"Processing file: {file_name}\")\n",
+    "\n",
+    "        # Load the data\n",
+    "        data = pd.read_csv(file_path)\n",
+    "\n",
+    "        # Total images analyzed\n",
+    "        total_images = data['id'].nunique()\n",
+    "\n",
+    "        # Count of images with no persons detected\n",
+    "        images_no_persons = data[data['n_persons'] == 0]['id'].nunique()\n",
+    "\n",
+    "        # Total persons detected (only using \"person\" detection type)\n",
+    "        total_persons_count = data[data['detection_type'] == 'person'].shape[0]\n",
+    "\n",
+    "        # Average age and standard deviation for male and female individuals\n",
+    "        male_age_stats = data[data['gender'] == 'male']['age'].agg(['mean', 'std']).fillna(0)\n",
+    "        female_age_stats = data[data['gender'] == 'female']['age'].agg(['mean', 'std']).fillna(0)\n",
+    "\n",
+    "        # Count of female and male subjects\n",
+    "        female_images_count = data[data['gender'] == 'female']['id'].nunique()\n",
+    "        male_images_count = data[data['gender'] == 'male']['id'].nunique()\n",
+    "\n",
+    "        # Female to male ratio\n",
+    "        female_to_male_ratio = female_images_count / male_images_count if male_images_count else None\n",
+    "\n",
+    "        # Browsing level analysis for females\n",
+    "        female_browsing_level_1 = data[(data['gender'] == 'female') & (data['browsingLevel'] == 1)]['id'].nunique()\n",
+    "        female_browsing_level_2_16 = data[(data['gender'] == 'female') & (data['browsingLevel'].isin([2, 4, 8, 16]))]['id'].nunique()\n",
+    "        \n",
+    "        female_browsing_level_1_percentage = (female_browsing_level_1 / female_images_count * 100) if female_images_count else 0\n",
+    "        female_browsing_level_2_16_percentage = (female_browsing_level_2_16 / female_images_count * 100) if female_images_count else 0\n",
+    "\n",
+    "        # Browsing level analysis for males\n",
+    "        male_browsing_level_1 = data[(data['gender'] == 'male') & (data['browsingLevel'] == 1)]['id'].nunique()\n",
+    "        male_browsing_level_2_16 = data[(data['gender'] == 'male') & (data['browsingLevel'].isin([2, 4, 8, 16]))]['id'].nunique()\n",
+    "\n",
+    "        male_browsing_level_1_percentage = (male_browsing_level_1 / male_images_count * 100) if male_images_count else 0\n",
+    "        male_browsing_level_2_16_percentage = (male_browsing_level_2_16 / male_images_count * 100) if male_images_count else 0\n",
+    "\n",
+    "        # Add row to table data\n",
+    "        table_rows.append([\n",
+    "            file_name.replace('.csv', ''),  # Remove file extension\n",
+    "            total_images,\n",
+    "            total_persons_count,\n",
+    "            images_no_persons,\n",
+    "            f\"{female_browsing_level_1_percentage:.2f}\",\n",
+    "            f\"{female_browsing_level_2_16_percentage:.2f}\",\n",
+    "            f\"{male_browsing_level_1_percentage:.2f}\",\n",
+    "            f\"{male_browsing_level_2_16_percentage:.2f}\",\n",
+    "            f\"{female_to_male_ratio:.2f}\" if female_to_male_ratio is not None else \"N/A\",\n",
+    "            f\"{female_age_stats['mean']:.2f} ({female_age_stats['std']:.2f})\",\n",
+    "            f\"{male_age_stats['mean']:.2f} ({male_age_stats['std']:.2f})\"\n",
+    "        ])\n",
+    "\n",
+    "# Sort table rows by the filename (assumes filenames are formatted with sortable dates)\n",
+    "table_rows = sorted(table_rows, key=lambda x: x[0])\n",
+    "\n",
+    "# Generate LaTeX table\n",
+    "latex_table = r\"\"\"\n",
+    "\\begin{table}[H]\n",
+    "\\centering\n",
+    "\\scriptsize\n",
+    "\\renewcommand{\\arraystretch}{0.9}\n",
+    "\\caption{Summary of Image Classification for 2023-2024}\n",
+    "\\label{table:image_classification_2023_2024}\n",
+    "\\begin{tabular}{lrrrrrrrrrr}\n",
+    "\\toprule\n",
+    "File Name & Total Images & Total Persons & No Persons & \\multicolumn{2}{c}{Female (\\%)} & \\multicolumn{2}{c}{Male (\\%)} & Female:Male & Female Age (Mean ± SD) & Male Age (Mean ± SD) \\\\\n",
+    " & & & & L1 & L2-16 & L1 & L2-16 & & & \\\\\n",
+    "\\midrule\n",
+    "\"\"\"\n",
+    "for row in table_rows:\n",
+    "    latex_table += \" & \".join(map(str, row)) + r\" \\\\\\\\\\n\"\n",
+    "\n",
+    "latex_table += r\"\"\"\n",
+    "\\bottomrule\n",
+    "\\end{tabular}\n",
+    "\\vspace{1em}\n",
+    "\\noindent\n",
+    "\\textbf{Disclaimer:} \\(\\female\\) and \\(\\male\\) refer to female-read and male-read classifications as determined by the MiVOLO system's weights. \n",
+    "We acknowledge the complexities of gender presentations and stress that these terms do not necessarily correspond to biological sex.\n",
+    "\\end{table}\n",
+    "\"\"\"\n",
+    "\n",
+    "# Output LaTeX table\n",
+    "print(\"\\nGenerated LaTeX Table:\")\n",
+    "print(latex_table)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3ef428be-856b-4c4c-b1b0-a052d181d03b",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-3-1_Tag_occurences.ipynb

@@ -0,0 +1,801 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "71008ae2-4465-45d1-9ad0-6e6d54c99a69",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T20:15:54.294327Z",
+     "iopub.status.busy": "2025-12-09T20:15:54.294119Z",
+     "iopub.status.idle": "2025-12-09T20:15:54.296418Z",
+     "shell.execute_reply": "2025-12-09T20:15:54.295943Z",
+     "shell.execute_reply.started": "2025-12-09T20:15:54.294313Z"
+    }
+   },
+   "source": [
+    "# Tag occurence percentages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "1577b529-19b4-471a-af8b-bc331087bb61",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T20:33:24.090495Z",
+     "iopub.status.busy": "2025-12-09T20:33:24.090266Z",
+     "iopub.status.idle": "2025-12-09T20:33:35.733409Z",
+     "shell.execute_reply": "2025-12-09T20:33:35.732815Z",
+     "shell.execute_reply.started": "2025-12-09T20:33:24.090478Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "==================================================\n",
+      "Tag Analysis for 'anime'\n",
+      "==================================================\n",
+      "Models with tag: 74721\n",
+      "Total models: 232164\n",
+      "Percentage: 32.18%\n",
+      "==================================================\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import pandas as pd\n",
+    "import sys\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# ============================================\n",
+    "# INPUT: Change these values\n",
+    "# ============================================\n",
+    "csv_file = current_dir.parent / \"data/CSV/models/Civi_models.csv\" # Your CSV file path\n",
+    "tag_to_find = \"anime\"  # Tag to search for\n",
+    "# ============================================\n",
+    "\n",
+    "def calculate_tag_percentage(csv_file, tag_to_find):\n",
+    "    \"\"\"\n",
+    "    Calculate what percentage of models contain a specific tag.\n",
+    "    \"\"\"\n",
+    "    # Read the CSV file\n",
+    "    df = pd.read_csv(csv_file)\n",
+    "    \n",
+    "    # Get all tag columns\n",
+    "    tag_columns = [col for col in df.columns if col.startswith('tag_')]\n",
+    "    \n",
+    "    # Count total models\n",
+    "    total_models = len(df)\n",
+    "    \n",
+    "    # Count models containing the tag (case-insensitive search)\n",
+    "    tag_lower = tag_to_find.lower()\n",
+    "    models_with_tag = 0\n",
+    "    \n",
+    "    for idx, row in df.iterrows():\n",
+    "        # Check if the tag appears in any of the tag columns\n",
+    "        for tag_col in tag_columns:\n",
+    "            tag_value = str(row[tag_col]).lower().strip()\n",
+    "            if tag_value == tag_lower:\n",
+    "                models_with_tag += 1\n",
+    "                break  # Count each model only once\n",
+    "    \n",
+    "    # Calculate percentage\n",
+    "    percentage = (models_with_tag / total_models * 100) if total_models > 0 else 0\n",
+    "    \n",
+    "    return {\n",
+    "        'tag': tag_to_find,\n",
+    "        'count': models_with_tag,\n",
+    "        'total': total_models,\n",
+    "        'percentage': percentage\n",
+    "    }\n",
+    "\n",
+    "# Calculate and display results\n",
+    "result = calculate_tag_percentage(csv_file, tag_to_find)\n",
+    "\n",
+    "print(f\"\\n{'='*50}\")\n",
+    "print(f\"Tag Analysis for '{result['tag']}'\")\n",
+    "print(f\"{'='*50}\")\n",
+    "print(f\"Models with tag: {result['count']}\")\n",
+    "print(f\"Total models: {result['total']}\")\n",
+    "print(f\"Percentage: {result['percentage']:.2f}%\")\n",
+    "print(f\"{'='*50}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "id": "825dc9b8-ff0e-4afd-b2f1-e4bca1036aea",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T20:22:12.641296Z",
+     "iopub.status.busy": "2025-12-09T20:22:12.641105Z",
+     "iopub.status.idle": "2025-12-09T20:22:25.293171Z",
+     "shell.execute_reply": "2025-12-09T20:22:25.292530Z",
+     "shell.execute_reply.started": "2025-12-09T20:22:12.641281Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "=== Tag Analysis for '-f' ===\n",
+      "Models with tag: 0\n",
+      "Total models: 232164\n",
+      "Percentage: 0.00%\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import pandas as pd\n",
+    "import sys\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "\n",
+    "def calculate_tag_percentage(csv_file, tag_to_find):\n",
+    "    # Read the CSV file\n",
+    "    df = pd.read_csv(csv_file)\n",
+    "    \n",
+    "    # Get all tag columns\n",
+    "    tag_columns = [col for col in df.columns if col.startswith('tag_')]\n",
+    "    \n",
+    "    # Count total models\n",
+    "    total_models = len(df)\n",
+    "    \n",
+    "    # Count models containing the tag (case-insensitive search)\n",
+    "    tag_lower = tag_to_find.lower()\n",
+    "    models_with_tag = 0\n",
+    "    \n",
+    "    for idx, row in df.iterrows():\n",
+    "        # Check if the tag appears in any of the tag columns\n",
+    "        for tag_col in tag_columns:\n",
+    "            tag_value = str(row[tag_col]).lower().strip()\n",
+    "            if tag_value == tag_lower:\n",
+    "                models_with_tag += 1\n",
+    "                break  # Count each model only once\n",
+    "    \n",
+    "    # Calculate percentage\n",
+    "    percentage = (models_with_tag / total_models * 100) if total_models > 0 else 0\n",
+    "    \n",
+    "    return {\n",
+    "        'tag': tag_to_find,\n",
+    "        'count': models_with_tag,\n",
+    "        'total': total_models,\n",
+    "        'percentage': percentage\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "def analyze_all_tags(csv_file):\n",
+    "\n",
+    "    df = pd.read_csv(csv_file)\n",
+    "    tag_columns = [col for col in df.columns if col.startswith('tag_')]\n",
+    "    total_models = len(df)\n",
+    "    \n",
+    "    # Collect all tags and count occurrences\n",
+    "    tag_counts = {}\n",
+    "    for tag_col in tag_columns:\n",
+    "        for tag in df[tag_col].dropna():\n",
+    "            tag = str(tag).strip()\n",
+    "            if tag:  # Ignore empty strings\n",
+    "                tag_counts[tag] = tag_counts.get(tag, 0) + 1\n",
+    "    \n",
+    "    # Create results DataFrame\n",
+    "    results = []\n",
+    "    for tag, count in tag_counts.items():\n",
+    "        percentage = (count / total_models * 100)\n",
+    "        results.append({\n",
+    "            'tag': tag,\n",
+    "            'count': count,\n",
+    "            'percentage': round(percentage, 2)\n",
+    "        })\n",
+    "    \n",
+    "    results_df = pd.DataFrame(results)\n",
+    "    results_df = results_df.sort_values('count', ascending=False)\n",
+    "    \n",
+    "    return results_df\n",
+    "\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    # Default CSV file path\n",
+    "    csv_file = current_dir.parent / \"data/CSV/models/Civi_models.csv\"\n",
+    "    \n",
+    "    # Check if a specific tag is provided as argument\n",
+    "    if len(sys.argv) > 1:\n",
+    "        tag = sys.argv[1]\n",
+    "        result = calculate_tag_percentage(csv_file, tag)\n",
+    "        \n",
+    "        print(f\"\\n=== Tag Analysis for '{result['tag']}' ===\")\n",
+    "        print(f\"Models with tag: {result['count']}\")\n",
+    "        print(f\"Total models: {result['total']}\")\n",
+    "        print(f\"Percentage: {result['percentage']:.2f}%\\n\")\n",
+    "    else:\n",
+    "        # If no specific tag provided, show all tags\n",
+    "        print(\"\\n=== All Tags Analysis ===\\n\")\n",
+    "        results_df = analyze_all_tags(csv_file)\n",
+    "        print(results_df.to_string(index=False))\n",
+    "        print(f\"\\nTotal unique tags: {len(results_df)}\")\n",
+    "        print(f\"Total models: {len(pd.read_csv(csv_file))}\\n\")\n",
+    "        \n",
+    "        # Show example usage\n",
+    "        print(\"\\nTo search for a specific tag, run:\")\n",
+    "        print(\"  python tag_percentage_calculator.py <tag_name>\")\n",
+    "        print(\"\\nExample:\")\n",
+    "        print(\"  python tag_percentage_calculator.py anime\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "63187f58-9777-4ffb-bdf9-93b191a60241",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T19:08:49.572641Z",
+     "iopub.status.busy": "2025-12-09T19:08:49.572453Z",
+     "iopub.status.idle": "2025-12-09T19:08:49.634561Z",
+     "shell.execute_reply": "2025-12-09T19:08:49.634109Z",
+     "shell.execute_reply.started": "2025-12-09T19:08:49.572627Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import json\n",
+    "from collections import defaultdict\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "\n",
+    "\n",
+    "def load_data(filepath):\n",
+    "    \"\"\"Load the JSON data from file.\"\"\"\n",
+    "    with open(filepath, 'r', encoding='utf-8') as f:\n",
+    "        return json.load(f)\n",
+    "\n",
+    "def calculate_cooccurrence_rate(data, target_tag, cooccurring_tags):\n",
+    "    \"\"\"\n",
+    "    Calculate what percentage of target_tag occurrences co-occur with each tag in cooccurring_tags.\n",
+    "    \n",
+    "    Args:\n",
+    "        data: Dictionary with 'nodes' and 'links'\n",
+    "        target_tag: The main tag to analyze (e.g., \"woman\")\n",
+    "        cooccurring_tags: List of tags to check co-occurrence with (e.g., [\"sexy\", \"pose\"])\n",
+    "    \n",
+    "    Returns:\n",
+    "        Dictionary with results\n",
+    "    \"\"\"\n",
+    "    # Find the target tag's total occurrences\n",
+    "    target_size = None\n",
+    "    for node in data['nodes']:\n",
+    "        if node['id'] == target_tag:\n",
+    "            target_size = node['size']\n",
+    "            break\n",
+    "    \n",
+    "    if target_size is None:\n",
+    "        print(f\"Warning: Tag '{target_tag}' not found in nodes!\")\n",
+    "        return None\n",
+    "    \n",
+    "    print(f\"\\n{'='*60}\")\n",
+    "    print(f\"Analysis for tag: '{target_tag}'\")\n",
+    "    print(f\"{'='*60}\")\n",
+    "    print(f\"Total occurrences of '{target_tag}': {target_size:,}\")\n",
+    "    print()\n",
+    "    \n",
+    "    # Find co-occurrences in links\n",
+    "    results = {}\n",
+    "    for cooccurring_tag in cooccurring_tags:\n",
+    "        cooccurrence_count = 0\n",
+    "        \n",
+    "        # Check both directions in links\n",
+    "        for link in data['links']:\n",
+    "            if (link['source'] == target_tag and link['target'] == cooccurring_tag) or \\\n",
+    "               (link['source'] == cooccurring_tag and link['target'] == target_tag):\n",
+    "                cooccurrence_count = link['value']\n",
+    "                break\n",
+    "        \n",
+    "        if cooccurrence_count > 0:\n",
+    "            percentage = (cooccurrence_count / target_size) * 100\n",
+    "            results[cooccurring_tag] = {\n",
+    "                'count': cooccurrence_count,\n",
+    "                'percentage': percentage\n",
+    "            }\n",
+    "            print(f\"Tag: '{cooccurring_tag}'\")\n",
+    "            print(f\"  Co-occurrences: {cooccurrence_count:,}\")\n",
+    "            print(f\"  Percentage: {percentage:.2f}%\")\n",
+    "            print(f\"  (i.e., {percentage:.2f}% of '{target_tag}' occurrences also have '{cooccurring_tag}')\")\n",
+    "        else:\n",
+    "            results[cooccurring_tag] = {\n",
+    "                'count': 0,\n",
+    "                'percentage': 0.0\n",
+    "            }\n",
+    "            print(f\"Tag: '{cooccurring_tag}'\")\n",
+    "            print(f\"  No co-occurrences found\")\n",
+    "        print()\n",
+    "    \n",
+    "    # Calculate combined co-occurrence (both tags together)\n",
+    "    print(f\"\\n{'='*60}\")\n",
+    "    print(\"Combined Analysis\")\n",
+    "    print(f\"{'='*60}\")\n",
+    "    \n",
+    "    # To find items with ALL tags, we'd need to look at the underlying data\n",
+    "    # With just the graph structure, we can only report individual co-occurrences\n",
+    "    print(f\"Individual co-occurrence rates calculated above.\")\n",
+    "    print(f\"Note: To calculate how often ALL tags appear together,\")\n",
+    "    print(f\"we would need access to the raw item-level data.\")\n",
+    "    \n",
+    "    return results\n",
+    "\n",
+    "def main():\n",
+    "    # Load the data\n",
+    "    filepath = current_dir.parent / \"public/json/nodes_all.json\"\n",
+    "    print(\"Loading data...\")\n",
+    "    data = load_data(filepath)\n",
+    "    print(f\"Loaded {len(data['nodes']):,} nodes and {len(data['links']):,} links\")\n",
+    "    \n",
+    "    # Calculate co-occurrence rates\n",
+    "    target_tag = \"woman\"\n",
+    "    cooccurring_tags = [\"sexy\", \"pose\"]\n",
+    "    \n",
+    "    results = calculate_cooccurrence_rate(data, target_tag, cooccurring_tags)\n",
+    "    \n",
+    "    # Summary\n",
+    "    print(f\"\\n{'='*60}\")\n",
+    "    print(\"SUMMARY\")\n",
+    "    print(f\"{'='*60}\")\n",
+    "    if results:\n",
+    "        for tag, stats in results.items():\n",
+    "            print(f\"'{target_tag}' + '{tag}': {stats['percentage']:.2f}% ({stats['count']:,} occurrences)\")\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "id": "6e0e8c6b-547e-4899-b001-1d4c6b31476f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T19:48:36.009465Z",
+     "iopub.status.busy": "2025-12-09T19:48:36.009249Z",
+     "iopub.status.idle": "2025-12-09T19:48:36.063251Z",
+     "shell.execute_reply": "2025-12-09T19:48:36.062750Z",
+     "shell.execute_reply.started": "2025-12-09T19:48:36.009449Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading data...\n",
+      "Loaded 60,330 nodes and 16,921 links\n",
+      "\n",
+      "================================================================================\n",
+      "Top 100 Co-occurring Tags for: 'anime'\n",
+      "================================================================================\n",
+      "Total occurrences of 'anime': 74,187\n",
+      "\n",
+      "Rank   Tag                            Count        Percentage  \n",
+      "------ ------------------------------ ------------ ------------\n",
+      "1      character                      53,792            72.51%\n",
+      "2      woman                          30,731            41.42%\n",
+      "3      girls                          21,434            28.89%\n",
+      "4      female                         14,286            19.26%\n",
+      "5      style                          11,593            15.63%\n",
+      "6      game character                 9,309             12.55%\n",
+      "7      sexy                           8,876             11.96%\n",
+      "8      male                           4,476              6.03%\n",
+      "9      video game                     3,411              4.60%\n",
+      "10     man                            3,333              4.49%\n",
+      "11     lora                           3,260              4.39%\n",
+      "12     concept                        2,568              3.46%\n",
+      "13     girl                           2,472              3.33%\n",
+      "14     base model                     2,197              2.96%\n",
+      "15     photorealistic                 2,191              2.95%\n",
+      "16     manga                          2,015              2.72%\n",
+      "17     boys                           2,006              2.70%\n",
+      "18     anime character                1,841              2.48%\n",
+      "19     cartoon                        1,759              2.37%\n",
+      "20     game                           1,536              2.07%\n",
+      "21     men                            1,431              1.93%\n",
+      "22     cute                           1,325              1.79%\n",
+      "23     hentai                         1,222              1.65%\n",
+      "24     clothing                       1,131              1.52%\n",
+      "25     furry                          1,114              1.50%\n",
+      "26     realistic                      1,074              1.45%\n",
+      "27     styles                         1,057              1.42%\n",
+      "28     illustration                   971                1.31%\n",
+      "29     characters                     967                1.30%\n",
+      "30     art style                      949                1.28%\n",
+      "31     pokemon                        932                1.26%\n",
+      "32     vtuber                         890                1.20%\n",
+      "33     person                         872                1.18%\n",
+      "34     artstyle                       801                1.08%\n",
+      "35     anime girl                     756                1.02%\n",
+      "36     blue archive                   684                0.92%\n",
+      "37     2d                             666                0.90%\n",
+      "38     fantasy                        643                0.87%\n",
+      "39     art                            633                0.85%\n",
+      "40     poses                          619                0.83%\n",
+      "41     3d                             578                0.78%\n",
+      "42     nsfw                           550                0.74%\n",
+      "43     artist                         544                0.73%\n",
+      "44     genshin impact                 519                0.70%\n",
+      "45     idolmaster                     483                0.65%\n",
+      "46     fire emblem                    481                0.65%\n",
+      "47     fate                           432                0.58%\n",
+      "48     waifu                          406                0.55%\n",
+      "49     azur lane                      399                0.54%\n",
+      "50     dragon ball                    388                0.52%\n",
+      "51     ponyxl                         379                0.51%\n",
+      "52     naruto                         379                0.51%\n",
+      "53     precure                        379                0.51%\n",
+      "54     videogame                      372                0.50%\n",
+      "55     retro                          356                0.48%\n",
+      "56     meme                           355                0.48%\n",
+      "57     arknights                      354                0.48%\n",
+      "58     hololive                       347                0.47%\n",
+      "59     virtual youtuber               347                0.47%\n",
+      "60     umamusume                      333                0.45%\n",
+      "61     falcom                         326                0.44%\n",
+      "62     one piece                      325                0.44%\n",
+      "63     boy                            321                0.43%\n",
+      "64     chibi                          315                0.42%\n",
+      "65     comics                         303                0.41%\n",
+      "66     idolm@ster                     295                0.40%\n",
+      "67     gundam                         294                0.40%\n",
+      "68     bleach                         294                0.40%\n",
+      "69     pose                           288                0.39%\n",
+      "70     guy                            284                0.38%\n",
+      "71     milf                           281                0.38%\n",
+      "72     my hero academia               279                0.38%\n",
+      "73     genshin                        276                0.37%\n",
+      "74     porn                           268                0.36%\n",
+      "75     kawaii                         257                0.35%\n",
+      "76     kantai collection              255                0.34%\n",
+      "77     galgame                        250                0.34%\n",
+      "78     eiyuu densetsu                 246                0.33%\n",
+      "79     animals                        239                0.32%\n",
+      "80     yu-gi-oh!                      236                0.32%\n",
+      "81     comic                          232                0.31%\n",
+      "82     sex                            228                0.31%\n",
+      "83     cinderella girls               227                0.31%\n",
+      "84     kancolle                       223                0.30%\n",
+      "85     huge breasts                   220                0.30%\n",
+      "86     clothes                        219                0.30%\n",
+      "87     digital art                    217                0.29%\n",
+      "88     oc                             216                0.29%\n",
+      "89     scenery                        215                0.29%\n",
+      "90     nintendo                       215                0.29%\n",
+      "91     manhwa                         214                0.29%\n",
+      "92     final fantasy                  211                0.28%\n",
+      "93     nikke                          211                0.28%\n",
+      "94     cosplay                        208                0.28%\n",
+      "95     beautiful                      207                0.28%\n",
+      "96     dragon ball z                  207                0.28%\n",
+      "97     concepts                       207                0.28%\n",
+      "98     videogame character            207                0.28%\n",
+      "99     thick thighs                   205                0.28%\n",
+      "100    wide hips                      202                0.27%\n",
+      "\n",
+      "================================================================================\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import json\n",
+    "from collections import defaultdict\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "def load_data(filepath):\n",
+    "    \"\"\"Load the JSON data from file.\"\"\"\n",
+    "    with open(filepath, 'r', encoding='utf-8') as f:\n",
+    "        return json.load(f)\n",
+    "\n",
+    "def get_top_cooccurrences(data, target_tag, top_n=10):\n",
+    "    \"\"\"\n",
+    "    Find the top N tags that co-occur with the target tag.\n",
+    "    \n",
+    "    Args:\n",
+    "        data: Dictionary with 'nodes' and 'links'\n",
+    "        target_tag: The main tag to analyze (e.g., \"woman\")\n",
+    "        top_n: Number of top co-occurring tags to return (default: 10)\n",
+    "    \n",
+    "    Returns:\n",
+    "        List of tuples (tag, count, percentage) sorted by count\n",
+    "    \"\"\"\n",
+    "    # Find the target tag's total occurrences\n",
+    "    target_size = None\n",
+    "    for node in data['nodes']:\n",
+    "        if node['id'] == target_tag:\n",
+    "            target_size = node['size']\n",
+    "            break\n",
+    "    \n",
+    "    if target_size is None:\n",
+    "        print(f\"Error: Tag '{target_tag}' not found in nodes!\")\n",
+    "        return None, None\n",
+    "    \n",
+    "    # Find all co-occurrences in links\n",
+    "    cooccurrences = []\n",
+    "    \n",
+    "    for link in data['links']:\n",
+    "        if link['source'] == target_tag:\n",
+    "            cooccurrences.append({\n",
+    "                'tag': link['target'],\n",
+    "                'count': link['value']\n",
+    "            })\n",
+    "        elif link['target'] == target_tag:\n",
+    "            cooccurrences.append({\n",
+    "                'tag': link['source'],\n",
+    "                'count': link['value']\n",
+    "            })\n",
+    "    \n",
+    "    # Sort by count (descending) and take top N\n",
+    "    cooccurrences.sort(key=lambda x: x['count'], reverse=True)\n",
+    "    top_cooccurrences = cooccurrences[:top_n]\n",
+    "    \n",
+    "    # Calculate percentages\n",
+    "    results = []\n",
+    "    for item in top_cooccurrences:\n",
+    "        percentage = (item['count'] / target_size) * 100\n",
+    "        results.append((item['tag'], item['count'], percentage))\n",
+    "    \n",
+    "    return results, target_size\n",
+    "\n",
+    "def display_results(target_tag, results, target_size, top_n):\n",
+    "    \"\"\"Display the results in a formatted table.\"\"\"\n",
+    "    if results is None:\n",
+    "        return\n",
+    "    \n",
+    "    print(f\"\\n{'='*80}\")\n",
+    "    print(f\"Top {top_n} Co-occurring Tags for: '{target_tag}'\")\n",
+    "    print(f\"{'='*80}\")\n",
+    "    print(f\"Total occurrences of '{target_tag}': {target_size:,}\\n\")\n",
+    "    \n",
+    "    if not results:\n",
+    "        print(f\"No co-occurrences found for '{target_tag}'\")\n",
+    "        return\n",
+    "    \n",
+    "    # Print header\n",
+    "    print(f\"{'Rank':<6} {'Tag':<30} {'Count':<12} {'Percentage':<12}\")\n",
+    "    print(f\"{'-'*6} {'-'*30} {'-'*12} {'-'*12}\")\n",
+    "    \n",
+    "    # Print results\n",
+    "    for i, (tag, count, percentage) in enumerate(results, 1):\n",
+    "        print(f\"{i:<6} {tag:<30} {count:<12,} {percentage:>10.2f}%\")\n",
+    "    \n",
+    "    print(f\"\\n{'='*80}\\n\")\n",
+    "\n",
+    "def main():\n",
+    "    # Load the data\n",
+    "    filepath = current_dir.parent / \"public/json/nodes_all.json\"\n",
+    "    print(\"Loading data...\")\n",
+    "    data = load_data(filepath)\n",
+    "    print(f\"Loaded {len(data['nodes']):,} nodes and {len(data['links']):,} links\")\n",
+    "    \n",
+    "    # Analyze different tags\n",
+    "    target_tags = [\"anime\"]  # Add more tags here to analyze multiple\n",
+    "    top_n = 100\n",
+    "    \n",
+    "    for target_tag in target_tags:\n",
+    "        results, target_size = get_top_cooccurrences(data, target_tag, top_n)\n",
+    "        display_results(target_tag, results, target_size, top_n)\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "id": "e8af35af-a4b9-4011-b8c3-5ec8e75ce6c1",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T19:47:44.592679Z",
+     "iopub.status.busy": "2025-12-09T19:47:44.592464Z",
+     "iopub.status.idle": "2025-12-09T19:47:44.649077Z",
+     "shell.execute_reply": "2025-12-09T19:47:44.648523Z",
+     "shell.execute_reply.started": "2025-12-09T19:47:44.592664Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading data...\n",
+      "Loaded 60,330 nodes and 16,921 links\n",
+      "\n",
+      "\n",
+      "================================================================================\n",
+      "Co-occurrence Analysis: 'anime' + 'dragon ball'\n",
+      "================================================================================\n",
+      "\n",
+      "Total occurrences of 'anime': 74,187\n",
+      "Total occurrences of 'dragon ball': 479\n",
+      "\n",
+      "Items with BOTH tags: 388\n",
+      "\n",
+      ">>> 0.52% of 'anime' occurrences also have 'dragon ball'\n",
+      "\n",
+      "================================================================================\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "import json\n",
+    "from collections import defaultdict\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "def load_data(filepath):\n",
+    "    \"\"\"Load the JSON data from file.\"\"\"\n",
+    "    with open(filepath, 'r', encoding='utf-8') as f:\n",
+    "        return json.load(f)\n",
+    "\n",
+    "def get_tag_cooccurrence(data, tag1, tag2):\n",
+    "    \"\"\"\n",
+    "    Find what percentage of tag1 occurrences also have tag2.\n",
+    "    \n",
+    "    Args:\n",
+    "        data: Dictionary with 'nodes' and 'links'\n",
+    "        tag1: Primary tag to analyze (e.g., \"cat\")\n",
+    "        tag2: Secondary tag to check for (e.g., \"dog\")\n",
+    "    \n",
+    "    Returns:\n",
+    "        Dictionary with co-occurrence information\n",
+    "    \"\"\"\n",
+    "    # Find the tags' total occurrences\n",
+    "    tag1_size = None\n",
+    "    tag2_size = None\n",
+    "    \n",
+    "    for node in data['nodes']:\n",
+    "        if node['id'] == tag1:\n",
+    "            tag1_size = node['size']\n",
+    "        if node['id'] == tag2:\n",
+    "            tag2_size = node['size']\n",
+    "    \n",
+    "    if tag1_size is None:\n",
+    "        print(f\"Error: Tag '{tag1}' not found in nodes!\")\n",
+    "        return None\n",
+    "    \n",
+    "    if tag2_size is None:\n",
+    "        print(f\"Error: Tag '{tag2}' not found in nodes!\")\n",
+    "        return None\n",
+    "    \n",
+    "    # Find co-occurrence count in links\n",
+    "    # This represents how many items have BOTH tag1 AND tag2\n",
+    "    cooccurrence_count = 0\n",
+    "    \n",
+    "    for link in data['links']:\n",
+    "        if (link['source'] == tag1 and link['target'] == tag2) or \\\n",
+    "           (link['source'] == tag2 and link['target'] == tag1):\n",
+    "            cooccurrence_count = link['value']\n",
+    "            break\n",
+    "    \n",
+    "    # Calculate percentage: what % of tag1 items also have tag2\n",
+    "    percentage_with_tag2 = (cooccurrence_count / tag1_size) * 100 if tag1_size > 0 else 0\n",
+    "    \n",
+    "    return {\n",
+    "        'primary_tag': tag1,\n",
+    "        'secondary_tag': tag2,\n",
+    "        'primary_tag_total': tag1_size,\n",
+    "        'secondary_tag_total': tag2_size,\n",
+    "        'cooccurrence_count': cooccurrence_count,\n",
+    "        'percentage_with_secondary': percentage_with_tag2\n",
+    "    }\n",
+    "\n",
+    "def display_cooccurrence_results(result):\n",
+    "    \"\"\"Display the co-occurrence results in a formatted way.\"\"\"\n",
+    "    if result is None:\n",
+    "        return\n",
+    "    \n",
+    "    print(f\"\\n{'='*80}\")\n",
+    "    print(f\"Co-occurrence Analysis: '{result['primary_tag']}' + '{result['secondary_tag']}'\")\n",
+    "    print(f\"{'='*80}\\n\")\n",
+    "    \n",
+    "    print(f\"Total occurrences of '{result['primary_tag']}': {result['primary_tag_total']:,}\")\n",
+    "    print(f\"Total occurrences of '{result['secondary_tag']}': {result['secondary_tag_total']:,}\")\n",
+    "    print(f\"\\nItems with BOTH tags: {result['cooccurrence_count']:,}\")\n",
+    "    print(f\"\\n>>> {result['percentage_with_secondary']:.2f}% of '{result['primary_tag']}' occurrences also have '{result['secondary_tag']}'\")\n",
+    "    \n",
+    "    print(f\"\\n{'='*80}\\n\")\n",
+    "\n",
+    "def analyze_multiple_pairs(data, tag_pairs):\n",
+    "    \"\"\"\n",
+    "    Analyze multiple tag pairs at once.\n",
+    "    \n",
+    "    Args:\n",
+    "        data: Dictionary with 'nodes' and 'links'\n",
+    "        tag_pairs: List of tuples, each containing two tags to compare\n",
+    "    \"\"\"\n",
+    "    results = []\n",
+    "    \n",
+    "    for tag1, tag2 in tag_pairs:\n",
+    "        result = get_tag_cooccurrence(data, tag1, tag2)\n",
+    "        if result:\n",
+    "            results.append(result)\n",
+    "            display_cooccurrence_results(result)\n",
+    "    \n",
+    "    return results\n",
+    "\n",
+    "def main():\n",
+    "    # Load the data\n",
+    "    filepath = current_dir.parent / \"public/json/nodes_all.json\"\n",
+    "    print(\"Loading data...\")\n",
+    "    data = load_data(filepath)\n",
+    "    print(f\"Loaded {len(data['nodes']):,} nodes and {len(data['links']):,} links\\n\")\n",
+    "    \n",
+    "    # Analyze: What percentage of \"cat\" occurrences also have \"dog\"?\n",
+    "    primary_tag = \"anime\"      # The main tag you're interested in\n",
+    "    secondary_tag = \"dragon ball\"    # The tag you want to check for\n",
+    "    \n",
+    "    result = get_tag_cooccurrence(data, primary_tag, secondary_tag)\n",
+    "    display_cooccurrence_results(result)\n",
+    "    \n",
+    "    # You can also check the reverse: What percentage of \"dog\" occurrences also have \"cat\"?\n",
+    "    # result_reverse = get_tag_cooccurrence(data, \"dog\", \"cat\")\n",
+    "    # display_cooccurrence_results(result_reverse)\n",
+    "    \n",
+    "    # Option: Analyze multiple pairs at once\n",
+    "    # Uncomment the lines below to analyze multiple pairs\n",
+    "    \"\"\"\n",
+    "    tag_pairs = [\n",
+    "        (\"cat\", \"dog\"),\n",
+    "        (\"boy\", \"anime\"),\n",
+    "        (\"girl\", \"anime\"),\n",
+    "        (\"man\", \"photorealistic\")\n",
+    "    ]\n",
+    "    results = analyze_multiple_pairs(data, tag_pairs)\n",
+    "    \"\"\"\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a79ee96c-a060-4cc5-82d8-5642ccbef328",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-3-2_top_10_most_popular_checkpoints.ipynb

@@ -0,0 +1,210 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "14f6b6e3-5edb-458a-9553-09616455ba9f",
+   "metadata": {},
+   "source": [
+    "# Section 6.3: Download Models"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64d0907d-780d-48a0-b2c4-a9de650a3760",
+   "metadata": {},
+   "source": [
+    "### Download models from CIVITAI to the respective folders to ~/ComfyUI/models\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "2f788cc1-d8ff-43b8-9051-a3d5d8d79086",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:25:06.875713Z",
+     "iopub.status.busy": "2025-02-06T18:25:06.875579Z",
+     "iopub.status.idle": "2025-02-06T18:25:07.380814Z",
+     "shell.execute_reply": "2025-02-06T18:25:07.380184Z",
+     "shell.execute_reply.started": "2025-02-06T18:25:06.875698Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import csv\n",
+    "import json\n",
+    "import time\n",
+    "import requests\n",
+    "from itertools import cycle\n",
+    "import pandas as pd\n",
+    "from requests.adapters import HTTPAdapter\n",
+    "from requests.packages.urllib3.util.retry import Retry\n",
+    "from pathlib import Path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "7b6349d2-b839-4b8e-895e-cc7eb3fba88f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:25:07.848340Z",
+     "iopub.status.busy": "2025-02-06T18:25:07.848183Z",
+     "iopub.status.idle": "2025-02-06T18:25:07.851420Z",
+     "shell.execute_reply": "2025-02-06T18:25:07.850952Z",
+     "shell.execute_reply.started": "2025-02-06T18:25:07.848325Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "api_karussell = current_dir.parent / 'misc/api_keys.txt'\n",
+    "target_dir = current_dir.parent / 'data/models/checkpoints/'  \n",
+    "target_dir.parent.mkdir(parents=True, exist_ok=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3c2a9244-5070-4159-821f-39bb92051daf",
+   "metadata": {},
+   "source": [
+    "## function definition:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "aeaaf1eb-9768-47ad-9298-7d7a8be59094",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:25:09.379460Z",
+     "iopub.status.busy": "2025-02-06T18:25:09.379303Z",
+     "iopub.status.idle": "2025-02-06T18:25:09.382034Z",
+     "shell.execute_reply": "2025-02-06T18:25:09.381557Z",
+     "shell.execute_reply.started": "2025-02-06T18:25:09.379446Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "csv_file_path = current_dir.parent / 'data/CSV/model_subsets/Civiverse_checkpoint_only.csv'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2cdc7051-af24-439c-b985-b5088bb13cc0",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-06T18:25:10.192171Z",
+     "iopub.status.busy": "2025-02-06T18:25:10.191997Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Downloaded: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/models/checkpoints/Pony/realDream_sdxlPony14.safetensors\n",
+      "Downloaded: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/models/checkpoints/Flux.1 D/acornIsSpinningFLUX_aisFluxV15.safetensors\n",
+      "Failed to download https://civitai.com/api/download/models/1376998: 401 Client Error: Unauthorized for url: https://civitai.com/api/download/models/1376998\n",
+      "Failed to download https://civitai.com/api/download/models/1379842: 401 Client Error: Unauthorized for url: https://civitai.com/api/download/models/1379842\n",
+      "Downloaded: /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/models/checkpoints/Illustrious/miaomiaoHarem_v15a.safetensors\n"
+     ]
+    }
+   ],
+   "source": [
+    "with open(api_karussell, 'r') as f:\n",
+    "    api_keys = [line.strip() for line in f if line.strip()]\n",
+    "api_key_cycle = cycle(api_keys)  # Rotate API keys\n",
+    "\n",
+    "# Load CSV file\n",
+    "\n",
+    "df = pd.read_csv(csv_file_path)\n",
+    "\n",
+    "\n",
+    "\n",
+    "def get_filename_from_response(response, url):\n",
+    "    if 'content-disposition' in response.headers:\n",
+    "        content_disposition = response.headers['content-disposition']\n",
+    "        filename = content_disposition.split(\"filename=\")[-1].strip(\"\\\"\")\n",
+    "    else:\n",
+    "        filename = url.split(\"/\")[-1]\n",
+    "    return filename\n",
+    "\n",
+    "def download_file(url, dest_folder, api_key):\n",
+    "    headers = {'Authorization': f'Bearer {api_key}'}  # If authentication is needed\n",
+    "    \n",
+    "    try:\n",
+    "        response = requests.get(url, headers=headers, stream=True)\n",
+    "        response.raise_for_status()\n",
+    "        \n",
+    "        filename = get_filename_from_response(response, url)\n",
+    "        dest_path = dest_folder / filename\n",
+    "        \n",
+    "        with open(dest_path, 'wb') as f:\n",
+    "            for chunk in response.iter_content(chunk_size=8192):\n",
+    "                f.write(chunk)\n",
+    "        \n",
+    "        print(f\"Downloaded: {dest_path}\")\n",
+    "    except requests.exceptions.RequestException as e:\n",
+    "        print(f\"Failed to download {url}: {e}\")\n",
+    "\n",
+    "def main():\n",
+    "    for _, row in df.iterrows():\n",
+    "        url = row['downloadUrl']\n",
+    "        base_model = row['baseModel']\n",
+    "        \n",
+    "        if pd.isna(url) or pd.isna(base_model):\n",
+    "            continue  # Skip missing values\n",
+    "        \n",
+    "        # Create model-specific folder\n",
+    "        model_folder = target_dir / base_model\n",
+    "        model_folder.mkdir(parents=True, exist_ok=True)\n",
+    "        \n",
+    "        # Rotate API keys\n",
+    "        api_key = next(api_key_cycle)\n",
+    "        \n",
+    "        # Download filfile\n",
+    "        download_file(url, model_folder, api_key)\n",
+    "        \n",
+    "        # Sleep to avoid rate limits\n",
+    "        time.sleep(2)  # Adjust based on API limits\n",
+    "\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c99e2fee-eef2-4530-b69a-9a3db3bda84f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-3-4_Figure_8_Step_1_LLM_annotation.ipynb

@@ -0,0 +1,1941 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "23d0ae58",
+   "metadata": {},
+   "source": [
+    "# Deepfake Adapter Dataset - LLM Annotation Pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e4407358",
+   "metadata": {},
+   "source": [
+    "### Unified Model Loading & Inference\n",
+    "Code for querying Mistral, Gemma, and Qwen models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1a1b9d0e",
+   "metadata": {},
+   "source": [
+    "## CLEANING & PREPROCESSING"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3df42c46",
+   "metadata": {},
+   "source": [
+    "#### Named Entity Recognitition (NER) using SpaCy "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a287eef4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "import emoji\n",
+    "import spacy\n",
+    "\n",
+    "# Load spaCy model\n",
+    "# You may need to download it first: python -m spacy download en_core_web_sm\n",
+    "try:\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model loaded: en_core_web_sm\")\n",
+    "except OSError:\n",
+    "    print(\"❌ spaCy model not found. Downloading...\")\n",
+    "    import subprocess\n",
+    "    subprocess.run([\"python\", \"-m\", \"spacy\", \"download\", \"en_core_web_sm\"])\n",
+    "    nlp = spacy.load(\"en_core_web_sm\")\n",
+    "    print(\"✅ spaCy model downloaded and loaded\")\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "#input_file = current_dir.parent / \"data/CSV/real_person_adapters.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter.csv\"\n",
+    "\n",
+    "# Load dataset\n",
+    "df = pd.read_csv(input_file)\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "\n",
+    "def translate_leetspeak(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Translate common leetspeak patterns to normal letters.\n",
+    "    Examples: 4kira -> Akira, 3mma -> Emma, 1rene -> Irene\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return text\n",
+    "    \n",
+    "    # Common leetspeak mappings (order matters!)\n",
+    "    leetspeak_map = {\n",
+    "        '4': 'a',\n",
+    "        '3': 'e', \n",
+    "        '1': 'i',\n",
+    "        '0': 'o',\n",
+    "        '7': 't',\n",
+    "        '5': 's',\n",
+    "        '8': 'b',\n",
+    "        '9': 'g',\n",
+    "        '@': 'a',\n",
+    "        '$': 's',\n",
+    "        '!': 'i',\n",
+    "    }\n",
+    "    \n",
+    "    result = text\n",
+    "    # Apply mappings at word boundaries or start of string\n",
+    "    for leet, normal in leetspeak_map.items():\n",
+    "        # Replace at start of word\n",
+    "        result = re.sub(rf'\\b{re.escape(leet)}', normal, result, flags=re.IGNORECASE)\n",
+    "        # Replace standalone numbers that look like letters in context\n",
+    "        result = re.sub(rf'(?<=[a-z]){re.escape(leet)}(?=[a-z])', normal, result, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    return result\n",
+    "\n",
+    "def preprocess_for_ner(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Preprocess the name before spaCy NER.\n",
+    "    Remove noise but keep the actual name parts.\n",
+    "    \"\"\"\n",
+    "    if pd.isna(name):\n",
+    "        return \"\"\n",
+    "    \n",
+    "    name = str(name)\n",
+    "    \n",
+    "    # FIRST: Translate leetspeak\n",
+    "    name = translate_leetspeak(name)\n",
+    "    \n",
+    "    # Remove emoji\n",
+    "    name = emoji.replace_emoji(name, replace=' ')\n",
+    "    \n",
+    "    # Remove version indicators (v1, v2, v1.0, etc.)\n",
+    "    name = re.sub(r'\\s*[vV]\\d+(\\.\\d+)?\\s*', ' ', name)\n",
+    "    \n",
+    "    # Remove LoRA-related terms (case insensitive)\n",
+    "    lora_terms = ['lora', 'loha', 'lycoris', 'controlnet', 'textual inversion', \n",
+    "                  'embedding', 'ti', 'checkpoint', 'model', 'adapter', 'pony', 'sdxl', 'flux', 'illustrious', 'sd14', 'sd14', 'sd2', 'sd3', 'diffusion', 'stable', 'hunyuan']\n",
+    "    for term in lora_terms:\n",
+    "        name = re.sub(rf'\\b{term}\\b', '', name, flags=re.IGNORECASE)\n",
+    "    \n",
+    "    # Remove content in parentheses or brackets (often metadata)\n",
+    "    name = re.sub(r'\\([^)]*\\)', '', name)\n",
+    "    name = re.sub(r'\\[[^\\]]*\\]', '', name)\n",
+    "    \n",
+    "    # Remove special characters like 「」\n",
+    "    name = re.sub(r'[「」『』【】〈〉《》]', '', name)\n",
+    "    \n",
+    "    # Handle pipe - keep first part\n",
+    "    if '|' in name:\n",
+    "        name = name.split('|')[0]\n",
+    "    \n",
+    "    # Handle forward slash - keep first part\n",
+    "    if '/' in name:\n",
+    "        name = name.split('/')[0]\n",
+    "    \n",
+    "    # Replace underscores with spaces\n",
+    "    name = name.replace('_', ' ')\n",
+    "    \n",
+    "    # Remove multiple spaces\n",
+    "    name = re.sub(r'\\s+', ' ', name)\n",
+    "    \n",
+    "    # Strip\n",
+    "    name = name.strip()\n",
+    "    \n",
+    "    return name\n",
+    "\n",
+    "def extract_person_name(text: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Use spaCy NER to extract person names from text.\n",
+    "    Falls back to cleaned text if no PERSON entity found.\n",
+    "    \"\"\"\n",
+    "    if not text:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Run spaCy NER\n",
+    "    doc = nlp(text)\n",
+    "    \n",
+    "    # Extract PERSON entities\n",
+    "    person_entities = [ent.text for ent in doc.ents if ent.label_ == \"PERSON\"]\n",
+    "    \n",
+    "    if person_entities:\n",
+    "        # Return the first (usually longest/best) person name\n",
+    "        return person_entities[0].strip()\n",
+    "    \n",
+    "    # If no PERSON entity found, try to extract capitalized words (likely names)\n",
+    "    # This helps with names spaCy might miss\n",
+    "    words = text.split()\n",
+    "    capitalized_words = [w for w in words if w and w[0].isupper() and len(w) > 1]\n",
+    "    \n",
+    "    if capitalized_words:\n",
+    "        # Join first few capitalized words (likely the name)\n",
+    "        return ' '.join(capitalized_words[:3]).strip()\n",
+    "    \n",
+    "    # Last resort: return cleaned text\n",
+    "    return text.strip()\n",
+    "\n",
+    "def clean_name_with_spacy(name: str) -> str:\n",
+    "    \"\"\"\n",
+    "    Complete name cleaning pipeline with spaCy NER.\n",
+    "    \n",
+    "    Pipeline:\n",
+    "    1. Translate leetspeak (4→a, 3→e, 1→i, etc.)\n",
+    "    2. Remove noise (emoji, version tags, LoRA terms)\n",
+    "    3. Use spaCy to extract PERSON entities\n",
+    "    4. Fallback to capitalized words or cleaned text\n",
+    "    \"\"\"\n",
+    "    # Step 1 & 2: Preprocess (leetspeak + noise removal)\n",
+    "    preprocessed = preprocess_for_ner(name)\n",
+    "    \n",
+    "    if not preprocessed:\n",
+    "        return \"\"\n",
+    "    \n",
+    "    # Step 3: Extract person name using spaCy NER\n",
+    "    person_name = extract_person_name(preprocessed)\n",
+    "    \n",
+    "    return person_name\n",
+    "\n",
+    "# Apply name cleaning with spaCy\n",
+    "print(\"\\n🔄 Processing names with spaCy NER...\")\n",
+    "df['real_name'] = df['name'].apply(clean_name_with_spacy)\n",
+    "\n",
+    "# Show examples with detailed comparison\n",
+    "print(\"\\n📊 Name cleaning examples (with spaCy NER):\")\n",
+    "print(\"=\" * 100)\n",
+    "print(f\"{'Original Name':<50} | {'Cleaned Name':<30}\")\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "examples = df[['name', 'real_name']].head(30)\n",
+    "shown = 0\n",
+    "for idx, row in examples.iterrows():\n",
+    "    if row['name'] != row['real_name'] and shown < 20:\n",
+    "        print(f\"{row['name']:<50} | {row['real_name']:<30}\")\n",
+    "        shown += 1\n",
+    "\n",
+    "print(\"=\" * 100)\n",
+    "\n",
+    "# Show specific test cases\n",
+    "print(\"\\n🧪 Leetspeak translation examples:\")\n",
+    "test_names = ['4kira LoRA', '3mma Watson v2', '1rene LORA', 'L3vi Ackerman']\n",
+    "for test in test_names:\n",
+    "    result = clean_name_with_spacy(test)\n",
+    "    print(f\"  {test:<30} -> {result}\")\n",
+    "\n",
+    "# Statistics\n",
+    "print(f\"\\n📈 Statistics:\")\n",
+    "print(f\"  Total rows: {len(df)}\")\n",
+    "print(f\"  Non-empty names: {(df['real_name'] != '').sum()}\")\n",
+    "print(f\"  Empty names: {(df['real_name'] == '').sum()}\")\n",
+    "\n",
+    "# Show some examples of what spaCy identified\n",
+    "print(\"\\n🎯 Sample spaCy NER results:\")\n",
+    "sample_names = df['real_name'].head(20).tolist()\n",
+    "for i, name in enumerate(sample_names[:10], 1):\n",
+    "    if name:\n",
+    "        print(f\"  {i}. {name}\")\n",
+    "\n",
+    "print(f\"\\n✅ Cleaned {len(df)} names using spaCy NER\")\n",
+    "\n",
+    "# Save intermediate result\n",
+    "output_step1 = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "df.to_csv(output_step1, index=False)\n",
+    "print(f\"💾 Saved to {output_step1}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "64687c72",
+   "metadata": {},
+   "source": [
+    "#### STEP 02: Nationality tag to Country hint\n",
+    "here tags related to nationality gets converted to the country equivalent."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d6eaef5b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Set up paths\n",
+    "current_dir = Path.cwd()\n",
+    "countries_file = current_dir.parent / \"misc/lists/countries.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_01_NER.csv\"\n",
+    "output_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "\n",
+    "# Load datasets\n",
+    "poi_df = pd.read_csv(input_file)\n",
+    "countries_df = pd.read_csv(countries_file)\n",
+    "professions_df = pd.read_csv(professions_file)\n",
+    "\n",
+    "# Define uninhabited or non-relevant territories to exclude\n",
+    "excluded_territories = {\n",
+    "    'isle of man', 'bouvet island', 'heard island and mcdonald islands',\n",
+    "    'french southern territories', 'south georgia and the south sandwich islands',\n",
+    "    'svalbard and jan mayen', 'british indian ocean territory', 'antarctica',\n",
+    "    'christmas island', 'cocos (keeling) islands', 'norfolk island',\n",
+    "    'pitcairn', 'tokelau', 'united states minor outlying islands',\n",
+    "    'wallis and futuna', 'western sahara'\n",
+    "}\n",
+    "\n",
+    "# Step 1: Combine tags into one lowercase list\n",
+    "def combine_tags(row):\n",
+    "    return [str(row[f\"tag_{i}\"]).strip().lower() for i in range(1, 8) if pd.notna(row.get(f\"tag_{i}\"))]\n",
+    "\n",
+    "poi_df[\"tags\"] = poi_df.apply(combine_tags, axis=1)\n",
+    "\n",
+    "# Step 2: Build tag → (country, nationality) mapping with PRIORITIES\n",
+    "tag_to_country_nationality = {}\n",
+    "# We'll use a priority score: direct country name = 3, nationality = 2, word parts = 1\n",
+    "\n",
+    "for _, row in countries_df.iterrows():\n",
+    "    country = str(row[\"en_short_name\"]).strip()\n",
+    "    nationality = str(row[\"nationality\"]).strip()\n",
+    "    \n",
+    "    # Skip excluded territories\n",
+    "    if country.lower() in excluded_territories:\n",
+    "        continue\n",
+    "\n",
+    "    country_lc = country.lower()\n",
+    "    nationality_lc = nationality.lower()\n",
+    "\n",
+    "    # Store as (country, nationality, priority)\n",
+    "    # Exact country name match = highest priority\n",
+    "    if country_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_lc] = (country, \"\", 3)\n",
+    "    \n",
+    "    # Exact nationality match = medium priority  \n",
+    "    if nationality_lc not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_lc] = (\"\", nationality, 2)\n",
+    "    \n",
+    "    # No-space versions\n",
+    "    country_no_space = country_lc.replace(\" \", \"\")\n",
+    "    nationality_no_space = nationality_lc.replace(\" \", \"\")\n",
+    "    \n",
+    "    if country_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[country_no_space] = (country, \"\", 3)\n",
+    "    if nationality_no_space not in tag_to_country_nationality:\n",
+    "        tag_to_country_nationality[nationality_no_space] = (\"\", nationality, 2)\n",
+    "\n",
+    "    # Word parts = lowest priority (only for longer words to avoid false matches)\n",
+    "    for part in country_lc.split():\n",
+    "        if len(part) > 4:  # Only words longer than 4 chars\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (country, \"\", 1)\n",
+    "    for part in nationality_lc.split():\n",
+    "        if len(part) > 4:\n",
+    "            if part not in tag_to_country_nationality:\n",
+    "                tag_to_country_nationality[part] = (\"\", nationality, 1)\n",
+    "\n",
+    "print(f\"Built country/nationality mapping with {len(tag_to_country_nationality)} entries\")\n",
+    "\n",
+    "# Step 3: Infer likely_country and likely_nationality by checking ALL tags\n",
+    "def infer_country_and_nationality(tags):\n",
+    "    \"\"\"\n",
+    "    Check ALL tags and return the best match based on priority.\n",
+    "    Priority: exact country name > nationality > word parts\n",
+    "    \"\"\"\n",
+    "    best_match = None\n",
+    "    best_priority = 0\n",
+    "    \n",
+    "    for tag in tags:\n",
+    "        # Try cleaned version (no spaces)\n",
+    "        cleaned = tag.replace(\" \", \"\").lower()\n",
+    "        \n",
+    "        # Check cleaned version\n",
+    "        if cleaned in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[cleaned]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "        \n",
+    "        # Also check original tag\n",
+    "        if tag in tag_to_country_nationality:\n",
+    "            country, nationality, priority = tag_to_country_nationality[tag]\n",
+    "            if priority > best_priority and country and country.lower() not in excluded_territories:\n",
+    "                best_match = (country, nationality)\n",
+    "                best_priority = priority\n",
+    "    \n",
+    "    if best_match:\n",
+    "        return pd.Series(best_match)\n",
+    "    return pd.Series([\"\", \"\"])\n",
+    "\n",
+    "poi_df[[\"likely_country\", \"likely_nationality\"]] = poi_df[\"tags\"].apply(infer_country_and_nationality)\n",
+    "\n",
+    "# Step 4: Build tag → profession mapping\n",
+    "profession_alias_map = {}\n",
+    "\n",
+    "for _, row in professions_df.iterrows():\n",
+    "    canonical = str(row['profession']).strip().lower()\n",
+    "    profession_alias_map[canonical] = canonical\n",
+    "    for alias_col in ['alias_1', 'alias_2', 'alias_3']:\n",
+    "        alias = row.get(alias_col)\n",
+    "        if pd.notna(alias):\n",
+    "            profession_alias_map[str(alias).strip().lower()] = canonical\n",
+    "\n",
+    "# Step 5: Infer likely profession from tags\n",
+    "def infer_profession_from_tags(tags):\n",
+    "    matched = []\n",
+    "    for tag in tags:\n",
+    "        cleaned = tag.strip().lower()\n",
+    "        if cleaned in profession_alias_map:\n",
+    "            matched.append(profession_alias_map[cleaned])\n",
+    "\n",
+    "    if not matched:\n",
+    "        return \"\"\n",
+    "    if \"celebrity\" in matched and len(set(matched)) > 1:\n",
+    "        # Drop 'celebrity' if other professions are present\n",
+    "        matched = [m for m in matched if m != \"celebrity\"]\n",
+    "\n",
+    "    return matched[0]  # Return the first specific match\n",
+    "\n",
+    "\n",
+    "poi_df[\"likely_profession\"] = poi_df[\"tags\"].apply(infer_profession_from_tags)\n",
+    "\n",
+    "# Step 6: Save enriched dataset\n",
+    "poi_df.to_csv(output_file, index=False)\n",
+    "\n",
+    "# Preview results\n",
+    "print(f\"\\nProcessed {len(poi_df)} rows\")\n",
+    "print(f\"Rows with country: {(poi_df['likely_country'] != '').sum()}\")\n",
+    "print(f\"Rows with nationality: {(poi_df['likely_nationality'] != '').sum()}\")\n",
+    "print(f\"Rows with profession: {(poi_df['likely_profession'] != '').sum()}\")\n",
+    "\n",
+    "print(f\"\\nTop 10 countries:\")\n",
+    "print(poi_df[poi_df['likely_country'] != '']['likely_country'].value_counts().head(10))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4a4a58b3",
+   "metadata": {},
+   "source": [
+    "## LLM ANNOTATION"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b298844d",
+   "metadata": {},
+   "source": [
+    "#### Model Configurations"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "39f3d65e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "import signal\n",
+    "from contextlib import contextmanager\n",
+    "\n",
+    "# Configuration\n",
+    "current_dir = Path.cwd()\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Model configurations\n",
+    "MODEL_CONFIGS = {\n",
+    "    'mistral': {\n",
+    "        'name': 'mistralai/Mistral-7B-Instruct-v0.3',\n",
+    "        'dtype': torch.bfloat16,\n",
+    "        'quantization': None,\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.05,\n",
+    "            'do_sample': True,\n",
+    "            'top_p': 1.0,\n",
+    "        }\n",
+    "    },\n",
+    "    'gemma': {\n",
+    "        'name': 'google/gemma-3-27b-it',\n",
+    "        'dtype': torch.bfloat16,\n",
+    "        'quantization': None,\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.1,\n",
+    "            'do_sample': True,\n",
+    "            'top_p': 1.0,\n",
+    "        }\n",
+    "    },\n",
+    "    'qwen': {\n",
+    "        'name': 'Qwen/Qwen2.5-32B-Instruct',\n",
+    "        'dtype': None,  # Will use quantization\n",
+    "        'quantization': BitsAndBytesConfig(\n",
+    "            load_in_8bit=True,\n",
+    "            llm_int8_threshold=6.0,\n",
+    "            llm_int8_has_fp16_weight=False\n",
+    "        ),\n",
+    "        'generation_params': {\n",
+    "            'max_new_tokens': 512,\n",
+    "            'temperature': 0.1,\n",
+    "            'do_sample': False,\n",
+    "        }\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c215b38c",
+   "metadata": {},
+   "source": [
+    "#### Load Model Function"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfb5b13e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def load_model(model_type='mistral'):\n",
+    "    \"\"\"\n",
+    "    Load model and tokenizer based on type.\n",
+    "    \n",
+    "    Args:\n",
+    "        model_type: 'mistral', 'gemma', or 'qwen'\n",
+    "    \n",
+    "    Returns:\n",
+    "        tuple: (model, tokenizer, config)\n",
+    "    \"\"\"\n",
+    "    if model_type not in MODEL_CONFIGS:\n",
+    "        raise ValueError(f\"Unknown model type: {model_type}. Choose from {list(MODEL_CONFIGS.keys())}\")\n",
+    "    \n",
+    "    config = MODEL_CONFIGS[model_type]\n",
+    "    model_name = config['name']\n",
+    "    \n",
+    "    device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "    print(f\"Loading model: {model_name}\")\n",
+    "    print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "    print(f\"Device: {device}\\n\")\n",
+    "    \n",
+    "    if device == \"cpu\":\n",
+    "        print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    \n",
+    "    # Load tokenizer\n",
+    "    try:\n",
+    "        tokenizer = AutoTokenizer.from_pretrained(\n",
+    "            model_name,\n",
+    "            cache_dir=str(CACHE_DIR),\n",
+    "            use_fast=True\n",
+    "        )\n",
+    "    except:\n",
+    "        tokenizer = AutoTokenizer.from_pretrained(\n",
+    "            model_name,\n",
+    "            cache_dir=str(CACHE_DIR),\n",
+    "            use_fast=False\n",
+    "        )\n",
+    "    \n",
+    "    if tokenizer.pad_token is None:\n",
+    "        tokenizer.pad_token = tokenizer.eos_token\n",
+    "    \n",
+    "    # Load model\n",
+    "    model_kwargs = {\n",
+    "        'cache_dir': str(CACHE_DIR),\n",
+    "        'device_map': 'auto',\n",
+    "        'trust_remote_code': False\n",
+    "    }\n",
+    "    \n",
+    "    if config['quantization']:\n",
+    "        model_kwargs['quantization_config'] = config['quantization']\n",
+    "    else:\n",
+    "        model_kwargs['torch_dtype'] = config['dtype']\n",
+    "    \n",
+    "    model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)\n",
+    "    model.eval()\n",
+    "    \n",
+    "    # Check VRAM\n",
+    "    if torch.cuda.is_available():\n",
+    "        vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "        print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "    \n",
+    "    return model, tokenizer, config\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "11b2221a",
+   "metadata": {},
+   "source": [
+    "#### Inference Code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "229f96bd",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "@contextmanager\n",
+    "def timeout(duration):\n",
+    "    \"\"\"Context manager for timeout.\"\"\"\n",
+    "    def handler(signum, frame):\n",
+    "        raise TimeoutError(\"Operation timed out\")\n",
+    "    \n",
+    "    signal.signal(signal.SIGALRM, handler)\n",
+    "    signal.alarm(duration)\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        signal.alarm(0)\n",
+    "\n",
+    "def query_model(prompt, model, tokenizer, config, use_timeout=False):\n",
+    "    \"\"\"\n",
+    "    Query model with given prompt.\n",
+    "    \n",
+    "    Args:\n",
+    "        prompt: Input prompt string\n",
+    "        model: Loaded model\n",
+    "        tokenizer: Loaded tokenizer\n",
+    "        config: Model configuration dict\n",
+    "        use_timeout: Whether to use 60s timeout (for Qwen)\n",
+    "    \n",
+    "    Returns:\n",
+    "        str: Model response or None on error\n",
+    "    \"\"\"\n",
+    "    try:\n",
+    "        device = next(model.parameters()).device\n",
+    "        \n",
+    "        # Format as chat message\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are a data extraction assistant. Respond with exactly 5 numbered lines containing ONLY values. No labels, no explanations, no prefixes. Follow the format precisely.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generation parameters\n",
+    "        gen_kwargs = config['generation_params'].copy()\n",
+    "        gen_kwargs['pad_token_id'] = tokenizer.eos_token_id\n",
+    "        \n",
+    "        # Generate\n",
+    "        generation_fn = lambda: model.generate(**inputs, **gen_kwargs)\n",
+    "        \n",
+    "        if use_timeout:\n",
+    "            with timeout(60):\n",
+    "                with torch.no_grad():\n",
+    "                    outputs = generation_fn()\n",
+    "        else:\n",
+    "            with torch.no_grad():\n",
+    "                outputs = generation_fn()\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except TimeoutError:\n",
+    "        print(f\"[ERROR] Generation timed out after 60 seconds\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"[ERROR] Generation failed: {e}\")\n",
+    "        return None\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "88f005f8",
+   "metadata": {},
+   "source": [
+    "#### Prompt creation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "dfe05463",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def create_prompt(row):\n",
+    "    \"\"\"Create annotation prompt from row data.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if needed\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Extract information about '{name}' ({hint_text}).\n",
+    "\n",
+    "Respond with EXACTLY 5 numbered lines. Each line must contain ONLY the value, no labels or extra text.\n",
+    "\n",
+    "FORMAT REQUIREMENTS:\n",
+    "1. Full legal name in Western order (first last). VALUE ONLY.\n",
+    "2. Stage names/aliases, comma-separated. If none, write \"None\". VALUE ONLY.\n",
+    "3. Gender: MUST be exactly one word: Male, Female, Other, or Unknown. VALUE ONLY.\n",
+    "4. Professions: Choose up to 3 from this list ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality. Comma-separated. VALUE ONLY.\n",
+    "5. Primary country: Country name only (e.g., \"China\", \"United States\", \"Colombia\"). VALUE ONLY.\n",
+    "\n",
+    "RULES:\n",
+    "- Professions MUST match the exact categories listed (actress = actor)\n",
+    "- \"online personality\" includes streamers, cosplayers, YouTubers, influencers\n",
+    "- \"public figure\" includes politicians, activists, journalists, authors\n",
+    "- Use \"Unknown\" when uncertain or for fictional characters\n",
+    "- NO explanatory text, NO labels like \"Gender:\", NO prefixes\n",
+    "- For multi-role people, list up to 3 categories by relevance\n",
+    "\n",
+    "EXAMPLE FORMAT:\n",
+    "1. Taylor Swift\n",
+    "2. None\n",
+    "3. Female\n",
+    "4. singer/musician, public figure\n",
+    "5. United States\"\"\"\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "854fa668",
+   "metadata": {},
+   "source": [
+    "#### Response parsing code"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a4be2ee",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def parse_response(response):\n",
+    "    \"\"\"Parse model response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            gender_raw = line[2:].strip()\n",
+    "            gender_raw = re.sub(r'^(Gender:|gender:)\\s*', '', gender_raw, flags=re.IGNORECASE)\n",
+    "            gender_match = re.search(r'\\b(Male|Female|Other|Unknown)\\b', gender_raw, re.IGNORECASE)\n",
+    "            fields['gender'] = gender_match.group(1).capitalize() if gender_match else gender_raw\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            country_raw = line[2:].strip()\n",
+    "            country_raw = re.sub(r'^(Primary\\s+)?(associated\\s+)?country:\\s*', '', country_raw, flags=re.IGNORECASE)\n",
+    "            fields['country'] = country_raw\n",
+    "    \n",
+    "    return fields\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7e2f7a86",
+   "metadata": {},
+   "source": [
+    "#### CSV annotation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5f3dd5d6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def annotate_dataset(model_type='mistral', test_mode=False, test_size=100, max_rows=50862, save_interval=10):\n",
+    "    \"\"\"\n",
+    "    Annotate dataset using specified model.\n",
+    "    \n",
+    "    Args:\n",
+    "        model_type: 'mistral', 'gemma', or 'qwen'\n",
+    "        test_mode: If True, only process test_size rows\n",
+    "        test_size: Number of rows to process in test mode\n",
+    "        max_rows: Maximum rows to process\n",
+    "        save_interval: Save progress every N rows\n",
+    "    \"\"\"\n",
+    "    # Setup paths\n",
+    "    input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "    output_file = current_dir.parent / f\"data/CSV/{model_type}_local_annotated_POI{'_test' if test_mode else ''}.csv\"\n",
+    "    index_file = current_dir.parent / f\"misc/query_indicies/{model_type}_local_query_index.txt\"\n",
+    "    index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "    \n",
+    "    # Load model\n",
+    "    model, tokenizer, config = load_model(model_type)\n",
+    "    \n",
+    "    # Load data\n",
+    "    print(f\"Loaded {len(df)} rows from input file\")\n",
+    "    df = pd.read_csv(input_file)\n",
+    "    \n",
+    "    # Merge existing annotations if available\n",
+    "    if output_file.exists():\n",
+    "        existing_df = pd.read_csv(output_file)\n",
+    "        annotation_cols = ['full_name', 'aliases', 'gender', 'profession_llm', 'country']\n",
+    "        for col in annotation_cols:\n",
+    "            if col in existing_df.columns:\n",
+    "                df[col] = existing_df[col][:len(df)]\n",
+    "    \n",
+    "    # Apply limits\n",
+    "    if test_mode:\n",
+    "        df = df.head(test_size).copy()\n",
+    "    elif max_rows:\n",
+    "        df = df.head(max_rows).copy()\n",
+    "    \n",
+    "    # Create prompts\n",
+    "    df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "    \n",
+    "    # Load progress index\n",
+    "    current_index = 0\n",
+    "    if index_file.exists():\n",
+    "        try:\n",
+    "            current_index = int(index_file.read_text().strip())\n",
+    "        except:\n",
+    "            current_index = 0\n",
+    "    \n",
+    "    print(f\"Resuming from index {current_index}\")\n",
+    "    \n",
+    "    # Process rows\n",
+    "    use_timeout = (model_type == 'qwen')\n",
+    "    \n",
+    "    for i in tqdm(range(current_index, len(df)), desc=f\"{model_type.capitalize()} Annotation\"):\n",
+    "        prompt = df.at[i, \"prompt\"]\n",
+    "        \n",
+    "        # Query with retries\n",
+    "        response = None\n",
+    "        for attempt in range(3):\n",
+    "            response = query_model(prompt, model, tokenizer, config, use_timeout)\n",
+    "            \n",
+    "            if response and len(response.strip()) > 10:\n",
+    "                break\n",
+    "            \n",
+    "            print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "            time.sleep(0.5)\n",
+    "        \n",
+    "        # Skip if invalid\n",
+    "        if not response or len(response.strip()) <= 10:\n",
+    "            print(f\"❌ Row {i}: failed after retries, skipping\")\n",
+    "            continue\n",
+    "        \n",
+    "        # Parse and validate\n",
+    "        parsed = parse_response(response)\n",
+    "        \n",
+    "        if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "            print(f\"❌ Row {i}: parsed as all Unknown, skipping\")\n",
+    "            continue\n",
+    "        \n",
+    "        # Write fields\n",
+    "        for key, value in parsed.items():\n",
+    "            df.at[i, key] = value\n",
+    "        \n",
+    "        current_index = i + 1\n",
+    "        \n",
+    "        # GPU cleanup\n",
+    "        if torch.cuda.is_available():\n",
+    "            torch.cuda.empty_cache()\n",
+    "            torch.cuda.synchronize()\n",
+    "        \n",
+    "        # Save progress\n",
+    "        if (i + 1) % save_interval == 0 or (i + 1) == len(df):\n",
+    "            df.to_csv(output_file, index=False)\n",
+    "            index_file.write_text(str(current_index))\n",
+    "            print(f\"💾 Progress saved after row {i+1}\")\n",
+    "    \n",
+    "    # Final save\n",
+    "    df.to_csv(output_file, index=False)\n",
+    "    index_file.write_text(str(current_index))\n",
+    "    print(f\"✓ Finished annotation with {model_type}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55da2f4c",
+   "metadata": {},
+   "source": [
+    "### Usage Examples\n",
+    "Run annotation with your chosen model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "351ea40c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example 1: Annotate with Mistral (13.5 GB VRAM)\n",
+    "# annotate_dataset(model_type='mistral', test_mode=False)\n",
+    "\n",
+    "# Example 2: Annotate with Gemma (56.3 GB VRAM)\n",
+    "# annotate_dataset(model_type='gemma', test_mode=False)\n",
+    "\n",
+    "# Example 3: Annotate with Qwen (32.7 GB VRAM, 8-bit)\n",
+    "# annotate_dataset(model_type='qwen', test_mode=False)\n",
+    "\n",
+    "# Test mode (first 100 rows)\n",
+    "# annotate_dataset(model_type='mistral', test_mode=True, test_size=100)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6431d347-d80c-4e8b-83a7-531e5df95a72",
+   "metadata": {},
+   "source": [
+    "## EuroLLM-9B-Instruct"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8203abc-e7c3-4cb6-aaeb-fdc6933981fc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "import signal\n",
+    "from contextlib import contextmanager\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/eurollm_local_query_index.txt\"\n",
+    "output_file = current_dir.parent / f\"data/CSV/eurollm_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "\n",
+    "# Model settings\n",
+    "MODEL_NAME = \"utter-project/EuroLLM-9B-Instruct\"\n",
+    "#MODEL_NAME = \"Qwen/Qwen2.5-32B-Instruct\"\n",
+    "#MODEL_NAME = \"Qwen/Qwen2.5-14B-Instruct\"\n",
+    "#MODEL_NAME =  \"Qwen/Qwen3-235B-A22B-Instruct-2507-FP8\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Get HF token from credentials file\n",
+    "import os\n",
+    "credentials_dir = current_dir.parent / \"misc/credentials\"\n",
+    "hf_token_file = credentials_dir / \"hf_token.txt\"\n",
+    "\n",
+    "HF_TOKEN = None\n",
+    "if hf_token_file.exists():\n",
+    "    HF_TOKEN = hf_token_file.read_text().strip()\n",
+    "    print(\"✅ HF token loaded from credentials file\")\n",
+    "else:\n",
+    "    print(\"⚠️  HF token file not found at:\", hf_token_file)\n",
+    "    print(\"    The script will try to use cached credentials from 'huggingface-cli login'\")\n",
+    "    print(\"    Or create the file: misc/credentials/hf_token.txt with your token\")\n",
+    "    HF_TOKEN = None  # Will use cached token if available\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True,\n",
+    "        token=HF_TOKEN\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False,\n",
+    "        token=HF_TOKEN\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "# Configure 8-bit quantization for A100\n",
+    "print(\"Configuring 8-bit quantization...\")\n",
+    "quantization_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True,\n",
+    "    llm_int8_threshold=6.0,\n",
+    "    llm_int8_has_fp16_weight=False\n",
+    ")\n",
+    "\n",
+    "# Load model with 8-bit quantization\n",
+    "print(\"Loading model with 8-bit quantization (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    quantization_config=quantization_config,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False,\n",
+    "    token=HF_TOKEN\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded with 8-bit quantization\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "if output_file.exists():\n",
+    "    print(\"Loading annotated CSV...\")\n",
+    "    df = pd.read_csv(output_file)\n",
+    "else:\n",
+    "    print(\"Loading raw input CSV...\")\n",
+    "    df = pd.read_csv(input_file)\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS (OPTIMIZED FOR CLEAN OUTPUTS) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for EuroLLM annotation with strict formatting requirements.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Extract information about '{name}'. \n",
+    "Context hints (DO NOT copy these as professions): {hint_text}\n",
+    "\n",
+    "Respond with EXACTLY 5 numbered lines. Each line must contain ONLY the value, no labels or extra text.\n",
+    "\n",
+    "FORMAT REQUIREMENTS:\n",
+    "1. Full legal name in Western order (first last). VALUE ONLY.\n",
+    "2. Stage names/aliases, comma-separated. If none, write \"None\". VALUE ONLY.\n",
+    "3. Gender: MUST be exactly one word: Male, Female, Other, or Unknown. VALUE ONLY.\n",
+    "4. Professions: Choose up to 3 from this list ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality. Comma-separated. VALUE ONLY.\n",
+    "5. Primary country: Country name only (e.g., \"China\", \"United States\", \"Colombia\"). VALUE ONLY.\n",
+    "\n",
+    "CRITICAL RULES FOR PROFESSIONS (Line 4):\n",
+    "- ONLY use the exact profession categories listed above\n",
+    "- DO NOT use descriptive words like \"sexy\", \"photorealistic\", \"celebrity\"\n",
+    "- DO NOT copy the hint words as professions\n",
+    "- If uncertain about profession, write \"Unknown\"\n",
+    "- Valid professions are ONLY: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality\n",
+    "- Actress = actor, streamer = online personality, YouTuber = online personality\n",
+    "\n",
+    "OTHER RULES:\n",
+    "- Use \"Unknown\" when uncertain or for fictional characters\n",
+    "- NO explanatory text, NO labels like \"Gender:\", NO prefixes\n",
+    "- For multi-role people, list up to 3 categories by relevance\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "@contextmanager\n",
+    "def timeout(duration):\n",
+    "    def handler(signum, frame):\n",
+    "        raise TimeoutError(\"Operation timed out\")\n",
+    "    \n",
+    "    # Set the signal handler and alarm\n",
+    "    signal.signal(signal.SIGALRM, handler)\n",
+    "    signal.alarm(duration)\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        signal.alarm(0)  # Disable the alarm\n",
+    "\n",
+    "\n",
+    "def query_eurollm_local(prompt: str) -> str:\n",
+    "    \"\"\"Query EuroLLM locally via transformers with very low temperature.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for EuroLLM with strict system prompt\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are a data extraction assistant. Respond with exactly 5 numbered lines containing ONLY values. No labels, no explanations, no prefixes. Follow the format precisely.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template') and tokenizer.chat_template is not None:\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for models without chat template\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate with timeout and very low temperature\n",
+    "        with timeout(60):\n",
+    "            with torch.no_grad():\n",
+    "                outputs = model.generate(\n",
+    "                    **inputs,\n",
+    "                    max_new_tokens=100,\n",
+    "                    temperature=0.01,  # Very low temperature for more deterministic outputs\n",
+    "                    do_sample=True,     # Must be True when temperature is set\n",
+    "                    pad_token_id=tokenizer.eos_token_id\n",
+    "                )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except TimeoutError:\n",
+    "        print(f\"[ERROR] Generation timed out after 60 seconds\")\n",
+    "        return None\n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        import traceback\n",
+    "        traceback.print_exc()\n",
+    "        return None\n",
+    "\n",
+    "    \n",
+    "# === PARSE RESPONSE WITH CLEANING ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse EuroLLM response into structured fields with cleaning.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Valid profession categories\n",
+    "    VALID_PROFESSIONS = {\n",
+    "        \"actor\", \"adult performer\", \"singer/musician\", \"model\", \n",
+    "        \"online personality\", \"public figure\", \"voice actor/asmr\", \n",
+    "        \"sports professional\", \"tv personality\"\n",
+    "    }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            # Clean gender field - remove any labels\n",
+    "            gender_raw = line[2:].strip()\n",
+    "            # Remove common prefixes\n",
+    "            gender_raw = re.sub(r'^(Gender:|gender:)\\s*', '', gender_raw, flags=re.IGNORECASE)\n",
+    "            # Extract just the gender word\n",
+    "            gender_match = re.search(r'\\b(Male|Female|Other|Unknown)\\b', gender_raw, re.IGNORECASE)\n",
+    "            fields['gender'] = gender_match.group(1).capitalize() if gender_match else gender_raw\n",
+    "        elif line.startswith('4.'):\n",
+    "            # Clean and validate profession field\n",
+    "            profession_raw = line[2:].strip()\n",
+    "            \n",
+    "            # Split by comma and validate each profession\n",
+    "            professions = [p.strip().lower() for p in profession_raw.split(',')]\n",
+    "            valid_profs = []\n",
+    "            \n",
+    "            for prof in professions:\n",
+    "                # Check if it's a valid profession\n",
+    "                if prof in VALID_PROFESSIONS:\n",
+    "                    valid_profs.append(prof)\n",
+    "                # Check for common invalid entries\n",
+    "                elif prof in ['unknown', '']:\n",
+    "                    continue\n",
+    "                # Reject descriptive words that aren't professions\n",
+    "                elif prof in ['sexy', 'photorealistic', 'celebrity', 'famous', 'popular', \n",
+    "                            'beautiful', 'attractive', 'hot', 'gorgeous']:\n",
+    "                    continue\n",
+    "                # If it looks like it might be close to a valid profession, keep it\n",
+    "                elif any(valid in prof for valid in VALID_PROFESSIONS):\n",
+    "                    # Try to extract the valid part\n",
+    "                    for valid in VALID_PROFESSIONS:\n",
+    "                        if valid in prof:\n",
+    "                            valid_profs.append(valid)\n",
+    "                            break\n",
+    "            \n",
+    "            # Set the cleaned professions or Unknown if none are valid\n",
+    "            if valid_profs:\n",
+    "                fields['profession_llm'] = ', '.join(valid_profs)\n",
+    "            else:\n",
+    "                fields['profession_llm'] = 'Unknown'\n",
+    "                \n",
+    "        elif line.startswith('5.'):\n",
+    "            # Clean country field - remove any labels\n",
+    "            country_raw = line[2:].strip()\n",
+    "            # Remove common prefixes like \"Primary country:\", \"Country:\", etc.\n",
+    "            country_raw = re.sub(r'^(Primary\\s+)?(associated\\s+)?country:\\s*', '', country_raw, flags=re.IGNORECASE)\n",
+    "            fields['country'] = country_raw\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"EuroLLM Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_eurollm_local(prompt)\n",
+    "        \n",
+    "        # DEBUG: Print first few responses to see what's happening\n",
+    "        if i < 5:\n",
+    "            print(f\"\\n=== DEBUG Row {i}, Attempt {attempt+1} ===\")\n",
+    "            print(f\"Response length: {len(response) if response else 0}\")\n",
+    "            print(f\"Response: {response[:500] if response else 'None'}\")\n",
+    "            print(\"=\" * 50)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # DEBUG: Print first few parsed results\n",
+    "    if i < 5:\n",
+    "        print(f\"\\n=== PARSED Row {i} ===\")\n",
+    "        for key, value in parsed.items():\n",
+    "            print(f\"  {key}: {value}\")\n",
+    "        print(\"=\" * 50)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "472e5ac2-ec04-4bfa-8a67-116277238c15",
+   "metadata": {},
+   "source": [
+    "## Mistral 24b instruct"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a55a5e30-83f3-4f7c-a537-b1216d4e8a07",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-12-09T22:16:21.002786Z",
+     "iopub.status.busy": "2025-12-09T22:16:21.002337Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from .autonotebook import tqdm as notebook_tqdm\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Loading model: mistralai/Mistral-Small-Instruct-2409\n",
+      "Cache directory: /shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/data/models\n",
+      "This may take a while on first run (~65GB download)...\n",
+      "\n",
+      "Device: cuda\n",
+      "Loading tokenizer...\n",
+      "✅ Tokenizer loaded\n",
+      "Loading model (this may take several minutes)...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "`torch_dtype` is deprecated! Use `dtype` instead!\n",
+      "Loading checkpoint shards: 100%|██████████| 9/9 [02:42<00:00, 18.06s/it]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Model loaded\n",
+      "VRAM used: 21.40 GB\n",
+      "\n",
+      "Loading raw input CSV...\n",
+      "Loaded 50861 rows from input file\n",
+      "Found existing annotations, merging...\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_3104208/1997558719.py:113: DtypeWarning: Columns (52,53,54,55,56) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  existing_df = pd.read_csv(output_file)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Existing annotations has 50861 rows\n",
+      "Merged annotations, continuing with 50861 total rows\n",
+      "✅ Loaded professions.csv\n",
+      "✅ Loaded profession mapping with 9 categories\n",
+      "Loaded 50861 rows\n",
+      "\n",
+      "Profession categories (9):\n",
+      "  - actor\n",
+      "  - adult performer\n",
+      "  - singer/musician\n",
+      "  - model\n",
+      "  - online personality\n",
+      "  - public figure\n",
+      "  - voice actor/ASMR\n",
+      "  - sports professional\n",
+      "  - tv personality\n",
+      "\n",
+      "Creating prompts...\n",
+      "✅ Prompts created\n",
+      "Resuming from index 8810\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Mistral Local:   0%|          | 0/42051 [00:00<?, ?it/s]/shares/weddigen.ki.uzh/laura_wagner/phase_01/pm-paper/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:181: UserWarning: MatMul8bitLt: inputs will be cast from torch.bfloat16 to float16 during quantization\n",
+      "  warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n",
+      "Mistral Local:   0%|          | 7/42051 [00:57<93:01:03,  7.96s/it] "
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from pathlib import Path\n",
+    "from tqdm import tqdm\n",
+    "import torch\n",
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_file = current_dir.parent / \"data/CSV/model_adapter/real_person_adapter_step_02_NER.csv\"\n",
+    "professions_file = current_dir.parent / \"misc/lists/professions.csv\"\n",
+    "professions_mapped_file = current_dir.parent / \"misc/lists/professions_mapped.csv\"\n",
+    "# === PROCESS DATA ===\n",
+    "\n",
+    "\n",
+    "# === CONFIGURATION ===\n",
+    "TEST_MODE = False\n",
+    "TEST_SIZE = 100\n",
+    "MAX_ROWS = 50862\n",
+    "SAVE_INTERVAL = 10\n",
+    "\n",
+    "output_file = current_dir.parent / f\"data/CSV/mistral24_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/mistral24_local_query_index.txt\"\n",
+    "\n",
+    "\n",
+    "# Model settings\n",
+    "#MODEL_NAME = \"mistralai/Mistral-Small-3.1-24B-Instruct-2503\"\n",
+    "MODEL_NAME = \"mistralai/Mistral-Small-Instruct-2409\"\n",
+    "#MODEL_NAME = \"mistralai/Mistral-7B-Instruct-v0.3\"\n",
+    "CACHE_DIR = current_dir.parent / \"data/models\"\n",
+    "CACHE_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Define the SPECIFIC profession categories\n",
+    "PROFESSION_CATEGORIES = [\n",
+    "    \"actor\",\n",
+    "    \"adult performer\",\n",
+    "    \"singer/musician\",\n",
+    "    \"model\",\n",
+    "    \"online personality\",\n",
+    "    \"public figure\",\n",
+    "    \"voice actor/ASMR\",\n",
+    "    \"sports professional\",\n",
+    "    \"tv personality\"\n",
+    "]\n",
+    "\n",
+    "# === LOAD MODEL ===\n",
+    "print(f\"Loading model: {MODEL_NAME}\")\n",
+    "print(f\"Cache directory: {CACHE_DIR}\")\n",
+    "print(f\"This may take a while on first run (~65GB download)...\\n\")\n",
+    "\n",
+    "# Check GPU availability\n",
+    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
+    "print(f\"Device: {device}\")\n",
+    "\n",
+    "if device == \"cpu\":\n",
+    "    print(\"⚠️  WARNING: No GPU detected! Inference will be VERY slow.\")\n",
+    "    print(\"   Consider using a GPU or reducing model size.\")\n",
+    "\n",
+    "# Load tokenizer\n",
+    "print(\"Loading tokenizer...\")\n",
+    "try:\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=True\n",
+    "    )\n",
+    "except Exception as e:\n",
+    "    print(f\"Failed with use_fast=True, trying use_fast=False...\")\n",
+    "    tokenizer = AutoTokenizer.from_pretrained(\n",
+    "        MODEL_NAME,\n",
+    "        cache_dir=str(CACHE_DIR),\n",
+    "        use_fast=False\n",
+    "    )\n",
+    "\n",
+    "# Ensure pad token is set\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "print(\"✅ Tokenizer loaded\")\n",
+    "\n",
+    "quantization_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Load model with optimizations\n",
+    "print(\"Loading model (this may take several minutes)...\")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    cache_dir=str(CACHE_DIR),\n",
+    "    torch_dtype=torch.bfloat16,\n",
+    "    quantization_config=quantization_config,\n",
+    "    device_map=\"auto\",\n",
+    "    trust_remote_code=False\n",
+    ")\n",
+    "model.eval()\n",
+    "print(\"✅ Model loaded\")\n",
+    "\n",
+    "# Check VRAM usage\n",
+    "if torch.cuda.is_available():\n",
+    "    vram_gb = torch.cuda.max_memory_allocated() / 1024**3\n",
+    "    print(f\"VRAM used: {vram_gb:.2f} GB\\n\")\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "print(\"Loading raw input CSV...\")\n",
+    "df = pd.read_csv(input_file)  # ALWAYS load the full input\n",
+    "print(f\"Loaded {len(df)} rows from input file\")\n",
+    "\n",
+    "# If we have previous annotations, merge them\n",
+    "if output_file.exists():\n",
+    "    print(\"Found existing annotations, merging...\")\n",
+    "    existing_df = pd.read_csv(output_file)\n",
+    "    print(f\"Existing annotations has {len(existing_df)} rows\")\n",
+    "    \n",
+    "    # Update df with existing annotations\n",
+    "    # Only update the columns that were annotated\n",
+    "    annotation_cols = ['full_name', 'aliases', 'gender', 'profession_llm', 'country']\n",
+    "    for col in annotation_cols:\n",
+    "        if col in existing_df.columns:\n",
+    "            df[col] = existing_df[col][:len(df)]  # Make sure we don't exceed df length\n",
+    "    \n",
+    "    print(f\"Merged annotations, continuing with {len(df)} total rows\")\n",
+    "\n",
+    "\n",
+    "# Try to load profession mapping files\n",
+    "try:\n",
+    "    professions_df = pd.read_csv(professions_file)\n",
+    "    print(f\"✅ Loaded professions.csv\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions.csv not found\")\n",
+    "\n",
+    "try:\n",
+    "    prof_mapped_df = pd.read_csv(professions_mapped_file)\n",
+    "    print(f\"✅ Loaded profession mapping with {len(prof_mapped_df)} categories\")\n",
+    "except:\n",
+    "    print(\"⚠️ Warning: professions_mapped.csv not found, using default categories\")\n",
+    "\n",
+    "profession_str = \", \".join(PROFESSION_CATEGORIES)\n",
+    "\n",
+    "print(f\"Loaded {len(df)} rows\")\n",
+    "print(f\"\\nProfession categories ({len(PROFESSION_CATEGORIES)}):\")\n",
+    "for cat in PROFESSION_CATEGORIES:\n",
+    "    print(f\"  - {cat}\")\n",
+    "\n",
+    "if TEST_MODE:\n",
+    "    print(f\"\\nRunning in TEST MODE with {TEST_SIZE} samples\")\n",
+    "    df = df.head(TEST_SIZE).copy()\n",
+    "elif MAX_ROWS:\n",
+    "    df = df.head(MAX_ROWS).copy()\n",
+    "\n",
+    "# === CREATE PROMPTS (DEEPSEEK STYLE) ===\n",
+    "def create_prompt(row):\n",
+    "    \"\"\"Create prompt for Mistral annotation with specific profession categories.\"\"\"\n",
+    "    name = row['real_name'] if pd.notna(row.get('real_name')) else row.get('name', '')\n",
+    "    \n",
+    "    # Gather hints\n",
+    "    hints = []\n",
+    "    if pd.notna(row.get('likely_profession')):\n",
+    "        hints.append(str(row['likely_profession']))\n",
+    "    if pd.notna(row.get('likely_nationality')):\n",
+    "        hints.append(str(row['likely_nationality']))\n",
+    "    if pd.notna(row.get('likely_country')):\n",
+    "        hints.append(str(row['likely_country']))\n",
+    "    \n",
+    "    # Add tags if we don't have enough hints\n",
+    "    if len(hints) < 3:\n",
+    "        for i in range(1, 8):\n",
+    "            tag_col = f'tag_{i}'\n",
+    "            if tag_col in row and pd.notna(row[tag_col]):\n",
+    "                tag_val = str(row[tag_col])\n",
+    "                if tag_val not in hints:\n",
+    "                    hints.append(tag_val)\n",
+    "                if len(hints) >= 5:\n",
+    "                    break\n",
+    "    \n",
+    "    hint_text = \", \".join(hints[:5]) if hints else \"none\"\n",
+    "    \n",
+    "    return f\"\"\"Given '{name}' ({hint_text}), provide:\n",
+    "1. Full legal name (Western order if non-latin script)\n",
+    "2. Any stage names/aliases (comma separated)\n",
+    "3. Gender (Male/Female/Other/Unknown)\n",
+    "4. Top 3 most likely professions from ONLY these categories:\n",
+    "   - actor\n",
+    "   - adult performer\n",
+    "   - singer/musician\n",
+    "   - model\n",
+    "   - online personality (includes streamers, cosplayers, influencers)\n",
+    "   - public figure (includes politicians, activists, journalists, authors)\n",
+    "   - voice actor/ASMR\n",
+    "   - sports professional\n",
+    "   - tv personality (includes hosts, presenters, reality TV)\n",
+    "\n",
+    "5. Primary country associated\n",
+    "\n",
+    "IMPORTANT:\n",
+    "- Choose professions ONLY from the 9 categories above\n",
+    "- Provide up to 3 professions, comma-separated, ordered by relevance\n",
+    "- Be SPECIFIC: choose the most accurate category for each role\n",
+    "- \"online personality\" includes: streamers, cosplayers, YouTubers, influencers, content creators\n",
+    "- Use 'Unknown' when uncertain or for fictional characters/places\n",
+    "- For multi-role people, list all relevant categories (e.g., \"actor, singer/musician, online personality\")\n",
+    "- For country respond with one word only, for example China or Columbia\n",
+    "- actress = actor\n",
+    "\n",
+    "Respond with exactly 5 numbered lines.\"\"\"\n",
+    "\n",
+    "# Create prompts\n",
+    "print(\"\\nCreating prompts...\")\n",
+    "df['prompt'] = df.apply(create_prompt, axis=1)\n",
+    "print(\"✅ Prompts created\")\n",
+    "\n",
+    "# === QUERY MISTRAL LOCAL ===\n",
+    "def query_mistral_local(prompt: str) -> str:\n",
+    "    \"\"\"Query Mistral locally via transformers.\"\"\"\n",
+    "    try:\n",
+    "        # Format as chat message for Mistral\n",
+    "        messages = [\n",
+    "            {\"role\": \"system\", \"content\": \"You are an assistant that extracts key data on a person based on the name. Respond with exactly 5 numbered lines. For professions, choose ONLY from these categories: actor, adult performer, singer/musician, model, online personality, public figure, voice actor/ASMR, sports professional, tv personality.\"},\n",
+    "            {\"role\": \"user\", \"content\": prompt}\n",
+    "        ]\n",
+    "        \n",
+    "        # Tokenize\n",
+    "        if hasattr(tokenizer, 'apply_chat_template'):\n",
+    "            text = tokenizer.apply_chat_template(\n",
+    "                messages,\n",
+    "                tokenize=False,\n",
+    "                add_generation_prompt=True\n",
+    "            )\n",
+    "        else:\n",
+    "            # Fallback for older tokenizers\n",
+    "            text = f\"[INST] {prompt} [/INST]\"\n",
+    "        \n",
+    "        inputs = tokenizer([text], return_tensors=\"pt\", padding=True).to(device)\n",
+    "        \n",
+    "        # Generate\n",
+    "        with torch.no_grad():\n",
+    "            outputs = model.generate(\n",
+    "                **inputs,\n",
+    "                max_new_tokens=512,\n",
+    "                temperature=0.05,\n",
+    "                do_sample=True,\n",
+    "                top_p=0.8,\n",
+    "                pad_token_id=tokenizer.pad_token_id if tokenizer.pad_token_id else tokenizer.eos_token_id\n",
+    "            )\n",
+    "        \n",
+    "        # Decode\n",
+    "        generated_ids = outputs[0][inputs['input_ids'].shape[1]:]\n",
+    "        response = tokenizer.decode(generated_ids, skip_special_tokens=True)\n",
+    "        \n",
+    "        return response.strip()\n",
+    "        \n",
+    "    except Exception as e:\n",
+    "        print(f\"Generation error: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# === PARSE RESPONSE (DEEPSEEK STYLE) ===\n",
+    "def parse_response(response):\n",
+    "    \"\"\"Parse Mistral response into structured fields.\"\"\"\n",
+    "    if not response:\n",
+    "        return {\n",
+    "            'full_name': 'Unknown',\n",
+    "            'aliases': 'Unknown',\n",
+    "            'gender': 'Unknown',\n",
+    "            'profession_llm': 'Unknown',\n",
+    "            'country': 'Unknown'\n",
+    "        }\n",
+    "    \n",
+    "    # Split into lines and clean\n",
+    "    lines = [line.strip() for line in response.split('\\n') if line.strip()]\n",
+    "    \n",
+    "    # Initialize with Unknown values\n",
+    "    fields = {\n",
+    "        'full_name': 'Unknown',\n",
+    "        'aliases': 'Unknown',\n",
+    "        'gender': 'Unknown',\n",
+    "        'profession_llm': 'Unknown',\n",
+    "        'country': 'Unknown'\n",
+    "    }\n",
+    "    \n",
+    "    # Extract information from each numbered line\n",
+    "    for line in lines:\n",
+    "        if line.startswith('1.'):\n",
+    "            fields['full_name'] = line[2:].strip()\n",
+    "        elif line.startswith('2.'):\n",
+    "            fields['aliases'] = line[2:].strip()\n",
+    "        elif line.startswith('3.'):\n",
+    "            fields['gender'] = line[2:].strip()\n",
+    "        elif line.startswith('4.'):\n",
+    "            fields['profession_llm'] = line[2:].strip()\n",
+    "        elif line.startswith('5.'):\n",
+    "            fields['country'] = line[2:].strip()\n",
+    "    \n",
+    "    return fields\n",
+    "\n",
+    "# === PROCESS DATA ===\n",
+    "output_file = current_dir.parent / f\"data/CSV/mistral24_local_annotated_POI{'_test' if TEST_MODE else ''}.csv\"\n",
+    "index_file = current_dir.parent / \"misc/query_indicies/mistral24_local_query_index.txt\"\n",
+    "\n",
+    "index_file.parent.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Load index\n",
+    "current_index = 0\n",
+    "if index_file.exists():\n",
+    "    try:\n",
+    "        current_index = int(index_file.read_text().strip())\n",
+    "    except:\n",
+    "        current_index = 0\n",
+    "\n",
+    "print(f\"Resuming from index {current_index}\")\n",
+    "\n",
+    "start_time = time.time()\n",
+    "\n",
+    "for i in tqdm(range(current_index, len(df)), desc=\"Mistral Local\"):\n",
+    "\n",
+    "    prompt = df.at[i, \"prompt\"]\n",
+    "\n",
+    "    # -------- MODEL QUERY WITH RETRIES --------\n",
+    "    response = None\n",
+    "    for attempt in range(3):\n",
+    "        response = query_mistral_local(prompt)\n",
+    "        \n",
+    "        # Valid response?\n",
+    "        if response and len(response.strip()) > 10:\n",
+    "            break\n",
+    "        \n",
+    "        print(f\"⚠️ Row {i}: Empty or invalid response, retry {attempt+1}/3\")\n",
+    "        time.sleep(0.5)\n",
+    "\n",
+    "    # If still invalid → DO NOT overwrite previous data\n",
+    "    if not response or len(response.strip()) <= 10:\n",
+    "        print(f\"❌ Row {i}: failed after retries, not writing, not advancing index\")\n",
+    "        continue\n",
+    "\n",
+    "    parsed = parse_response(response)\n",
+    "\n",
+    "    # Additional safety: skip rows that parsed as all 'Unknown'\n",
+    "    if all(v == \"Unknown\" for v in parsed.values()):\n",
+    "        print(f\"❌ Row {i}: parsed as all Unknown (likely model crash); skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # -------- WRITE PARSED FIELDS SAFELY --------\n",
+    "    for key, value in parsed.items():\n",
+    "        df.at[i, key] = value\n",
+    "\n",
+    "    # Advance progress ONLY after successful write\n",
+    "    current_index = i + 1\n",
+    "\n",
+    "    # -------- GPU MEMORY CLEANUP --------\n",
+    "    if torch.cuda.is_available():\n",
+    "        torch.cuda.empty_cache()\n",
+    "        torch.cuda.synchronize()\n",
+    "\n",
+    "    # -------- SAVE LIKE YOUR DEEPSEEK VERSION --------\n",
+    "    if (i + 1) % SAVE_INTERVAL == 0 or (i + 1) == len(df):\n",
+    "        df.to_csv(output_file, index=False)\n",
+    "        with open(index_file, \"w\") as f:\n",
+    "            f.write(str(current_index))\n",
+    "        print(f\"💾 Progress saved after row {i+1}\")\n",
+    "\n",
+    "# Final save\n",
+    "df.to_csv(output_file, index=False)\n",
+    "index_file.write_text(str(current_index))\n",
+    "print(\"✅ Finished full dataset.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d7212e75-0ff6-45a0-8695-c4a3d3e02818",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import transformers\n",
+    "print(f\"Transformers version: {transformers.__version__}\")\n",
+    "\n",
+    "# Check if Mistral3 is available\n",
+    "try:\n",
+    "    from transformers import Mistral3ForCausalLM\n",
+    "    print(\"✅ Mistral3 is available\")\n",
+    "except ImportError:\n",
+    "    print(\"❌ Mistral3 not available in this transformers version\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a6ab032e-246e-4c4e-9776-ff0bfbf6fd9c",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "pm-paper",
+   "language": "python",
+   "name": "pm-paper"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-3-4__Figure_8a_sunburst_gender.ipynb

@@ -0,0 +1,129 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prepare *.json for Figure 8a"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✓ Saved 8a.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from collections import defaultdict\n",
+    "import json\n",
+    "from pathlib import Path \n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "sunburst_json = current_dir.parent / \"public/json/8a.json\"\n",
+    "\n",
+    "# Load consensus CSV\n",
+    "consensus_file = current_dir.parent / \"data/CSV/analyzed_llm_agreement_consensus.csv\"\n",
+    "df = pd.read_csv(consensus_file)\n",
+    "\n",
+    "# ---- Normalize Gender (group Non-binary and Unknown into 'Other') ----\n",
+    "def normalize_gender(g):\n",
+    "    g = str(g).strip().lower()\n",
+    "    if g in [\"female\", \"woman\", \"female (group)\", \"female (transgender)\", \"female (virtual persona)\", \"female (group members)\"]:\n",
+    "        return \"Female\"\n",
+    "    elif g in [\"male\", \"male (android)\", \"male (character)\"]:\n",
+    "        return \"Male\"\n",
+    "    else:\n",
+    "        return \"Other\"\n",
+    "\n",
+    "df['gender_normalized'] = df['consensus_gender'].apply(normalize_gender)\n",
+    "\n",
+    "# ---- Step 1: Limit to top 10 countries ----\n",
+    "df['country_cleaned'] = df['consensus_country'].apply(lambda x: x if x not in ['Unknown', '', None] else 'Other')\n",
+    "top_countries = df['country_cleaned'].value_counts().nlargest(12).index.tolist()\n",
+    "df['country_limited'] = df['country_cleaned'].apply(lambda x: x if x in top_countries else 'Other')\n",
+    "\n",
+    "# ---- Step 2: Normalize and limit professions ----\n",
+    "valid_categories = [\n",
+    "    \"actor\", \"adult performer\", \"singer/musician\", \"model\",\n",
+    "    \"online personality\", \"tv personality\", \"voice actor/asmr\", \"public figure\", \"sports professional\"\n",
+    "]\n",
+    "\n",
+    "def remap_profession(profession):\n",
+    "    profession_lower = str(profession).strip().lower()\n",
+    "    if profession_lower == 'unknown' or profession_lower not in valid_categories:\n",
+    "        return 'Other'\n",
+    "    elif profession_lower == 'fictional character':\n",
+    "        return 'actor'\n",
+    "    elif profession_lower in ['voice actor', 'voice actor/asmr']:\n",
+    "        return 'voice actor/ASMR'\n",
+    "    return profession_lower\n",
+    "\n",
+    "df['profession_limited'] = df['consensus_primary_profession'].apply(remap_profession)\n",
+    "\n",
+    "# ---- Step 3: Group by gender and profession ----\n",
+    "sunburst_data = df.groupby(['gender_normalized', 'profession_limited']).size().reset_index(name='count')\n",
+    "\n",
+    "# ---- Step 4: Create nested structure for D3.js ----\n",
+    "sunburst_dict = {\"name\": \"root\", \"children\": []}\n",
+    "gender_map = defaultdict(list)\n",
+    "\n",
+    "for _, row in sunburst_data.iterrows():\n",
+    "    gender = row['gender_normalized']\n",
+    "    profession = row['profession_limited']\n",
+    "    count = int(row['count'])\n",
+    "    gender_map[gender].append({\"name\": profession, \"value\": count})\n",
+    "\n",
+    "# Sort 'Other' professions to appear last\n",
+    "for gender, professions in gender_map.items():\n",
+    "    professions_sorted = sorted(professions, key=lambda d: (d[\"name\"] == \"Other\", d[\"name\"]))\n",
+    "    gender_map[gender] = professions_sorted\n",
+    "\n",
+    "# Build the final nested structure\n",
+    "for gender, professions in gender_map.items():\n",
+    "    sunburst_dict[\"children\"].append({\"name\": gender, \"children\": professions})\n",
+    "\n",
+    "# ---- Step 5: Save to a JSON file ----\n",
+    "with open(sunburst_json, \"w\", encoding='utf-8') as f:\n",
+    "    json.dump(sunburst_dict, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "print(\"✓ Saved 8a.json\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/Section_2-3-4__Figure_8b_sunburst_profession.ipynb

@@ -0,0 +1,332 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prepare *.json for Figure 8b"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✓ Saved sunburst_countries_A.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from collections import defaultdict\n",
+    "import json\n",
+    "from pathlib import Path\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "sunburst_path = current_dir.parent / \"public/json/sunburst_countries_A.json\"\n",
+    "\n",
+    "# Load consensus CSV\n",
+    "consensus_file = current_dir.parent / \"data/CSV/analyzed_llm_agreement_consensus_va.csv\"\n",
+    "df = pd.read_csv(consensus_file)\n",
+    "\n",
+    "# ============================================================\n",
+    "# 1. NORMALIZE COUNTRIES\n",
+    "# ============================================================\n",
+    "\n",
+    "def normalize_country(x: str):\n",
+    "    if not isinstance(x, str) or x.strip() == \"\" or x.lower() == \"unknown\":\n",
+    "        return \"Other\"\n",
+    "    x = x.strip()\n",
+    "\n",
+    "    # Ensure matching with your HTML manualOrder\n",
+    "    replacements = {\n",
+    "        \"USA\": \"United States\",\n",
+    "        \"US\": \"United States\",\n",
+    "        \"U.S.\": \"United States\",\n",
+    "        \"UK\": \"United Kingdom\",\n",
+    "        \"U.K.\": \"United Kingdom\"\n",
+    "    }\n",
+    "    return replacements.get(x, x)\n",
+    "\n",
+    "df[\"country_clean\"] = df[\"consensus_country\"].apply(normalize_country)\n",
+    "\n",
+    "# Limit to the top 15 countries, merge the rest into \"Other\"\n",
+    "top_countries = df[\"country_clean\"].value_counts().nlargest(25).index.tolist()\n",
+    "\n",
+    "df[\"country_limited\"] = df[\"country_clean\"].apply(\n",
+    "    lambda x: x if x in top_countries else \"Other\"\n",
+    ")\n",
+    "\n",
+    "# ============================================================\n",
+    "# 2. NORMALIZE PROFESSIONS\n",
+    "# ============================================================\n",
+    "\n",
+    "def normalize_profession(x: str):\n",
+    "    if not isinstance(x, str) or x.strip() == \"\" or x.lower() == \"unknown\":\n",
+    "        return \"Other\"\n",
+    "    x = x.strip().lower()\n",
+    "    \n",
+    "    mapping = {\n",
+    "        \"actor\": \"Actor\",\n",
+    "        \"model\": \"Model\",\n",
+    "        \"adult performer\": \"Adult Performer\",\n",
+    "        \"singer/musician\": \"Singer, Musician\",\n",
+    "        \"online personality\": \"Online Personality\",\n",
+    "        \"sports professional\": \"Sports Professional\",\n",
+    "        \"voice actor/asmr\": \"Voice Actor\",        # ← fixed key\n",
+    "        \"public figure\": \"Public Figure\",          # ← now its own category\n",
+    "        \"tv personality\": \"Other\",\n",
+    "    }\n",
+    "    return mapping.get(x, \"Other\")\n",
+    "\n",
+    "df[\"profession_clean\"] = df[\"consensus_primary_profession\"].apply(normalize_profession)\n",
+    "\n",
+    "\n",
+    "df[\"profession_limited\"] = df[\"profession_clean\"]\n",
+    "\n",
+    "\n",
+    "# ============================================================\n",
+    "# 3. GROUP INTO SUNBURST STRUCTURE\n",
+    "# ============================================================\n",
+    "\n",
+    "sunburst_data = (\n",
+    "    df.groupby([\"country_limited\", \"profession_limited\"])\n",
+    "    .size()\n",
+    "    .reset_index(name=\"count\")\n",
+    ")\n",
+    "\n",
+    "sunburst_dict = {\"name\": \"root\", \"children\": []}\n",
+    "country_map = defaultdict(list)\n",
+    "\n",
+    "for _, row in sunburst_data.iterrows():\n",
+    "    c = row[\"country_limited\"]\n",
+    "    p = row[\"profession_limited\"]\n",
+    "    v = int(row[\"count\"])\n",
+    "\n",
+    "    country_map[c].append({\"name\": p, \"value\": v})\n",
+    "\n",
+    "# Sort professions inside each country so \"Other\" is last\n",
+    "for c, profs in country_map.items():\n",
+    "    profs_sorted = sorted(profs, key=lambda d: (d[\"name\"] == \"Other\", d[\"name\"]))\n",
+    "    country_map[c] = profs_sorted\n",
+    "\n",
+    "# Calculate total datapoints per country and sort\n",
+    "country_totals = []\n",
+    "for c, profs in country_map.items():\n",
+    "    total = sum(p[\"value\"] for p in profs)\n",
+    "    country_totals.append((c, total, profs))\n",
+    "\n",
+    "# Sort by total (descending), but put \"Other\" last\n",
+    "country_totals.sort(key=lambda x: (x[0] == \"Other\", -x[1]))\n",
+    "\n",
+    "# Build final JSON with sorted countries\n",
+    "for c, total, profs in country_totals:\n",
+    "    sunburst_dict[\"children\"].append({\"name\": c, \"children\": profs})\n",
+    "\n",
+    "# ============================================================\n",
+    "# 4. SAVE JSON\n",
+    "# ============================================================\n",
+    "\n",
+    "with open(sunburst_path, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(sunburst_dict, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "print(\"✓ Saved sunburst_countries_A.json\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# version that only considers data up until dec 31st 2024"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✓ Filtered to 16059 records published on or before December 31, 2024\n",
+      "✓ Saved sunburst_countries_A.json (2024 data only)\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from collections import defaultdict\n",
+    "import json\n",
+    "from pathlib import Path\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "sunburst_path = current_dir.parent / \"public/json/sunburst_countries_A.json\"\n",
+    "\n",
+    "# Load consensus CSV\n",
+    "consensus_file = current_dir.parent / \"data/CSV/analyzed_llm_agreement_consensus_va.csv\"\n",
+    "df = pd.read_csv(consensus_file)\n",
+    "\n",
+    "# ============================================================\n",
+    "# FILTER DATA UP TO DECEMBER 31, 2024\n",
+    "# ============================================================\n",
+    "# Convert publishedAt to datetime\n",
+    "df[\"publishedAt\"] = pd.to_datetime(df[\"publishedAt\"], errors=\"coerce\", utc=True)\n",
+    "\n",
+    "# Filter to only include data up to December 31, 2024\n",
+    "# Make cutoff_date timezone-aware (UTC) to match publishedAt\n",
+    "cutoff_date = pd.Timestamp(\"2024-12-31 23:59:59\", tz=\"UTC\")\n",
+    "df = df[df[\"publishedAt\"] <= cutoff_date]\n",
+    "\n",
+    "print(f\"✓ Filtered to {len(df)} records published on or before December 31, 2024\")\n",
+    "\n",
+    "# ============================================================\n",
+    "# 1. NORMALIZE COUNTRIES\n",
+    "# ============================================================\n",
+    "def normalize_country(x: str):\n",
+    "    if not isinstance(x, str) or x.strip() == \"\" or x.lower() == \"unknown\":\n",
+    "        return \"Other\"\n",
+    "    x = x.strip()\n",
+    "    # Ensure matching with your HTML manualOrder\n",
+    "    replacements = {\n",
+    "        \"USA\": \"United States\",\n",
+    "        \"US\": \"United States\",\n",
+    "        \"U.S.\": \"United States\",\n",
+    "        \"UK\": \"United Kingdom\",\n",
+    "        \"U.K.\": \"United Kingdom\"\n",
+    "    }\n",
+    "    return replacements.get(x, x)\n",
+    "\n",
+    "df[\"country_clean\"] = df[\"consensus_country\"].apply(normalize_country)\n",
+    "\n",
+    "# Limit to the top 15 countries, merge the rest into \"Other\"\n",
+    "top_countries = df[\"country_clean\"].value_counts().nlargest(25).index.tolist()\n",
+    "df[\"country_limited\"] = df[\"country_clean\"].apply(\n",
+    "    lambda x: x if x in top_countries else \"Other\"\n",
+    ")\n",
+    "\n",
+    "# ============================================================\n",
+    "# 2. NORMALIZE PROFESSIONS\n",
+    "# ============================================================\n",
+    "def normalize_profession(x: str):\n",
+    "    if not isinstance(x, str) or x.strip() == \"\" or x.lower() == \"unknown\":\n",
+    "        return \"Other\"\n",
+    "    x = x.strip().lower()\n",
+    "    mapping = {\n",
+    "        \"actor\": \"Actor\",\n",
+    "        \"model\": \"Model\",\n",
+    "        \"adult performer\": \"Adult Performer\",\n",
+    "        \"singer/musician\": \"Singer, Musician\",\n",
+    "        \"online personality\": \"Online Personality\",\n",
+    "        \"sports professional\": \"Sports Professional\",\n",
+    "        \"voice actor/asmr\": \"Voice Actor\",\n",
+    "        \"public figure\": \"Public Figure\",\n",
+    "        \"tv personality\": \"Other\",\n",
+    "    }\n",
+    "    return mapping.get(x, \"Other\")\n",
+    "\n",
+    "df[\"profession_clean\"] = df[\"consensus_primary_profession\"].apply(normalize_profession)\n",
+    "\n",
+    "# Define top professions to keep\n",
+    "top_prof = [\n",
+    "    \"Actor\",\n",
+    "    \"Model\", \n",
+    "    \"Adult Performer\",\n",
+    "    \"Singer, Musician\",\n",
+    "    \"Online Personality\",\n",
+    "    \"Sports Professional\",\n",
+    "    \"Voice Actor\",\n",
+    "    \"Public Figure\",\n",
+    "    \"Other\"\n",
+    "]\n",
+    "\n",
+    "# Re-limit professions, everything else → Other\n",
+    "df[\"profession_limited\"] = df[\"profession_clean\"].apply(\n",
+    "    lambda x: x if x in top_prof else \"Other\"\n",
+    ")\n",
+    "\n",
+    "# ============================================================\n",
+    "# 3. GROUP INTO SUNBURST STRUCTURE\n",
+    "# ============================================================\n",
+    "sunburst_data = (\n",
+    "    df.groupby([\"country_limited\", \"profession_limited\"])\n",
+    "        .size()\n",
+    "        .reset_index(name=\"count\")\n",
+    ")\n",
+    "\n",
+    "sunburst_dict = {\"name\": \"root\", \"children\": []}\n",
+    "country_map = defaultdict(list)\n",
+    "\n",
+    "for _, row in sunburst_data.iterrows():\n",
+    "    c = row[\"country_limited\"]\n",
+    "    p = row[\"profession_limited\"]\n",
+    "    v = int(row[\"count\"])\n",
+    "    country_map[c].append({\"name\": p, \"value\": v})\n",
+    "\n",
+    "# Sort professions inside each country so \"Other\" is last\n",
+    "for c, profs in country_map.items():\n",
+    "    profs_sorted = sorted(profs, key=lambda d: (d[\"name\"] == \"Other\", d[\"name\"]))\n",
+    "    country_map[c] = profs_sorted\n",
+    "\n",
+    "# Calculate total datapoints per country and sort\n",
+    "country_totals = []\n",
+    "for c, profs in country_map.items():\n",
+    "    total = sum(p[\"value\"] for p in profs)\n",
+    "    country_totals.append((c, total, profs))\n",
+    "\n",
+    "# Sort by total (descending), but put \"Other\" last\n",
+    "country_totals.sort(key=lambda x: (x[0] == \"Other\", -x[1]))\n",
+    "\n",
+    "# Build final JSON with sorted countries\n",
+    "for c, total, profs in country_totals:\n",
+    "    sunburst_dict[\"children\"].append({\"name\": c, \"children\": profs})\n",
+    "\n",
+    "# ============================================================\n",
+    "# 4. SAVE JSON\n",
+    "# ============================================================\n",
+    "with open(sunburst_path, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(sunburst_dict, f, ensure_ascii=False, indent=2)\n",
+    "\n",
+    "print(\"✓ Saved sunburst_countries_A.json (2024 data only)\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "the resulting *.json is the input for Figure_8.html"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

jupyter_notebooks/Section_2-3_Figure_5_co-occurence_promotional_tags.ipynb

@@ -0,0 +1,314 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create *.json for figure 5 (Co-occurence network of Tags)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: america\n",
+      "  ✅ Saved to /home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/public/json/tags_america.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import re\n",
+    "import os\n",
+    "\n",
+    "from pathlib import Path\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = current_dir.parent / \"data/CSV/Models/Civi_models.csv\"\n",
+    "output_dir = current_dir.parent / \"public/json/\"\n",
+    "#target_terms = [\"asian\", \"indian\", \"man\", \"woman\", \"german\", \"korean\", \"american\", \"russian\", \"style\", \"japanese\", \"chinese\"]  # Add any tags you want to process\n",
+    "#target_terms = [\"character\", \"instagram\", \"youtuber\", \"actor\", \"actress\", \"celebrity\", \"vtuber\", \"kpop\"]  # Add any tags you want to process\n",
+    "target_terms = [\"america\"]  # Add any tags you want to process\n",
+    "min_connections = 1   # minimum number of link connections per node\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === MAIN LOOP ===\n",
+    "for target_term in target_terms:\n",
+    "    print(f\"Processing: {target_term}\")\n",
+    "    \n",
+    "    pattern = re.compile(rf'\\b{re.escape(target_term)}\\b', flags=re.IGNORECASE)\n",
+    "    df_filtered = df_tags[df_tags.apply(\n",
+    "        lambda row: row.astype(str).apply(lambda x: bool(pattern.search(x))).any(),\n",
+    "        axis=1\n",
+    "    )]\n",
+    "\n",
+    "    # Skip if no data matches\n",
+    "    if df_filtered.empty:\n",
+    "        print(f\"  ⚠️ No matches for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === COUNT INDIVIDUAL TAGS ===\n",
+    "    all_tags = df_filtered.values.flatten()\n",
+    "    all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "    tag_counts = Counter(all_tags)\n",
+    "\n",
+    "    # === CO-OCCURRENCE ===\n",
+    "    co_occurrences = defaultdict(int)\n",
+    "    for tags in df_filtered.itertuples(index=False, name=None):\n",
+    "        tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "        for tag1, tag2 in combinations(tags, 2):\n",
+    "            co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "    edges = [(list(pair)[0], list(pair)[1], weight) for pair, weight in co_occurrences.items()]\n",
+    "\n",
+    "    # === FILTER BY CONNECTIONS ===\n",
+    "    connected_tags = Counter()\n",
+    "    for tag1, tag2, _ in edges:\n",
+    "        connected_tags[tag1] += 1\n",
+    "        connected_tags[tag2] += 1\n",
+    "\n",
+    "    nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts if connected_tags[tag] >= min_connections]\n",
+    "    valid_ids = set(node[\"id\"] for node in nodes)\n",
+    "    links = [{\"source\": tag1, \"target\": tag2, \"value\": weight}\n",
+    "             for tag1, tag2, weight in edges\n",
+    "             if tag1 in valid_ids and tag2 in valid_ids]\n",
+    "\n",
+    "    if not nodes or not links:\n",
+    "        print(f\"  ⚠️ Not enough connections for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === EXPORT ===\n",
+    "    d3_data = {\"nodes\": nodes, \"links\": links}\n",
+    "    safe_term = re.sub(r'\\W+', '_', target_term.lower())\n",
+    "    output_file = os.path.join(output_dir, f\"tags_{safe_term}.json\")\n",
+    "    \n",
+    "    with open(output_file, \"w\") as f:\n",
+    "        json.dump(d3_data, f, indent=4)\n",
+    "    \n",
+    "    print(f\"  ✅ Saved to {output_file}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Different Countries"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_68582/2797381217.py:15: DtypeWarning: Columns (50,51,54,55,56,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,458) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  df = pd.read_csv(file_path)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processing: united states\n",
+      "  ⚠️ No matches for 'united states', skipping.\n",
+      "Processing: korea\n",
+      "  ✅ Saved to data/json/tags_korea_poi.json\n",
+      "Processing: uk\n",
+      "  ✅ Saved to data/json/tags_uk_poi.json\n",
+      "Processing: russia\n",
+      "  ✅ Saved to data/json/tags_russia_poi.json\n",
+      "Processing: china\n",
+      "  ✅ Saved to data/json/tags_china_poi.json\n",
+      "Processing: canada\n",
+      "  ✅ Saved to data/json/tags_canada_poi.json\n",
+      "Processing: India\n",
+      "  ✅ Saved to data/json/tags_india_poi.json\n",
+      "Processing: germany\n",
+      "  ✅ Saved to data/json/tags_germany_poi.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import re\n",
+    "import os\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = \"data/model_subsets/all_models_poi_true.csv\"\n",
+    "output_dir = \"data/json/\"\n",
+    "target_terms = [\"united states\", \"korea\", \"uk\", \"russia\", \"china\", \"canada\", \"India\", \"germany\"] # Add any tags you want to process\n",
+    "min_connections = 1  # minimum number of link connections per node\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === MAIN LOOP ===\n",
+    "for target_term in target_terms:\n",
+    "    print(f\"Processing: {target_term}\")\n",
+    "    \n",
+    "    pattern = re.compile(rf'\\b{re.escape(target_term)}\\b', flags=re.IGNORECASE)\n",
+    "    df_filtered = df_tags[df_tags.apply(\n",
+    "        lambda row: row.astype(str).apply(lambda x: bool(pattern.search(x))).any(),\n",
+    "        axis=1\n",
+    "    )]\n",
+    "\n",
+    "    # Skip if no data matches\n",
+    "    if df_filtered.empty:\n",
+    "        print(f\"  ⚠️ No matches for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === COUNT INDIVIDUAL TAGS ===\n",
+    "    all_tags = df_filtered.values.flatten()\n",
+    "    all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "    tag_counts = Counter(all_tags)\n",
+    "\n",
+    "    # === CO-OCCURRENCE ===\n",
+    "    co_occurrences = defaultdict(int)\n",
+    "    for tags in df_filtered.itertuples(index=False, name=None):\n",
+    "        tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "        for tag1, tag2 in combinations(tags, 2):\n",
+    "            co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "    edges = [(list(pair)[0], list(pair)[1], weight) for pair, weight in co_occurrences.items()]\n",
+    "\n",
+    "    # === FILTER BY CONNECTIONS ===\n",
+    "    connected_tags = Counter()\n",
+    "    for tag1, tag2, _ in edges:\n",
+    "        connected_tags[tag1] += 1\n",
+    "        connected_tags[tag2] += 1\n",
+    "\n",
+    "    nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts if connected_tags[tag] >= min_connections]\n",
+    "    valid_ids = set(node[\"id\"] for node in nodes)\n",
+    "    links = [{\"source\": tag1, \"target\": tag2, \"value\": weight}\n",
+    "             for tag1, tag2, weight in edges\n",
+    "             if tag1 in valid_ids and tag2 in valid_ids]\n",
+    "\n",
+    "    if not nodes or not links:\n",
+    "        print(f\"  ⚠️ Not enough connections for '{target_term}', skipping.\")\n",
+    "        continue\n",
+    "\n",
+    "    # === EXPORT ===\n",
+    "    d3_data = {\"nodes\": nodes, \"links\": links}\n",
+    "    safe_term = re.sub(r'\\W+', '_', target_term.lower())\n",
+    "    output_file = os.path.join(output_dir, f\"tags_{safe_term}_poi.json\")\n",
+    "    \n",
+    "    with open(output_file, \"w\") as f:\n",
+    "        json.dump(d3_data, f, indent=4)\n",
+    "    \n",
+    "    print(f\"  ✅ Saved to {output_file}\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_79893/1420003123.py:13: DtypeWarning: Columns (50,51,54,55,56,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,458) have mixed types. Specify dtype option on import or set low_memory=False.\n",
+      "  df = pd.read_csv(file_path)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Exported 60330 nodes and 16921 links to public/json/nodes_all.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "from itertools import combinations\n",
+    "from collections import Counter, defaultdict\n",
+    "import json\n",
+    "import os\n",
+    "\n",
+    "# === CONFIG ===\n",
+    "file_path = \"data/model_subsets/all_models_poi_false.csv\"\n",
+    "output_file = \"public/json/nodes_all.json\"\n",
+    "min_link_threshold = 10  # Only keep edges with co-occurrence >= this\n",
+    "\n",
+    "# === LOAD DATA ===\n",
+    "df = pd.read_csv(file_path)\n",
+    "tag_columns = [f\"tag_{i}\" for i in range(1, 8)]\n",
+    "df_tags = df[tag_columns]\n",
+    "\n",
+    "# === COUNT INDIVIDUAL TAGS ===\n",
+    "all_tags = df_tags.values.flatten()\n",
+    "all_tags = [tag for tag in all_tags if pd.notna(tag)]\n",
+    "tag_counts = Counter(all_tags)\n",
+    "\n",
+    "# === CO-OCCURRENCE ===\n",
+    "co_occurrences = defaultdict(int)\n",
+    "for tags in df_tags.itertuples(index=False, name=None):\n",
+    "    tags = [tag for tag in tags if pd.notna(tag)]\n",
+    "    for tag1, tag2 in combinations(tags, 2):\n",
+    "        co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "# === Build Edges (Filtered by co-occurrence threshold)\n",
+    "edges = [\n",
+    "    {\"source\": list(pair)[0], \"target\": list(pair)[1], \"value\": weight}\n",
+    "    for pair, weight in co_occurrences.items()\n",
+    "    if weight >= min_link_threshold\n",
+    "]\n",
+    "\n",
+    "# === Build Nodes (All tags that appear, regardless of links)\n",
+    "nodes = [{\"id\": tag, \"size\": tag_counts[tag]} for tag in tag_counts]\n",
+    "\n",
+    "# === EXPORT ===\n",
+    "d3_data = {\"nodes\": nodes, \"links\": edges}\n",
+    "\n",
+    "os.makedirs(os.path.dirname(output_file), exist_ok=True)\n",
+    "with open(output_file, \"w\") as f:\n",
+    "    json.dump(d3_data, f, indent=4)\n",
+    "\n",
+    "print(f\"✅ Exported {len(nodes)} nodes and {len(edges)} links to {output_file}\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

jupyter_notebooks/Section_2-4_Figure_9_Training_tags_Sankey.ipynb

@@ -0,0 +1,203 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Sankey Diagram from model dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from pathlib import Path\n",
+    "current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_9667/181225218.py:8: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`\n",
+      "  df['has_tags'] = df['has_tags'].fillna(False)\n",
+      "/tmp/ipykernel_9667/181225218.py:9: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`\n",
+      "  df['has_sex_tags'] = df['has_sex_tags'].fillna(False)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "🟢 All Adapters with Tags → Explicit: 4970\n",
+      "🔵 All Adapters with Tags → Non-Explicit: 6181\n",
+      "✅ Sankey structure saved to sankey_tags_focus_output.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "import pandas as pd\n",
+    "# Load your dataset\n",
+    "df = pd.read_csv(\"data/CSV/all_models_with_tags.csv\", nrows=40000)\n",
+    "\n",
+    "\n",
+    "# Fill missing values\n",
+    "df['has_tags'] = df['has_tags'].fillna(False)\n",
+    "df['has_sex_tags'] = df['has_sex_tags'].fillna(False)\n",
+    "df['poi'] = df['poi'].fillna(False)\n",
+    "\n",
+    "# Normalize and map types\n",
+    "df['type_normalized'] = df['type'].str.lower().str.strip()\n",
+    "type_mapping = {\n",
+    "    'checkpoint': 'Checkpoint',\n",
+    "    'lora': 'LoRA',\n",
+    "    'locon': 'LOCON',\n",
+    "    'textualinversion': 'TextualInversion',\n",
+    "    'controlnet': 'Other',\n",
+    "    'vae': 'Other',\n",
+    "    'upscaler': 'Other',\n",
+    "    'poses': 'Other',\n",
+    "    'workflows': 'Other',\n",
+    "    'other': 'Other'\n",
+    "}\n",
+    "df['type_mapped'] = df['type_normalized'].map(type_mapping).fillna('Other')\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "# Identify Adapters\n",
+    "adapter_types = ['LoRA', 'LOCON', 'TextualInversion', 'Controlnet']\n",
+    "df['is_adapter'] = df['type_mapped'].isin(adapter_types)\n",
+    "adapter_df = df[df['is_adapter']]\n",
+    "\n",
+    "# Build Sankey links\n",
+    "sankey_links = []\n",
+    "\n",
+    "# Total → each model type\n",
+    "for t in df['type_mapped'].unique():\n",
+    "    count = df[df['type_mapped'] == t].shape[0]\n",
+    "    sankey_links.append({'source': 'Total', 'target': t, 'value': count})\n",
+    "\n",
+    "# Adapter types → Adapters\n",
+    "for t in adapter_types:\n",
+    "    count = df[df['type_mapped'] == t].shape[0]\n",
+    "    if count > 0:\n",
+    "        sankey_links.append({'source': t, 'target': 'Adapters', 'value': count})\n",
+    "# Redefine adapter_types to only include those relevant for POI analysis\n",
+    "adapter_types = ['LoRA', 'LOCON', 'TextualInversion']\n",
+    "df['is_adapter'] = df['type_mapped'].isin(adapter_types)\n",
+    "adapter_df = df[df['is_adapter']]\n",
+    "\n",
+    "# Adapters → has_tags / has_no_tags\n",
+    "tagged_df = adapter_df[adapter_df['has_tags'] == True]\n",
+    "no_tag_df = adapter_df[adapter_df['has_tags'] == False]\n",
+    "\n",
+    "# Add has_no_tags count\n",
+    "no_tag_count = no_tag_df.shape[0]\n",
+    "if no_tag_count > 0:\n",
+    "    sankey_links.append({'source': 'Adapters', 'target': 'has_no_tags', 'value': no_tag_count})\n",
+    "\n",
+    "# Now subdivide has_tags group\n",
+    "has_tag_count = tagged_df.shape[0]\n",
+    "if has_tag_count > 0:\n",
+    "    sankey_links.append({'source': 'Adapters', 'target': 'has_tags', 'value': has_tag_count})\n",
+    "\n",
+    "    for poi_val in [True, False]:\n",
+    "        poi_label = f'has_tags + POI {\"True\" if poi_val else \"False\"}'\n",
+    "        subset = tagged_df[tagged_df['poi'] == poi_val]\n",
+    "        poi_count = subset.shape[0]  # ✅ Fix: missing in original code\n",
+    "\n",
+    "        if poi_count > 0:\n",
+    "            sankey_links.append({'source': 'has_tags', 'target': poi_label, 'value': poi_count})\n",
+    "\n",
+    "            for explicit in [True, False]:\n",
+    "                explicit_label = f'{poi_label} + {\"explicit\" if explicit else \"non-explicit\"}'\n",
+    "                exp_count = subset[subset['has_sex_tags'] == explicit].shape[0]\n",
+    "                if exp_count > 0:\n",
+    "                    sankey_links.append({'source': poi_label, 'target': explicit_label, 'value': exp_count})\n",
+    "\n",
+    "# --- New Section: Count Models (Rows) Containing Specific Tags ---\n",
+    "\n",
+    "# --- Output total explicit and non-explicit from all adapters with tags ---\n",
+    "\n",
+    "explicit_total = tagged_df[tagged_df['has_sex_tags'] == True].shape[0]\n",
+    "non_explicit_total = tagged_df[tagged_df['has_sex_tags'] == False].shape[0]\n",
+    "\n",
+    "# Add to sankey_links (optional)\n",
+    "if explicit_total > 0:\n",
+    "    sankey_links.append({'source': 'has_tags', 'target': 'All Adapters has_tags + explicit', 'value': explicit_total})\n",
+    "\n",
+    "if non_explicit_total > 0:\n",
+    "    sankey_links.append({'source': 'has_tags', 'target': 'All Adapters has_tags + non-explicit', 'value': non_explicit_total})\n",
+    "\n",
+    "# Also print them clearly\n",
+    "print(f\"🟢 All Adapters with Tags → Explicit: {explicit_total}\")\n",
+    "print(f\"🔵 All Adapters with Tags → Non-Explicit: {non_explicit_total}\")\n",
+    "\n",
+    "\n",
+    "# Define target tags to check\n",
+    "target_tags = ['rape', 'loli', 'shota', 'lolicon']\n",
+    "\n",
+    "# Identify all tag columns\n",
+    "tag_columns = [col for col in df.columns if col.startswith('tag') and col[3:].isdigit()]\n",
+    "\n",
+    "# Lowercase tag values for matching\n",
+    "df_tags_lower = df[tag_columns].astype(str).apply(lambda col: col.str.lower().str.strip())\n",
+    "\n",
+    "# For each target tag, check how many rows contain it\n",
+    "for tag in target_tags:\n",
+    "    contains_tag = df_tags_lower.isin([tag])\n",
+    "    \n",
+    "    # Only count rows that are both explicit and contain the tag\n",
+    "    matching_rows = df[(df['has_sex_tags'] == True) & contains_tag.any(axis=1)]\n",
+    "    rows_with_tag = matching_rows.shape[0]\n",
+    "    \n",
+    "    if rows_with_tag > 0:\n",
+    "        sankey_links.append({'source': 'has_tags', 'target': f'has_tag_{tag}', 'value': int(rows_with_tag)})\n",
+    "\n",
+    "\n",
+    "# Save to CSV\n",
+    "sankey_df = pd.DataFrame(sankey_links)\n",
+    "sankey_df.to_csv(\"data/CSV/sankey_data.csv\", index=False)\n",
+    "print(\"✅ Sankey structure saved to sankey_tags_focus_output.csv\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/Section_2-4_Figure_9_ectract_LoRA_metadata_v2.ipynb

@@ -0,0 +1,414 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "f36422c8",
+   "metadata": {},
+   "source": [
+    "# LoRA metadata"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "8a2feb6e",
+   "metadata": {},
+   "source": [
+    "LoRA Metadata Processing Workflow\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Load CSV File        │ --> │ Read adapter metadata CSV file.                   │\n",
+    "│ Read Model Versions  │     │ Extract model version IDs and relevant data.      │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Download Adapter     │ --> │ Use stored download URLs to fetch adapter files   │\n",
+    "│ Files Using API      │     │ using rotating API keys.                          │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Parse Metadata       │ --> │ Extract safetensors metadata, such as training    │\n",
+    "│ from SafeTensor      │     │ images, model type, and architecture.             │\n",
+    "│ Files                │     │                                                   │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Store Parsed         │ --> │ Save extracted metadata into structured JSON      │\n",
+    "│ Metadata as JSON     │     │ files for later analysis.                         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Process JSON Files   │ --> │ Read saved JSON metadata, extract relevant        │\n",
+    "│ for Consolidation    │     │ details, and filter necessary attributes.         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Extract Training     │ --> │ Identify most frequent training tags, architectures│\n",
+    "│ Tags & Model Info    │     │ and systems used for model creation.               │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Save Consolidated    │ --> │ Store all processed metadata in a structured CSV  │\n",
+    "│ Metadata to CSV      │     │ format for final analysis.                        ���\n",
+    "└──────────────────────┘     └───────────────────────────────────────────────────┘\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "efc9939d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import json\n",
+    "import csv\n",
+    "import struct\n",
+    "import requests\n",
+    "from pathlib import Path\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "from pathlib import Path\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.font_manager import FontProperties\n",
+    "from matplotlib import font_manager\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "\n",
+    "# Define the current directory and important file paths\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# Define frequently used directories\n",
+    "\n",
+    "data_dir = current_dir.parent / 'data/csv/adapters.csv'\n",
+    "fonts_dir = current_dir.parent / 'misc/assets/fonts'\n",
+    "plots_dir = current_dir.parent / 'results/plots'\n",
+    "raw_data_dir = current_dir.parent / 'data/adapter_metadata/lora'  ### location of the LoRA metadata (JSON)\n",
+    "temp_dir = current_dir.parent / 'data/raw/adapters_safetensors'\n",
+    "misc_dir = current_dir.parent / 'misc'\n",
+    "\n",
+    "# File paths\n",
+    "adapters_csv = current_dir.parent / 'data/csv/adapters.csv'\n",
+    "output_json_dir = raw_data_dir\n",
+    "api_keys_file = misc_dir / 'credentials/civit.txt'\n",
+    "\n",
+    "# Ensure directories exist\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "\n",
+    "# Load fonts into Matplotlib\n",
+    "for font_path in font_paths:\n",
+    "    font_manager.fontManager.addfont(font_path)\n",
+    "\n",
+    "# Set default font family for plots\n",
+    "plt.rcParams['font.family'] = ['Noto Sans JP', 'Noto Sans SC', 'sans-serif']\n",
+    "\n",
+    "print('Paths and fonts initialized successfully.')\n",
+    "\n",
+    "print('Paths initialized successfully.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87a58593",
+   "metadata": {},
+   "source": [
+    "## Step 2: Download LoRA and extract *.safetensors metadata\n",
+    "This script downloads LoRA adapters from the filtered Civiverse-Models dataset and extracts the metadata found within the *.safetensors' data structure"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "abd3a0bc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import sys\n",
+    "import csv\n",
+    "import json\n",
+    "import struct\n",
+    "import time\n",
+    "import requests\n",
+    "import signal\n",
+    "import contextlib\n",
+    "from pathlib import Path\n",
+    "import re\n",
+    "\n",
+    "# === Paste your API keys here ===\n",
+    "API_KEYS = [\n",
+    "    \"399c06ea6d1b7349556a115376ec346b\",  #DISCORD\n",
+    "    \"213be9d373130f86e394c6fea4d75162\", #ASDD 1\n",
+    "    \"4f180c0c56334b74394b467c5e5b8201\", #ASDD 2\n",
+    "    \"bdfba7ac53290f66bc76130f25b74336\", #BSDD \n",
+    "    \"43294f4a27b388624a896db5a65f445a\"\n",
+    "]\n",
+    "if not API_KEYS or any(not isinstance(k, str) or not k.strip() for k in API_KEYS):\n",
+    "    raise ValueError(\"Please paste at least one valid API key into API_KEYS.\")\n",
+    "\n",
+    "# === Config (adjust paths as needed) ===\n",
+    "current_dir = Path.cwd()\n",
+    "output_json_dir = current_dir.parent / \"data/adapter_metadata/lora\"        # where JSON outputs go\n",
+    "temp_dir = current_dir.parent / \"data/raw/adapters_safetensors\"                            # where downloads go\n",
+    "csv_path = current_dir.parent / \"data/csv/adapters_poi_false_sfw.csv\"\n",
+    "\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "# === API key state ===\n",
+    "current_key_index = 0\n",
+    "\n",
+    "\n",
+    "\n",
+    "def safe_filename(name: str, max_length: int = 100) -> str:\n",
+    "    # Replace unsafe chars\n",
+    "    sanitized = re.sub(r'[^a-zA-Z0-9_\\-]', '_', name)\n",
+    "    # Truncate if too long\n",
+    "    if len(sanitized) > max_length:\n",
+    "        sanitized = sanitized[:max_length]\n",
+    "    return sanitized\n",
+    "\n",
+    "\n",
+    "def get_headers():\n",
+    "    global current_key_index\n",
+    "    return {\n",
+    "        \"Accept\": \"application/json\",\n",
+    "        \"Authorization\": f\"Bearer {API_KEYS[current_key_index].strip()}\"\n",
+    "    }\n",
+    "\n",
+    "def rotate_api_key():\n",
+    "    global current_key_index\n",
+    "    if current_key_index < len(API_KEYS) - 1:\n",
+    "        current_key_index += 1\n",
+    "        print(f\"🔁 Rotated to API key #{current_key_index + 1}\")\n",
+    "    else:\n",
+    "        raise Exception(\"All API keys have been exhausted.\")\n",
+    "\n",
+    "# === Utilities ===\n",
+    "def save_json(data, filename):\n",
+    "    with open(filename, 'w', encoding=\"utf-8\") as f:\n",
+    "        json.dump(data, f, indent=4, ensure_ascii=False)\n",
+    "\n",
+    "def parse_safetensors(file_path):\n",
+    "    # Minimal, tolerant metadata reader; returns {} on failure.\n",
+    "    try:\n",
+    "        with open(file_path, 'rb') as f:\n",
+    "            file_data = f.read()\n",
+    "        # Many safetensors use 8-byte header length; this code follows your original logic\n",
+    "        # (4-byte) but keeps the 8-byte skip. Keep if it's working in your dataset.\n",
+    "        metadata_size = struct.unpack('<I', file_data[:4])[0]\n",
+    "        metadata_bytes = file_data[8:8 + metadata_size]\n",
+    "        metadata_str = metadata_bytes.decode('utf-8', errors='replace')\n",
+    "        metadata = json.loads(metadata_str)\n",
+    "        return metadata.get('__metadata__', {})\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error parsing safetensors file: {e}\")\n",
+    "        return {}\n",
+    "\n",
+    "# === Timeout context ===\n",
+    "class TimeoutException(Exception):\n",
+    "    pass\n",
+    "\n",
+    "@contextlib.contextmanager\n",
+    "def time_limit(seconds):\n",
+    "    def signal_handler(signum, frame):\n",
+    "        raise TimeoutException(f\"Timed out after {seconds} seconds\")\n",
+    "    # Note: SIGALRM works on Unix-like OS; on Windows this will be a no-op.\n",
+    "    try:\n",
+    "        signal.signal(signal.SIGALRM, signal_handler)\n",
+    "        signal.alarm(seconds)\n",
+    "    except Exception:\n",
+    "        # Fallback: no hard alarm on non-Unix systems\n",
+    "        pass\n",
+    "    try:\n",
+    "        yield\n",
+    "    finally:\n",
+    "        try:\n",
+    "            signal.alarm(0)\n",
+    "        except Exception:\n",
+    "            pass\n",
+    "\n",
+    "# === Download with timeout, retries, backoff, and key rotation ===\n",
+    "def download_file(url, output_folder, timeout=30, overall_timeout=120, max_retries=3):\n",
+    "    filename = url.split(\"/\")[-1]\n",
+    "    output_path = os.path.join(output_folder, filename)\n",
+    "\n",
+    "    global current_key_index\n",
+    "    retries = 0\n",
+    "    backoff = 2\n",
+    "\n",
+    "    while current_key_index < len(API_KEYS):\n",
+    "        try:\n",
+    "            with time_limit(overall_timeout):  # global cap per download\n",
+    "                #print(f\"➡️  GET {url} using key #{current_key_index + 1}\")\n",
+    "                resp = requests.get(\n",
+    "                    url,\n",
+    "                    headers=get_headers(),\n",
+    "                    stream=True,\n",
+    "                    timeout=(10, timeout),  # (connect timeout, per-chunk read timeout)\n",
+    "                )\n",
+    "\n",
+    "                # Auth errors → rotate key\n",
+    "                if resp.status_code in (401, 403):\n",
+    "                    print(f\"❌ Auth {resp.status_code} with key #{current_key_index + 1}. Rotating.\")\n",
+    "                    rotate_api_key()\n",
+    "                    retries = 0\n",
+    "                    backoff = 2\n",
+    "                    continue\n",
+    "\n",
+    "                # Not found → bubble up as FileNotFoundError (do not rotate)\n",
+    "                if resp.status_code == 404:\n",
+    "                    raise FileNotFoundError(f\"Model not found at {url}\")\n",
+    "\n",
+    "                # Rate limit → either rotate or wait/backoff\n",
+    "                if resp.status_code == 429:\n",
+    "                    print(\"⏳ Rate limited (429).\", end=\" \")\n",
+    "                    if current_key_index < len(API_KEYS) - 1:\n",
+    "                        print(\"Rotating key.\")\n",
+    "                        rotate_api_key()\n",
+    "                        retries = 0\n",
+    "                        backoff = 2\n",
+    "                        continue\n",
+    "                    else:\n",
+    "                        print(f\"Waiting {backoff}s (no other keys).\")\n",
+    "                        time.sleep(backoff)\n",
+    "                        backoff = min(backoff * 2, 60)\n",
+    "                        continue\n",
+    "\n",
+    "                # Other HTTP errors → raise to RequestException path\n",
+    "                resp.raise_for_status()\n",
+    "\n",
+    "                # Save file\n",
+    "                with open(output_path, 'wb') as fh:\n",
+    "                    for chunk in resp.iter_content(chunk_size=8192):\n",
+    "                        if chunk:\n",
+    "                            fh.write(chunk)\n",
+    "\n",
+    "                return output_path, filename\n",
+    "\n",
+    "        except TimeoutException as e:\n",
+    "            # Hard overall timeout → propagate\n",
+    "            raise e\n",
+    "        except requests.exceptions.RequestException as e:\n",
+    "            # Network-ish errors: retry same key with backoff up to max_retries\n",
+    "            retries += 1\n",
+    "            if retries <= max_retries:\n",
+    "                print(f\"🌐 Network error (try {retries}/{max_retries}) with key #{current_key_index + 1}: {e}\")\n",
+    "                time.sleep(backoff)\n",
+    "                backoff = min(backoff * 2, 60)\n",
+    "                continue\n",
+    "            else:\n",
+    "                raise Exception(f\"Failed to download {url} after {max_retries} retries: {e}\")\n",
+    "\n",
+    "    # If we exit the loop, we truly ran out\n",
+    "    raise Exception(\"All API keys have been exhausted or failed.\")\n",
+    "\n",
+    "# === Main processing ===\n",
+    "def process_csv(csv_path):\n",
+    "    with open(csv_path, newline='', encoding='utf-8') as csvfile:\n",
+    "        reader = csv.DictReader(csvfile)\n",
+    "        for index, row in enumerate(reader):\n",
+    "            # Collect up to 20 version IDs; use the most recent\n",
+    "            version_ids = []\n",
+    "            for i in range(1, 21):\n",
+    "                k = f'version_id_{i}'\n",
+    "                if k in row and row[k]:\n",
+    "                    try:\n",
+    "                        version_ids.append(int(float(row[k])))\n",
+    "                    except ValueError:\n",
+    "                        print(f\"Invalid version_id value '{row[k]}' in row: {row}\")\n",
+    "\n",
+    "            if not version_ids:\n",
+    "                print(f\"No valid version IDs found in row: {row}\")\n",
+    "                continue\n",
+    "\n",
+    "            most_recent_version_id = str(max(version_ids))\n",
+    "            name = row.get('name', 'unknown')\n",
+    "            sanitized_name = safe_filename(name, max_length=100)\n",
+    "            new_json_file = os.path.join(\n",
+    "                output_json_dir,\n",
+    "                f\"{index:08d}_{most_recent_version_id}_{sanitized_name}.json\"\n",
+    "            )\n",
+    "\n",
+    "            # Skip if JSON already exists\n",
+    "            if os.path.exists(new_json_file):\n",
+    "                #print(f\"↩️  Skipping versionID {most_recent_version_id} (JSON already exists)\")\n",
+    "                continue\n",
+    "\n",
+    "            try:\n",
+    "                adapter_file, fname = download_file(\n",
+    "                    row['downloadUrl'], str(temp_dir),\n",
+    "                    timeout=30, overall_timeout=180\n",
+    "                )\n",
+    "                metadata = parse_safetensors(adapter_file)\n",
+    "\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                new_json_data = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": metadata,\n",
+    "                    \"versionID\": most_recent_version_id\n",
+    "                }\n",
+    "                save_json(new_json_data, new_json_file)\n",
+    "                #print(f\"✅ Created JSON for versionID {most_recent_version_id} with file {fname}\")\n",
+    "\n",
+    "            except FileNotFoundError as e:\n",
+    "                print(f\"⚠️ {e} — saving empty metadata.\")\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                empty_json = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": {},\n",
+    "                    \"versionID\": most_recent_version_id,\n",
+    "                    \"error\": \"Model not found (404)\"\n",
+    "                }\n",
+    "                save_json(empty_json, new_json_file)\n",
+    "            except Exception as e:\n",
+    "                print(f\"⚠️ Error processing versionID {most_recent_version_id}: {e}\")\n",
+    "                civitaidata = {\n",
+    "                    k: (int(v) if str(v).isdigit() else v)\n",
+    "                    for k, v in row.items()\n",
+    "                }\n",
+    "                empty_json = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": {},\n",
+    "                    \"versionID\": most_recent_version_id,\n",
+    "                    \"error\": str(e)\n",
+    "                }\n",
+    "                save_json(empty_json, new_json_file)\n",
+    "                print(f\"💾 Saved empty JSON for versionID {most_recent_version_id} due to failure.\")\n",
+    "\n",
+    "# === Run ===\n",
+    "if __name__ == \"__main__\":\n",
+    "    process_csv(csv_path)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/Section_2-4_Figure_9_extract_LoRA_metadata.ipynb

@@ -0,0 +1,557 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "781346f0-fab6-45f9-9193-dd547f201864",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T11:14:38.839305Z",
+     "iopub.status.busy": "2025-02-08T11:14:38.837963Z",
+     "iopub.status.idle": "2025-02-08T11:14:38.845161Z",
+     "shell.execute_reply": "2025-02-08T11:14:38.844121Z",
+     "shell.execute_reply.started": "2025-02-08T11:14:38.839221Z"
+    }
+   },
+   "source": [
+    "# Section 3-4: LoRA metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "90b42e45-31bc-4cbd-b820-cf2daccbd712",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:10:00.955922Z",
+     "iopub.status.busy": "2025-02-08T21:10:00.955285Z",
+     "iopub.status.idle": "2025-02-08T21:10:00.959160Z",
+     "shell.execute_reply": "2025-02-08T21:10:00.958709Z",
+     "shell.execute_reply.started": "2025-02-08T21:10:00.955902Z"
+    }
+   },
+   "source": [
+    "### How are models trained? \n",
+    "### What are the most common training tags? \n",
+    "### What tagging systems are used?"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "79b10960-c7c6-4aa6-b91e-f0caf5c2a7dd",
+   "metadata": {},
+   "source": [
+    "LoRA Metadata Processing Workflow\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Load CSV File        │ --> │ Read adapter metadata CSV file.                   │\n",
+    "│ Read Model Versions  │     │ Extract model version IDs and relevant data.      │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Download Adapter     │ --> │ Use stored download URLs to fetch adapter files   │\n",
+    "│ Files Using API      │     │ using rotating API keys.                          │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Parse Metadata       │ --> │ Extract safetensors metadata, such as training    │\n",
+    "│ from SafeTensor      │     │ images, model type, and architecture.             │\n",
+    "│ Files               │      │                                                   │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Store Parsed         │ --> │ Save extracted metadata into structured JSON      │\n",
+    "│ Metadata as JSON     │     │ files for later analysis.                         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Process JSON Files   │ --> │ Read saved JSON metadata, extract relevant        │\n",
+    "│ for Consolidation    │     │ details, and filter necessary attributes.         │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────��───────────────────────────┐\n",
+    "│ Extract Training     │ --> │ Identify most frequent training tags, architectures│\n",
+    "│ Tags & Model Info    │     │ and systems used for model creation.               │\n",
+    "└─────────┬────────────┘     └───────────────────────────────────────────────────┘\n",
+    "          ↓\n",
+    "┌──────────────────────┐     ┌───────────────────────────────────────────────────┐\n",
+    "│ Save Consolidated    │ --> │ Store all processed metadata in a structured CSV  │\n",
+    "│ Metadata to CSV      │     │ format for final analysis.                        │\n",
+    "└──────────────────────┘     └───────────────────────────────────────────────────┘\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fd202dbd-fc0f-45f1-9d88-94c7d3635207",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T20:54:00.478266Z",
+     "iopub.status.busy": "2025-02-08T20:54:00.477080Z",
+     "iopub.status.idle": "2025-02-08T20:54:00.481464Z",
+     "shell.execute_reply": "2025-02-08T20:54:00.480830Z",
+     "shell.execute_reply.started": "2025-02-08T20:54:00.478244Z"
+    }
+   },
+   "source": [
+    "### Define Common Paths, Fonts etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "id": "9da03b00",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T21:11:15.085872Z",
+     "iopub.status.busy": "2025-02-08T21:11:15.084865Z",
+     "iopub.status.idle": "2025-02-08T21:11:15.117479Z",
+     "shell.execute_reply": "2025-02-08T21:11:15.116955Z",
+     "shell.execute_reply.started": "2025-02-08T21:11:15.085846Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Paths and fonts initialized successfully.\n",
+      "Paths initialized successfully.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import json\n",
+    "import csv\n",
+    "import struct\n",
+    "import requests\n",
+    "from pathlib import Path\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "from pathlib import Path\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.font_manager import FontProperties\n",
+    "from matplotlib import font_manager\n",
+    "import pandas as pd\n",
+    "from collections import Counter\n",
+    "from concurrent.futures import ProcessPoolExecutor\n",
+    "\n",
+    "# Define the current directory and important file paths\n",
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "# Define frequently used directories\n",
+    "data_dir = current_dir.parent / 'data/CSV'\n",
+    "fonts_dir = current_dir.parent / 'misc/assets/fonts'\n",
+    "plots_dir = current_dir.parent / 'plots'\n",
+    "raw_data_dir = current_dir.parent / 'data/raw/adapter_metadata'\n",
+    "temp_dir = current_dir.parent / 'data/models/adapter_temp'\n",
+    "misc_dir = current_dir.parent / 'misc'\n",
+    "\n",
+    "# File paths\n",
+    "adapters_csv = data_dir / 'model_subsets/Civiverse_adapters_poi_false.csv'\n",
+    "output_json_dir = raw_data_dir\n",
+    "api_keys_file = misc_dir / 'api_keys.txt'\n",
+    "\n",
+    "# Ensure directories exist\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "# Font paths for Matplotlib\n",
+    "font_paths = [\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-ExtraBold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-Bold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-ExtraLight.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-Light.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-Medium.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-SemiBold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_JP/static/NotoSansJP-Regular.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-ExtraBold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-Bold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-ExtraLight.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-Light.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-Medium.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-SemiBold.ttf',\n",
+    "    fonts_dir / 'Noto_Sans_SC/static/NotoSansSC-Regular.ttf'\n",
+    "]\n",
+    "\n",
+    "# Load fonts into Matplotlib\n",
+    "for font_path in font_paths:\n",
+    "    font_manager.fontManager.addfont(font_path)\n",
+    "\n",
+    "# Set default font family for plots\n",
+    "plt.rcParams['font.family'] = ['Noto Sans JP', 'Noto Sans SC', 'sans-serif']\n",
+    "\n",
+    "print('Paths and fonts initialized successfully.')\n",
+    "\n",
+    "print('Paths initialized successfully.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9c6b84ec-c23e-4047-a288-cfab2f5d8fdb",
+   "metadata": {},
+   "source": [
+    "## Step 1: Download LoRA and extract *.safetensors metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e484c678-70a8-4b70-b0aa-9ce69414c43b",
+   "metadata": {},
+   "source": [
+    "This script downloads LoRA adapters from the filtered Civiverse-Models dataset and extracts the metadata found within the *.safetensors' data structure"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "ecf31f5c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T20:49:20.487791Z",
+     "iopub.status.busy": "2025-02-08T20:49:20.487464Z",
+     "iopub.status.idle": "2025-02-08T20:49:20.499881Z",
+     "shell.execute_reply": "2025-02-08T20:49:20.499471Z",
+     "shell.execute_reply.started": "2025-02-08T20:49:20.487771Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "output_json_dir = output_json_dir\n",
+    "temp_dir = temp_dir ### better delete these later for space efficiency\n",
+    "api_karussell = api_keys_file\n",
+    "os.makedirs(output_json_dir, exist_ok=True)\n",
+    "os.makedirs(temp_dir, exist_ok=True)\n",
+    "\n",
+    "# Load API keys\n",
+    "def load_api_keys(api_path):\n",
+    "    if not os.path.exists(api_path):\n",
+    "        raise FileNotFoundError(f\"API keys file does not exist: {api_path}\")\n",
+    "    with open(api_path, 'r') as file:\n",
+    "        return [line.strip() for line in file if line.strip()]\n",
+    "\n",
+    "api_keys = load_api_keys(api_karussell)  # Replace with the path to your API keys file\n",
+    "current_key_index = 0\n",
+    "\n",
+    "def get_headers():\n",
+    "    \"\"\"Generate request headers with the current API key.\"\"\"\n",
+    "    global current_key_index\n",
+    "    return {\n",
+    "        \"Accept\": \"application/json\",\n",
+    "        \"Authorization\": f\"Bearer {api_keys[current_key_index]}\"\n",
+    "    }\n",
+    "\n",
+    "def rotate_api_key():\n",
+    "    \"\"\"Rotate to the next API key.\"\"\"\n",
+    "    global current_key_index\n",
+    "    if current_key_index < len(api_keys) - 1:\n",
+    "        current_key_index += 1\n",
+    "    else:\n",
+    "        raise Exception(\"All API keys have been exhausted.\")\n",
+    "\n",
+    "# Function to parse .safetensors metadata\n",
+    "def parse_safetensors(file_path):\n",
+    "    try:\n",
+    "        with open(file_path, 'rb') as f:\n",
+    "            file_data = f.read()\n",
+    "        metadata_size = struct.unpack('<I', file_data[:4])[0]\n",
+    "        metadata_bytes = file_data[8:8 + metadata_size]\n",
+    "        metadata_str = metadata_bytes.decode('utf-8')\n",
+    "        metadata = json.loads(metadata_str)\n",
+    "        return metadata.get('__metadata__', {})\n",
+    "    except Exception as e:\n",
+    "        print(f\"Error parsing safetensors file: {e}\")\n",
+    "        return {}\n",
+    "\n",
+    "def save_json(data, filename):\n",
+    "    with open(filename, 'w') as f:\n",
+    "        json.dump(data, f, indent=4)\n",
+    "\n",
+    "def download_file(url, output_folder):\n",
+    "    \"\"\"Download file with API key rotation.\"\"\"\n",
+    "    filename = url.split(\"/\")[-1]\n",
+    "    output_path = os.path.join(output_folder, filename)\n",
+    "\n",
+    "    global current_key_index\n",
+    "    while current_key_index < len(api_keys):\n",
+    "        try:\n",
+    "            response = requests.get(url, headers=get_headers(), stream=True)\n",
+    "            if response.status_code == 401:  # Unauthorized\n",
+    "                print(f\"API key {current_key_index + 1} failed. Trying next key.\")\n",
+    "                rotate_api_key()\n",
+    "                continue\n",
+    "            elif response.status_code == 403:  # Forbidden\n",
+    "                print(f\"Access forbidden for API key {current_key_index + 1}. Rotating to the next key.\")\n",
+    "                rotate_api_key()\n",
+    "                continue\n",
+    "            response.raise_for_status()\n",
+    "\n",
+    "            # Save the file to the specified output folder\n",
+    "            with open(output_path, 'wb') as file:\n",
+    "                for chunk in response.iter_content(chunk_size=8192):\n",
+    "                    file.write(chunk)\n",
+    "            return output_path, filename\n",
+    "        except requests.exceptions.RequestException as e:\n",
+    "            print(f\"Error downloading file: {e}\")\n",
+    "            rotate_api_key()\n",
+    "    raise Exception(\"All API keys failed.\")\n",
+    "\n",
+    "def process_csv(csv_path):\n",
+    "    # Read the CSV and process each row\n",
+    "    with open(csv_path, newline='', encoding='utf-8') as csvfile:\n",
+    "        reader = csv.DictReader(csvfile)\n",
+    "        for index, row in enumerate(reader):\n",
+    "            version_ids = []\n",
+    "            for i in range(1, 21):\n",
+    "                key = f'version_id_{i}'\n",
+    "                if key in row and row[key]:\n",
+    "                    try:\n",
+    "                        version_ids.append(int(float(row[key])))\n",
+    "                    except ValueError:\n",
+    "                        print(f\"Invalid version_id value '{row[key]}' in row: {row}\")\n",
+    "\n",
+    "            if not version_ids:\n",
+    "                print(f\"No valid version IDs found in row: {row}\")\n",
+    "                continue\n",
+    "\n",
+    "            most_recent_version_id = str(max(version_ids))\n",
+    "\n",
+    "            try:\n",
+    "                adapter_file, filename = download_file(row['downloadUrl'], temp_dir)\n",
+    "                metadata = parse_safetensors(adapter_file)\n",
+    "                # Add all CSV data under 'civitaidata'\n",
+    "                civitaidata = {key: int(value) if value.isdigit() else value for key, value in row.items()}\n",
+    "                new_json_data = {\n",
+    "                    \"civitaidata\": civitaidata,\n",
+    "                    \"metadata\": metadata,\n",
+    "                    \"versionID\": most_recent_version_id\n",
+    "                }\n",
+    "                sanitized_name = row['name'].replace(\" \", \"_\").replace(\"/\", \"_\")\n",
+    "                new_json_file = os.path.join(\n",
+    "                    output_json_dir,\n",
+    "                    f\"{index:08d}_{most_recent_version_id}_{sanitized_name}.json\"\n",
+    "                )\n",
+    "                save_json(new_json_data, new_json_file)\n",
+    "                print(f\"Created new JSON for versionID {most_recent_version_id} with filename {filename}\")\n",
+    "            except Exception as e:\n",
+    "                print(f\"Error processing versionID {most_recent_version_id}: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "708131db-95d8-4427-9397-74c72d3edf48",
+   "metadata": {},
+   "source": [
+    "Uncomment the following to download and process model adapters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "51ce1943-e509-442d-8b92-1a67bc686471",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T20:49:21.547880Z",
+     "iopub.status.busy": "2025-02-08T20:49:21.547383Z",
+     "iopub.status.idle": "2025-02-08T20:49:21.550529Z",
+     "shell.execute_reply": "2025-02-08T20:49:21.550087Z",
+     "shell.execute_reply.started": "2025-02-08T20:49:21.547860Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "csv_path = adapters_csv \n",
+    "#process_csv(csv_path) #### UNCOMMENT HERE"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4c2c52ee-83ad-4269-a7bf-16225b71f427",
+   "metadata": {},
+   "source": [
+    "## Step 2: Accumulate training tags and compare with auto-tagging vocabulary"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "6de2d9a6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-02-08T20:49:31.218029Z",
+     "iopub.status.busy": "2025-02-08T20:49:31.217597Z",
+     "iopub.status.idle": "2025-02-08T20:49:32.646108Z",
+     "shell.execute_reply": "2025-02-08T20:49:32.645340Z",
+     "shell.execute_reply.started": "2025-02-08T20:49:31.218005Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Processed JSON files and summary saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/model_subsets/Section_6-5/Lora_metadata.csv\n",
+      "Tag summary saved to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/CSV/model_subsets/Section_6-5/LoRA_training_tags_acc.csv\n"
+     ]
+    }
+   ],
+   "source": [
+    "interrogator_vocab = current_dir.parent / 'misc/autotagging-vocabularies/clip_interrogator.csv'\n",
+    "deepbooru_vocab = current_dir.parent / 'misc/autotagging-vocabularies/deepbooru_tags.txt'\n",
+    "\n",
+    "\n",
+    "def load_tags():\n",
+    "    try:\n",
+    "        clip_interrogator_tags = pd.read_csv(\n",
+    "            interrogator_vocab, header=None, usecols=[0], on_bad_lines='skip'\n",
+    "        )[0].tolist()\n",
+    "    except pd.errors.ParserError as e:\n",
+    "        print(f\"Error reading 'clip_interrogator.csv': {e}\")\n",
+    "        clip_interrogator_tags = []\n",
+    "\n",
+    "    try:\n",
+    "        with open(deepbooru_vocab, 'r', encoding='utf-8') as file:\n",
+    "            deepbooru_tags = {line.strip() for line in file}\n",
+    "    except FileNotFoundError as e:\n",
+    "        print(f\"Error reading 'deepbooru_tags.txt': {e}\")\n",
+    "        deepbooru_tags = set()\n",
+    "\n",
+    "    return clip_interrogator_tags, deepbooru_tags\n",
+    "\n",
+    "clip_interrogator_tags, deepbooru_tags = load_tags()\n",
+    "\n",
+    "# Determines tagging system based on tag matches\n",
+    "def determine_tagging_system(tags):\n",
+    "    clip_interrogator_matches = sum(1 for tag in tags if tag.strip() in clip_interrogator_tags)\n",
+    "    deepbooru_matches = sum(1 for tag in tags if tag.strip() in deepbooru_tags or tag.strip().replace(' ', '_') in deepbooru_tags)\n",
+    "    \n",
+    "    if deepbooru_matches > clip_interrogator_matches:\n",
+    "        return 'deepbooru'\n",
+    "    elif clip_interrogator_matches > deepbooru_matches:\n",
+    "        return 'clip-interrogator'\n",
+    "    elif clip_interrogator_matches == 0 and deepbooru_matches == 0 and tags:\n",
+    "        return 'other'\n",
+    "    else:\n",
+    "        return 'no tag metadata'\n",
+    "\n",
+    "def process_json_file(file_path):\n",
+    "    try:\n",
+    "        with open(file_path, 'r', encoding='utf-8') as file:\n",
+    "            data = json.load(file)\n",
+    "    except (json.JSONDecodeError, IOError) as e:\n",
+    "        print(f\"Error processing file {file_path}: {e}\")\n",
+    "        return None\n",
+    "\n",
+    "    # Handle ss_tag_frequency - Ensure it's parsed correctly\n",
+    "    tag_frequency_raw = data.get('metadata', {}).get('ss_tag_frequency', {})\n",
+    "    if isinstance(tag_frequency_raw, str):\n",
+    "        try:\n",
+    "            tag_frequency_raw = json.loads(tag_frequency_raw)\n",
+    "        except json.JSONDecodeError:\n",
+    "            print(f\"Error decoding ss_tag_frequency in {file_path}\")\n",
+    "            return None  # Skip files with unreadable tag data\n",
+    "\n",
+    "    if not isinstance(tag_frequency_raw, dict) or not tag_frequency_raw:\n",
+    "        return None  # Skip files with empty or nonexistent ss_tag_frequency\n",
+    "\n",
+    "    filename = os.path.basename(file_path).replace('.json', '')\n",
+    "    modelspec_title = data.get('metadata', {}).get('ss_output_name', '')  # Using ss_output_name instead\n",
+    "    modelspec_architecture = data.get('metadata', {}).get('ss_network_module', '')\n",
+    "    ss_num_train_images = data.get('metadata', {}).get('ss_num_train_images', 0)\n",
+    "    ss_steps = data.get('metadata', {}).get('ss_steps', 0)\n",
+    "    ss_sd_model_name = data.get('metadata', {}).get('ss_sd_model_name', '')\n",
+    "\n",
+    "    # Extract first-level tag frequencies\n",
+    "    tag_frequency = next(iter(tag_frequency_raw.values()), {})  # Get the first nested dict\n",
+    "    tags = list(tag_frequency.keys())\n",
+    "    training_system = determine_tagging_system(tags) if tags else 'undetermined'\n",
+    "\n",
+    "    tag_items = list(tag_frequency.items())[:20]\n",
+    "    tag_data = {f'tag{i+1:02d}': tag_items[i][0] if i < len(tag_items) else None for i in range(20)}\n",
+    "    tag_data.update({f'tag{i+1:02d}_no': tag_items[i][1] if i < len(tag_items) else None for i in range(20)})\n",
+    "\n",
+    "    row = {\n",
+    "        'filename': filename,\n",
+    "        'modelspec_title': modelspec_title,\n",
+    "        'modelspec_architecture': modelspec_architecture,\n",
+    "        'ss_num_train_images': ss_num_train_images,\n",
+    "        'ss_steps': ss_steps,\n",
+    "        'ss_sd_model_name': ss_sd_model_name,\n",
+    "        'training_system': training_system\n",
+    "    }\n",
+    "    row.update(tag_data)\n",
+    "\n",
+    "    return row, tag_frequency\n",
+    "\n",
+    "def parallel_process_files(folder_path):\n",
+    "    rows = []\n",
+    "    tag_occurrences = Counter()\n",
+    "    json_files = [os.path.join(root, file) for root, _, files in os.walk(folder_path) for file in files if file.endswith('.json')]\n",
+    "\n",
+    "    with ProcessPoolExecutor(max_workers=4) as executor:\n",
+    "        results = executor.map(process_json_file, json_files, chunksize=1)  # Using small chunksize for debugging\n",
+    "\n",
+    "    for result in results:\n",
+    "        if result:\n",
+    "            row, tag_frequency = result\n",
+    "            rows.append(row)\n",
+    "            tag_occurrences.update(tag_frequency)\n",
+    "\n",
+    "    total_tags_count = sum(tag_occurrences.values())\n",
+    "    unique_tags_count = len(tag_occurrences)\n",
+    "\n",
+    "    return rows, total_tags_count, unique_tags_count, tag_occurrences\n",
+    "\n",
+    "def write_summary_and_csv(rows, total_tags_count, unique_tags_count, tag_occurrences, output_file, tag_summary_file):\n",
+    "    if not rows:\n",
+    "        print(\"No valid data to write to CSV.\")\n",
+    "        return\n",
+    "\n",
+    "    df = pd.DataFrame(rows)\n",
+    "    df.to_csv(output_file, index=False, encoding='utf-8')\n",
+    "\n",
+    "    tag_summary_df = pd.DataFrame(list(tag_occurrences.items()), columns=['Tag (ss_tag_frequency/tag_frequency)', 'No. of Occurrences'])\n",
+    "    tag_summary_df = tag_summary_df.sort_values(by='No. of Occurrences', ascending=False)\n",
+    "    tag_summary_df.to_csv(tag_summary_file, index=False, encoding='utf-8')\n",
+    "\n",
+    "    print(f\"Processed JSON files and summary saved to {output_file}\")\n",
+    "    print(f\"Tag summary saved to {tag_summary_file}\")\n",
+    "\n",
+    "\n",
+    "def main():\n",
+    "    folder_path = '/home/lauwag/shares/laura_wagner/Civitai_page_analysis/Civitai_visualizations/data/raw/adapter_metadata'  # Path to your JSON files\n",
+    "    \n",
+    "    output_dir = current_dir.parent / 'data/CSV/model_subsets/Section_6-5/'\n",
+    "    os.makedirs(output_dir, exist_ok=True)  # Ensure the directory exists\n",
+    "\n",
+    "    output_file = output_dir / 'Lora_metadata.csv'\n",
+    "    tag_summary_file = output_dir / 'LoRA_training_tags_acc.csv'\n",
+    "\n",
+    "    rows, total_tags_count, unique_tags_count, tag_occurrences = parallel_process_files(folder_path)\n",
+    "    write_summary_and_csv(rows, total_tags_count, unique_tags_count, tag_occurrences, output_file, tag_summary_file)\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/SuppM_Figure_13_Danbooru_categories.ipynb

@@ -0,0 +1,141 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from pathlib import Path\n",
+    "\n",
+    "current_dir = Path.cwd()\n",
+    "input_json = current_dir.parent / \"misc/lists/danbooru.json\"\n",
+    "output_json = current_dir.parent / \"public/json/danbooru_flat.json\"\n",
+    "\n",
+    "# ---- CATEGORY COLORS ----\n",
+    "CATEGORY_COLORS = {\n",
+    "    \"attire_and_body_accessories\": \"#DC143C\",\n",
+    "    \"body\": \"coral\",\n",
+    "    \"characters\": \"silver\",\n",
+    "    \"copyrights\": \"#264653\",\n",
+    "    \"creatures\": \"silver\",\n",
+    "    \"drawing software\": \"#219ebc\",\n",
+    "    \"games\": \"silver\",\n",
+    "    \"metatags\": \"silver\",\n",
+    "    \"more\": \"silver\",\n",
+    "    \"objects\": \"#6d6875\",\n",
+    "    \"plant\": \"#7cb518\",\n",
+    "    \"real_world\": \"#a5a58d\",\n",
+    "    \"sex\": \"#ef476f\",\n",
+    "    \"visual_characteristics\": \"#06d6a0\",\n",
+    "    \"subject\": \"#ffd166\",\n",
+    "    \"uncategorized\": \"#adb5bd\",\n",
+    "    \"actions_and_expressions\": \"#d00000\",\n",
+    "    \"objects_and_backgrounds\": \"#118ab2\"\n",
+    "}\n",
+    "\n",
+    "# ---- STEP 1: Load the original nested JSON ----\n",
+    "with open(input_json, \"r\", encoding=\"utf-8\") as f:\n",
+    "    nested_data = json.load(f)\n",
+    "\n",
+    "# ---- STEP 2: Recursively extract paths and limit tags ----\n",
+    "def extract_tags(data, path=None, result=None):\n",
+    "    if path is None:\n",
+    "        path = []\n",
+    "    if result is None:\n",
+    "        result = []\n",
+    "\n",
+    "    if isinstance(data, dict):\n",
+    "        for key, value in data.items():\n",
+    "            extract_tags(value, path + [key], result)\n",
+    "    elif isinstance(data, list) and len(data) > 0:\n",
+    "        limited_tags = [data[0]] + [\"...\"] if len(data) > 1 else data\n",
+    "        result.append({\n",
+    "            \"level_path\": \" > \".join(path),\n",
+    "            \"top_tags\": limited_tags,\n",
+    "            \"tag_count\": len(data)\n",
+    "        })\n",
+    "\n",
+    "    return result\n",
+    "\n",
+    "# ---- STEP 3: Build the nested structure ----\n",
+    "def build_tree_from_flat_list(flat_data):\n",
+    "    tree = {\"name\": \"root\", \"children\": []}\n",
+    "\n",
+    "    for row in flat_data:\n",
+    "        parts = row['level_path'].split(\" > \")\n",
+    "        current = tree\n",
+    "        for i, part in enumerate(parts):\n",
+    "            match = next((child for child in current.get(\"children\", []) if child[\"name\"] == part), None)\n",
+    "            if not match:\n",
+    "                match = {\"name\": part, \"children\": []}\n",
+    "                current.setdefault(\"children\", []).append(match)\n",
+    "\n",
+    "                # If this is the top-level category, set its color\n",
+    "                if current[\"name\"] == \"root\":\n",
+    "                    color = CATEGORY_COLORS.get(part.lower(), \"#888888\")\n",
+    "                    match[\"color\"] = color\n",
+    "                else:\n",
+    "                    # Inherit from parent\n",
+    "                    match[\"color\"] = current.get(\"color\", \"#888888\")\n",
+    "\n",
+    "            current = match\n",
+    "\n",
+    "        current[\"tags\"] = row[\"top_tags\"]\n",
+    "        current[\"direct_tag_count\"] = row[\"tag_count\"]\n",
+    "\n",
+    "    return tree\n",
+    "\n",
+    "# ---- STEP 4: Recursively compute tag and category sizes ----\n",
+    "def compute_sizes(node):\n",
+    "    tag_count = node.get(\"direct_tag_count\", 0)\n",
+    "    category_count = 0\n",
+    "\n",
+    "    for child in node.get(\"children\", []):\n",
+    "        compute_sizes(child)\n",
+    "        tag_count += child.get(\"tag_count\", 0)\n",
+    "        category_count += 1 + child.get(\"category_count\", 0)\n",
+    "\n",
+    "    node[\"tag_count\"] = tag_count\n",
+    "    node[\"category_count\"] = category_count\n",
+    "\n",
+    "# ---- STEP 5: Generate and save final JSON ----\n",
+    "flat_data = extract_tags(nested_data)\n",
+    "tree_data = build_tree_from_flat_list(flat_data)\n",
+    "compute_sizes(tree_data)\n",
+    "\n",
+    "with open(output_json, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(tree_data, f, ensure_ascii=False, indent=2)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

jupyter_notebooks/SuppM_Figure_S12_asset_types.ipynb

@@ -0,0 +1,129 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "c0d18a6a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.ticker import FuncFormatter\n",
+    "from pathlib import Path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "98d25755",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()\n",
+    "\n",
+    "def sortByFrequency_model_types_csv(csv_path, output_svg_path):\n",
+    "    hatch_pattern = '\\\\\\\\\\\\\\\\\\\\\\\\'  # Hatch pattern for the bars\n",
+    "\n",
+    "    # Read the CSV file\n",
+    "    df = pd.read_csv(csv_path)\n",
+    "\n",
+    "    if 'type' not in df.columns:\n",
+    "        return \"The CSV file does not contain a 'type' column.\"\n",
+    "\n",
+    "    # Count the occurrences of each model type\n",
+    "    type_counts = df['type'].value_counts().reset_index()\n",
+    "    type_counts.columns = ['Type', 'Count']\n",
+    "    total = type_counts['Count'].sum()\n",
+    "    type_counts['Percentage'] = (type_counts['Count'] / total * 100).round(2)\n",
+    "\n",
+    "    # Sort the data in ascending order\n",
+    "    type_counts = type_counts.sort_values(by='Count', ascending=True)\n",
+    "\n",
+    "    # Plotting\n",
+    "    plt.figure(figsize=(10, 3.5))\n",
+    "    bars = plt.barh(type_counts['Type'], type_counts['Count'], color='white', hatch=hatch_pattern, edgecolor='coral')\n",
+    "    plt.xlabel('Counts', fontweight='bold')\n",
+    "    plt.ylabel('Asset Type', fontweight='bold')\n",
+    "\n",
+    "    ax = plt.gca()\n",
+    "\n",
+    "    # Hide all axis spines\n",
+    "    for spine in ax.spines.values():\n",
+    "        spine.set_visible(False)\n",
+    "\n",
+    "    # Keep ticks visible\n",
+    "    ax.xaxis.set_ticks_position('bottom')\n",
+    "    ax.yaxis.set_ticks_position('left')\n",
+    "\n",
+    "    # Bold tick labels\n",
+    "    for label in ax.get_xticklabels() + ax.get_yticklabels():\n",
+    "        label.set_fontweight('bold')\n",
+    "\n",
+    "    # Format x-axis ticks: 25000 → 25 k\n",
+    "    ax.xaxis.set_major_formatter(FuncFormatter(lambda x, _: f'{int(x/1000)} k' if x >= 1000 else f'{int(x)}'))\n",
+    "\n",
+    "    # Add percentage labels to the bars\n",
+    "    for bar, percentage in zip(bars, type_counts['Percentage']):\n",
+    "        plt.text(bar.get_width() + 5, bar.get_y() + bar.get_height()/2,\n",
+    "                 f'  {percentage}%', va='center', color='blueviolet', fontweight='bold')\n",
+    "\n",
+    "    plt.tight_layout()\n",
+    "    plt.savefig(out_file, format='svg')\n",
+    "    plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "9e610dec",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA94AAAFUCAYAAADS2eS8AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAw3xJREFUeJzs3Xd4FNX6wPHv7G6y6Q0CIUBCDQQQkGJAWlBpIgoIVhREf3hRrFe9otJURFEsV0UvwqWKBcSChQ6hhg6GGkognZbeNlvO74+52bAmQIKEIL6f55knkzNnzpyzIXl45zRNKaUQQgghhBBCCCFElTBUdwWEEEIIIYQQQojrmQTeQgghhBBCCCFEFZLAWwghhBBCCCGEqEISeAshhBBCCCGEEFVIAm8hhBBCCCGEEKIKSeAthBBCCCGEEEJUIQm8hRBCCCGEEEKIKiSBtxBCCCGEEEIIUYUk8BZCCCGEEEIIIaqQBN5CCCGEEEIIIUQVksBbCCGEEEIIIYSoQhJ4CyGEEEIIIYQQVUgCbyGEEEIIIYQQogpJ4C2EEEIIIYQQQlQhCbyFEEIIIYQQQogqJIG3EEIIIYQQQghRhSTwFkIIIYQQQgghqpAE3kIIIYQQQgghRBWSwFsIIYQQQgghhKhCEngLIYQQQgghhBBVSAJvIYQQQgghhBCiCpmquwJCVImsM1CQU921EEIIIYQQ4vrj5QcBwdVdi78UTSmlqrsSQlxxk+8Dq6W6ayGEEEIIIcT1x+QOYz6pUPCdmWhj06fZJO+wUJBhx93bQI0mbnR42Jemt3i65D2yuoAd8/I4dbAYZQef2kZaD/Ym6jG/ClXrWEwhS5486/z+uZ31MJk1AOxWxe6v84hbkk92sg2TWaNBFw96POePb4jeH11c4GDlG5kcjynCYILI272IfjEAg1EvIzvFxuyB6fSeGEiL/t4VqlMJ6fEW1yerBQY/C2dTYf23ENkZug4GTdOv/foFZKRD/1FQK0y/J/EArJgLYZFw6zAwmsBug9ULIPEg9B4OYS30vKcT4ZcZEBQCt/8fuJlBKdi4BA5uge73QPOb9LxZZ+Dnz8DdA+4YDV6+evquVbBjGXToC+1u09MKcvW8xUV63pI/Zoe2STukHdIOaYe0Q9oh7ZB2SDukHdXfjlXzICFOH116icBbKcWiUafJTrZjdIcaTdzITrGRvMNC8k4LwxfVplZzdwC2z8lh3XvZAHjXNOAdbKQgw8HJ2KIKBd75Z+0sG5dxwevLJ2aw/8cCAGo2MZF/1sHBXwpI2W1hxHchmH0NbP0ihwNLC7hnVjDZSTaWT8ykRhM32gzxAWDFpEzqdzBXOugu+TCEUEop1aNHDwWo8PDwC+b58ssvVXR0tAoICFBubm6qfv366uGHH1b79+93yTdhwgQFOA9N01RAQIDq0aOHWrVqVbllz54925nfYDCoxMTEy2/MhIFKpRzVz3euVGrCIKWWfqaU3a6nFRUoNfNlpd56QKmkw6X3Hd6u1OtDlPpqilLWYj3NWqx///oQ/XqJpMP6/TNf1stTSi9/6Wf683auLM17Jlmp90Yq9fEYpXIyStPXfavXdd23pWk5GXq+90bq95WQdkg7pB3SDmmHtEPaIe2Qdkg7qrsdyUdc/699ETlpVjW1ZaKa2jJRxc7MVkopdXJroTPt6Fq9zOxUq3qvjZ62c0GOcjgczjIsefZLPkcppRb947Sa1jZRLXnqjLN8a5HDWca7N+hpa9/NVEopVZhtVx90SFJTWyaqLf/R67b4H6fV1JaJylbsUOcSitXUlolq5Zv6z2Tfj3nqg45JKjvVWqH6/JEE3sLpUoH3o48+6gyM/fz8VPPmzZXRaFSA8vT0VMuXL3fmPT/wbtu2rWrTpo1L3vKC6pLnlxxvvPHG5TdmwkCl9m8p/V7+qEo7pB3SDmmHtEPaIe2Qdkg7pB1/vh0pRysceNttDjWjX6qa2jJRTbsxUc0ZkqY+6pyk3muTqH577Zyy2/TAeMe8HDW1ZaL6oEOSWvrSWfXvm5PVpz2S1c//Oqtyz9gu+ZydC/T7t83JURs/ySoTeBfl2tXUVnraummZelpOaeD99SOnlFJKrf8wU01tmahOxBaqvYty1dSWiWrPolyVn2FTH3dNVjvm51yyLhcigbdwuljg/d133zkD4rvuuksVFRUppZT6/fffVc2aNRWgateurfLz85VSroF3QkKCUkqpWbNmOdMWL17sUv7x48eVpmkKUB06dFCAatKkyeU3ZsJApd55WP6oSjukHdIOaYe0Q9oh7ZB2SDukHVeyHSf2VzjwVkrv9Z47NM0ZDE9tmag+6Zasdi4sDWJXvH7Oee29Nonqv3elOnvA5wxJU7ZixwXLP3OkWL3fLkl9+3+nlcPhKDfwVkrvES9J/+/ANPVx12Tn91/0T1VKKWXJt6ufXz6rPuqcpD7umqxWT8lQdptDLX3prJr/QLo6d7xYfT3ylPqoU5Kad2+6Sv29qEKfgVISeIvzXCzwHjhwoDNoPnHihMu184PsH374oUxaeYH3tm3byi0jJCRE7d6925lvw4YNl6x3UVGRys7OdjmKXh2g1Aej5I+qtEPaIe2Qdkg7pB3SDmmHtEPacSXbMf3ZCgfeDrtDLR6tB7yrp2QoS75dHVqe7wx441fpnXbLJpQG3vt+ylNK6UO7S9JObi284DNmD0pTn3RLdvaMXyjwLsyyqxVvZKjPbklRH3RIUguHn1Lz7tFfCPz3rtQLln98Q4Ga1jZRnY63qPn3pat/d05WCZsL1Yy+qeqzW1Mu+lLgfBJ4C6eLBd6RkZEKUAEBAWWuff/9985A+Z133lFKuQbebdu2VW3btlVGo1G5ubmpV155xeV+h8OhGjZsqAD1/PPPK6WUat26tQLUo48+esl6n/+skmNCj2ZKHdktf1SlHdIOaYe0Q9oh7ZB2SDukHdKOK9mON++rcOCdsKl0Pnf6AYsz/cOoJJf50xs/LQ2Wzx3X63DueLEzLe77vAs+o6SX/IMOSeqDDknOnvKSoeu7vip/eLjD4VAz79CHwX//zJly81jy7erzXilqw8dZypJnd8m7ZmqGmtoyUZ0+bCn33j+SwFs4VSTwDgwMLHPtxx9/vGjgff4RFhamtmzZ4nL/2rVrndd3796tlFLq3Xffdc4lLxm+fiEX7PFOOSp/VKUd0g5ph7RD2iHtkHZIO6Qd0o4r2Y7dayoceJ/fu71nUa5SSumLlv1hvnXSziJnvv1L9SB7/9LSHu+knfqQ7p1f5qiZd6SqmXeU9lCfP4S9vGP7PD3wPnO0WOWfK50vHjsr25nn0LLy443Vb2eoWQNSldXi0OeJt0xUP71wVv9RvJ8pgbe4PJc71HzixImXHGoeHx+vGjdurABVp04dlZdX+tZq+PDhzrz+/v7K399feXt7O9Pmz59f+cac/8dA/qhKO6Qd0g5ph7RD2iHtkHZIO6QdV6YdlVhcrSDTpv7dWZ9L/e4N+tzqkgXN3muTqNIPlgatJauRv9dGz1fSc/31o6ecec4fRn4hFxpqvvW/2eq9Nolq1oBUNb1nijPP90+fcVlFvUTq70XqvTalQb9SSs27N119fluKyjtjU/PuSVOf3SJDzcVlKAm8w8LCVGFhocuxePFiZyA8cOBA5+JqcXFxFV5c7aeffnKmTZ06VSmlVG5urkuQXd5x6623Vr4xEwYqtfTz0u/lj6q0Q9oh7ZB2SDukHdIOaYe0Q9rx59tRicBbKaXOHi1WS186qz6/LUVNuzFRfdojWS3+x2mVvLvIJZ+1yKHWvZ+pPrtVz/dF/1S14eMsVVxod+b5M4H38Q0Fas6QNPXhTUlqWlt9XvfWWdnKbi0bONutDjV7UJpa8fo5l/Rzx4vVwodPqQ86JKk5Q9JUyp6iMvdeiATewumP23mdf3zwwQdq5MiRLj3TkZGRzi3CPDw8LridWEng7XA4VKtWrZyLqBUWFrrs3b1v3z6X+nz44YeXv6f3hIHyR1XaIe2Qdkg7pB3SDmmHtEPaIe240u3Y8lOlAm+hk8BbOF0q8FZKqQULFqgePXoof39/ZTKZVN26ddVDDz1UJmguL/BWSqn58+c70z/99FPnMyMiIsrUJzEx8fL39C7p8ZY/qtIOaYe0Q9oh7ZB2SDukHdIOaceVa8ekuyXwvgyaUkohxPVm4iAY/CwkHoIdy6BDX2h3m36tIBd+/gyKi+CO0RAQrKcf2gbrv4XIztB1MGgaWC3w6xeQkQ79R0GtMD1v4gFYMRfCIuHWYWA0gd0GqxdA4kHoPRzCWuh5TyfCLzMgKARu/z9wM4NSsHEJHNwC3e+B5jfpebPO6HVz99Dr5uWrp+9aJe2Qdkg7pB3SDmmHtEPaIe2QdlR/O379AtKOwaj3ILQxomIk8BbXp8n36X9EhBBCCCGEEFeWyR3GfFIa0ItLksBbXJ+yzkBBTnXXQgghhBBCiOuPl58E3ZUkgbcQQgghhBBCCFGFTNVdASGqxLXW4y1vBYUQQgghhPjbkh5vcX261uZ4m9xgzKeVCr6L8x3MuTud7GQ7AL3GBdL2Xp+L5t/4cTbJuyzkpNqxFip8Q4w07+vFTSN9cfc2OPOejC1i68wczsRbseQ68AgwULetmZtH+xEc4Q7AsZhCYt7PIjvFTs3GJm59NZDQ1mZnGSvfyCB5p4WHF4VgdNMq+4kIIYQQQgjxtyE93tVoxIgRzJ07lx49erBu3boqf56m6cHR7NmzGTFiRJU/b86cOTzyyCMAXPX3O1aLvqq5yVz9qz+eOgk/fqz3wFci8F41OdMZdFdEYZaDnQvyMLpDUEM38k7byTxpY8t/ckg/UMyQz/RnZ5yw8t3oM9it4OFnoEYTN84esRK/spDknRZGrw2lOF+x9IVz1Gnjzv3zarFw2Gl+fO4co1eHApC8y8Lv3+Vz/9xaEnQLIYQQQghxCYZLZxGXq6ioiPfff5+oqCj8/Pzw8vIiIiKCxx9/nOPHj1d39apccHAwUVFRREVFVfreESNGoGka0dHRl1+BmvWgRScY+Za+9cGyWeAbpG97cMfj0PMBPfg+sktPa9IWHn1bHxb+2xd6sB7aGG65H+58Eg7Gwu5VENIQwlvo5YY0gN9mgsOu5+00AO57GZIOwabv9YC8dnilq35oWQH7fyqgWR/PCt9jNGv0+Kc/YzbUZcR3IfxjVSh12ui91wkbiijKdgCQFleM3arfc/fnNRm+KISox/wAPXi3FigyT1qxFipCb3DH099I7Uh38k7ZKci0Y7cqVkzMoO09PoS2MZdbFyGEEEIIIUQpCbyrSGZmJjfffDP//Oc/2bZtGwCNGzfm1KlTzJgxg/Xr11dzDate//79iY2NJTY2tnoqcEj/3KlZF4a/DpYCmDsecjP19B5D9eB77UKIWaSn+Qbqec1eet6zKXp6u9vgzidgxwr4dQY4HGD2hGHj9eB6/iRIjtfzRnSAe/8FR3bC4ml60F8JOWk2VryeQe0WbnR72r/C9/nUNHLTI37OIeUms0adlnrgrRnA8L/xLaGt3TG66effjT7L3KHpbJ2Zg9lX45axAZh9DQSGueHmqZEaV0xhtp1TB4vxqW3EK9DIlv/kYC1UdHu24nUTQgghhBDi70wC7yoyZswYdu/eDcCLL75IRkYGcXFxZGdnExMTQ7NmzVzyz5w5k4YNG+Lr68sdd9xBenq6y/UFCxbQsWNHvLy88PX1pW/fvuzZs8clT3p6OqNGjaJ+/fq4u7tTu3ZtHnjggQvW8fvvv8fNzQ1N05g8eTIADRo0QNM0Xn75ZcaMGUNQUBD+/v488cQTWCylc6YLCwt59dVXadKkCe7u7gQFBTFw4EDi4uKceebMmYOmac4h7gDR0dFomsbDDz/MhAkTqFOnDoGBgQwbNozc3FxnHebOnQtATEyMs4xKD8df/60+pByqN/hevaDCVVYOxa9jM3BY4Y6pNTCYLn8Yd/45O/GrCgFo3s/LGZAHhrtxz8xaeAUZKMp2cPqgFYcNfGsbqdlYj8g9/A0MeK8G+aftfH5rGm6eGne9X4Ozx6xsm5VDr3GB7P4qj89vS2V6dApr383EYZPlIoQQQgghhCiXEldcVlaWMplMClBt2rRRDoej3HzDhw9XgPL09FQeHh6qadOmClCAeuCBB5z53nnnHWd6RESECg0NVYDy9vZWBw4cUEopdfbsWRUeHu7M17RpUxUWFqYCAgKc5ZRcmz17tvrtt9+Uu7u7AtSUKVOceUrKMJvNqkaNGqphw4bO+5577jlnvttuu00BStM01bx5c+Xj46MA5ePjow4ePKiUUmr27NnOe0v06NFDAcrNzU35+vq6lP/KK68opZQaOHCgqlmzpgKUr6+vioqKUlFRUWrnzp3lfo5FRUUqOzvb5Sh6dYBSX7+j1IRBSu1cWZr5TLJS741U6uMxSuVklKav+1apCQP1ryVyMvR8743U7yuxc6Ve7tLPlLLb/1eJAqVmvqzUWw8olXS4NO/h7UpNulsvO+VoufU/3/a5OWpqy0S1d3GuUkqprGSrmtoyUU1tmah2f517yftLZJy0qi9uT1VTWyaqL4elK0uevbRZ6VY1o59+7eBv+cqSb1er385QU1smqvfbJ6nc07Zyy3TYHWrBg+lq6Ytn1bH1BWpqy0S14o0MteU/2ZWunxBCCCGEEH8n0uNdBeLj47HZ9OHF3bp1c+nxLY/FYiE2Npb4+HgGDRoEwOrVqwEoKChg0qRJAEyaNInDhw9z8uRJOnToQH5+Pm+99RYAn376KSdPngTg22+/JT4+npMnT7JmzZoyz1u/fj2DBw+muLiYd955h5dffrlMnrCwMBISEjh+/Dj333+/8xnZ2dmsXbuWVav0nuT333+fgwcPcvDgQXx8fMjLy2PKlCmX/Iw8PDw4ePAgR48epX379i5t/v777+nfvz8A7dq1cw5Xb9euXbllTZkyBX9/f5djysZ4fSG0Dr3hp+nV2/Pde/glP48Spw8XA7Dm7Sw+7JjM7IGlIx/WvJPJlw+eumQZKXssfPngKTJP2mgc7cHQGcEuK5rv+TqPrEQb7j4azft64e5loOWd3gDYihQpu8tfDX7313lknrBxy8sBnIwtAqDtPd60e1Bfaf3ElqIKt1MIIYQQQoi/Ewm8q4A6bwXvSwXdADfccANt2rQBoEWLFgCcOqUHWPv376egoACACRMmoGkabm5u7NixA8A5f3rr1q0ANGnShKFDhzrLvvHGG8s8b/bs2RQWFvLcc8/x0ksvlVunO+64A19ffQXw++67D4Di4mLi4+PZvn27M1/JUPZ69erRrVs3AGfdLuaWW26hbt26GAwGmjdv7tLmyho7dizZ2dkux9iuEfrq47ePqv7gO6xFpdtkLVTOo4S9GKxF+ve5p2zMGpDGrAFpxK8qcOY5vKKAbx89Q2Gmg3YP+DDo3zVx83T9Nbfk6WUU5ysyTuirrKXvL3Zed/Ms+282N93Gho+yiX4xAK8gIyX/xI1u2p8aDi+EEEIIIcTfgWwnVgWaNWuGyWTCZrOxceNGlFIXDcADAgKc5ybThX8kkZGR+Pn5uaTVqFGj0vUr6Zn+6quvePLJJ2ncuHGly/izymuzuswtx8xmM2bzH1bXNhn1Lb8MBj34Bj34Bj1oLgm+547Xj+Gv60F2j/+9tFi7UP/aY2hp8H1+3pp1S7chKyn39lGlwfeC1/Xg+6EJYDBWuC23T67B7ZNLf6bZKTZm9EkDXPfxdtggI0EfVVH8v0A677Sdn/55DhQY3SBtXzFfDjvtLKvXa4HUbuFO01s92f11HiiYN/QU/vVMnDumB+B+oUbqdyy7UvnKNzMJbWOm1V16z3iDTh7snJfH8Y1F+Ibo7QvvJCucCyGEEEIIUR7p8a4C/v7+3HPPPQDs3r2bV155xTn0HGDVqlVs3ry5QmW1bNkST099S6m+ffuyZcsW59Drzz77jFdffRXAuWXX0aNHWbJkifP+Py7ABvDGG2/QqlUr0tPT6dWrF2lpaWXy/PLLL+Tl5QH60HUAd3d3IiIi6NixozPfwoV6gJqcnMyGDRsA6NChQ4XadjFeXl4A5OfnX34hv34BlsLS4Lu6er5PJ15+GyrBblX6bHnAboW034tdDkuevp1YeCcPhnxWk/BOZty8NDJPWvGrY6T13d7cP7cWbh6ufxYOLSsgcauFXuMDnWmNunvS9Sl/ts3KYeWkTNo96EObIT5XpZ1CCCGEEEL85VTzHPPr1rlz51Tbtm2dC4f5+fmp1q1bq8DAQOcCZyWLq/Xo0cN534QJE8osSPbWW28500JDQ1WbNm1UUFCQAtSECROUUmUXV4uIiFANGjS44OJqSUlJql69egpQN9xwg8rI0BcaKynD29tb1axZUzVq1Mh53zPPPOMs6/zF1SIjI5Wvr2+lFlcbPny4M63kcwgPD3emffTRR857W7VqpaKiolRBQUHFfwATBir15n36gmdF/7vPbtcXRLvaC669eV+FF1cTQgghhBBCXH+kx7uKBAUFsWXLFt577z06duyIw+Hg8OHDBAYG8thjj9G9e/cKlzV27Fjmzp1Lx44dyczM5OjRo9SqVYt//OMfDB48GNCHnMfGxvJ///d/1K1bl+PHj1NQUEDfvn3LLbNevXosW7aMgIAA4uLi6N+/v3MuOcAzzzzDsGHDyMzMxNfXl8cff5y3337bef2nn37ilVdeoWHDhhw5cgSTycRdd93F5s2bnXO2/4yRI0dy99134+/vz759+9i6dSt2u71yhfQfpfc2L3i9enu+g0L+9OchhBBCCCGE+OvSlLrMibXiutSgQQNOnjzJhAkTmDhxYnVX5/JNHASDn9UD4F9m6MHv7f8HbmZQCjYugYNboPs90Pwm/Z6sM/DzZ+DuAXeMBi99cTl2rYIdy6BD39J53QW5et7iIj1vQLCefmibvn94ZGd9VXVNg7TjsHQ6jHoPQq/+fHohhBBCCCFE9ZLAW7i4bgLvyfeCtfjS+a4WNzM8+XFpgC6EEEIIIYT425BVzcX16clPoCCnumtRystPgm4hhBBCCCH+pqTHWwghhBBCCCGEqELS4y2uT1lnrp0eb+ntFkIIIYQQ4m9NerzF9WnyfWC1VHctdCZ3GPNJpYLv4nwHc+5OJztZX8m917hA2t578X2y93ybx4Gl+Zw+ZMVaqP9aj/wphBqN3MrNf+pgMV8+cAq7lTJ5j8UUEvN+Ftkpdmo2NnHrq4GEtjY77135RgbJOy08vCgEo5tW4XYJIYQQQgjxdyQ93tVkxIgRzJ07lx49erBu3brqrs4VMWfOHB555BEAqv19jtWir2oeGAKrF0DiQeg9HMJa6NdPJ16d1c5tFljyod77XonAe9XkTGfQXVEJG4s4fciKZ6ABa+HF77UWOfj5pXPOoPt8RTkOlr5wjjpt3Ll/Xi0WDjvNj8+dY/TqUACSd1n4/bt87p9bS4JuIYQQQgghKuAvs493gwYN0DTtoseVWIV7zpw5zvKutpI2RkdHX/VnXwnBwcFERUURFRVV3VXRBYZA/Wb6XtoRHWDlPMjL1Lf0atsThk+CrNOwaj7UCIW6TWDoC9ChD6xfBOkJet4WnWDkW2C3wbJZ4Bukp9/xuL7P945lcGSXntakLTz6tj68/LcvwGS+ZDX/6NCyAvb/VECzPp6Vuq/Xa4E8HVuXLk/4XzLv2qlZZCTYyn1G5km9xzz0Bnc8/Y3UjnQn75Sdgkw7dqtixcQM2t7jQ2ibyrdNCCGEEEKIv6O/TOB94403OoO6unXrOtPbtm3rTK9Xr1411vD6Zbfbsdsv3fvav39/YmNjiY2NvQq1qoDVC8BmBZMbDPknNG0P37wD8Tv06/Ui4KEJeu/3gtfBUggGA9w+Cjr0hp+m673aADXrwvDXwVIAc8dDbqae3mOoHnyvXQgxi/Q030A9r9lL7/2uhJw0Gytez6B2Cze6PX3pAPp8PrWMGIyXfmF0dF0he7/Np90DPjTqVjbwDgxzw81TIzWumMJsO6cOFuNT24hXoJEt/8nBWqjo9mzl6iaEEEIIIcTf2V8m8P7++++dQd1jjz1WJn358uXExcURHh6Ou7s79erV4/nnn6egoACAxMREAgIC0DSNSZMmAZCSkuJMGz9+PCNGjHAOlQbK9KSXfD9nzhxnnujoaDRNY8SIEc604cOH07RpU3x9fXF3dyc8PJynn36anJzKL/ZV0gv+r3/9izFjxlCjRg1q1arFM888g81mA6Bv375omsagQYOc9ymlCAsLQ9M0Xn75ZQAsFgsTJkygadOmuLu7U6tWLUaOHMnZs2ed902cOBFN02jQoAHz5s2jcePGuLu7k5SUxOHDh7nzzjupVasWZrOZevXq0a9fP7Zt2wZceLTA7Nmzad++PZ6ennh7e9OlSxd+/PFH5/UTJ064fLZ33HEHXl5eNGzYkFmzZlX6M3NKPAiLp1Vv8O3uUeHqKofi17EZOKxwx9QaGExXftRF3lk7y8dnULOpGz3+GVBuHg9/AwPeq0H+aTuf35qGm6fGXe/X4OwxK9tm5dBrXCC7v8rj89tSmR6dwtp3M3HYZKkIIYQQQgghLuQvE3hfTHFxMdHR0fz73//m9OnTREZGcu7cOT744AMGDBjgDEI//fRTAN566y3279/P448/TnZ2NjfddBPjx4+ncePGNGrUyFnu5fak//jjj2RmZtK4cWPq169PYmIiH3/8MY8++uhlt/GDDz7gq6++wtPTkzNnzvDvf/+b2bNnA3qgD/Dbb785g/stW7aQlJQE4HwpMHjwYF5//XUSEhKIjIzEYrEwe/ZsevToQWFhocvzUlNTGTFiBCaTidq1awNw//33s3TpUmw2Gy1btsThcLBs2TIOHDhwwXq/+eabjBw5kl27dlGrVi38/PzYvHkzAwcOZMGCBWXyjxo1iv379+Pm5saJEycYNWoUhw4duuhnY7FYyMnJcTksNrs+p/vIzuoNvu8YfdG6n2/ngjySdli45eUAghqUvyDan7VyUgbF+Yo7pgZhMl84sG/cw5ORP9XhuR31ePjbEOrc4M7yCRlE9PICDdZ/kE3jaE/aPeDLjrl5/P5dfpXUVwghhBBCiOvBdRF4f/XVV+zZswd3d3d+//139u7d6xzuvGbNGtasWQPAgw8+yL333ktxcTG33norv/zyC97e3ixYsACTycS4ceMYN26cs9zyetgrIiYmhrNnz7Jnzx6OHTvGq6++CsAPP/xAUVHRZbWxXr16HD9+nKNHjxIaqi9ytXr1agAGDhyIn58fFouFH374AYBvvvkGgJtuuonmzZsTExPDr7/+6vxM9u7dy6FDh/D09OTAgQMsXLjQ5XlWq5Xp06dz+PBhUlJSCAsL48iRIwAsXbqUXbt2kZqayvHjxy84Jz0/P5+33noLgEGDBpGQkMCJEye46SZ90bLXXnutzD133XUXx48fZ8OGDQA4HI5LLj43ZcoU/P39XY4pG+P1hdTu/Vf1Bt8lC7FVwOnDxQCseTuLDzsmM3tguvPamncy+fLBUxUu68LPsGK3Kr584DQfdkxmxesZzmvz7z1FzPtZ5d63++s8Mk/YuOXlAE7G6v+G297jTbsH9ZXWT2y5vH/XQgghhBBC/B1cF4F3yVDn4uJiIiIi0DSNtm3bOq+fP+f4s88+IzQ0lFOn9CDmvffeo2nTple0PqtWraJVq1Z4enqiaRqTJ08GwGazcebMmcsq884778Tf3x8PDw8aNmwI4GyDp6cn99xzDwBff/01DoeDRYv0wK+kt7vkMwLo0aMHmqYRGhrq7On+47xsT09PRo0aBehD7A0GAwMGDACgZ8+eREZGcvfdd7Ns2TLq1KlTbp3379/vLP++++7DYDBgNpu5++67ATh58mSZz+PBBx9E0zRatGjhTCtp54WMHTuW7Oxsl2Ns1wj9YkSH6g++K8laqJxHCXsxWIv073NP2Zg1II1ZA9KIX1VQ6fKVo/QZ9mLX59qLyw4Zz023seGjbKJfDMAryEjJgvVGN61KhsMLIYQQQghxvbmuthNzd3fnxhtvLJMeGBjoPM/IyHCZa3306NFKP+f8hcays7Ndrn355Ze88MILANSpU4f69etz9uxZjh8/XubeyggICHCem0z6j+38LbuGDx/OzJkzWbVqFT/++CNpaWmYzWbuu+++MmWVt+p4SEiIy/fBwcEYDK7vZebNm8edd97JunXrOHDgAL/++itLlixh3759zmH8f1ZJO0vaCJfemsxsNmM2/2GFbZNRD6BDG5cG39+8owffQ/5ZGnwvnqan3/svPV9J8D1/kh58DxsPZk89+AY9+AZ9+7CS4HvueP0Y/ro+vLzHUD3P2oWQm0FF3T65BrdPruH8PjvFxow+aYDrPt4OG2Qk6PP7i/NKP5uY97OIX1lIcb7Dmbb48TMYTBrtHvSh/TBfHl8R6vLMfT/k89treh0vtOf3yjczCW1jptVd3gA06OTBznl5HN9YhG+IEYDwTrLCuRBCCCGEEBdyXfR4d+zYEdCD2unTpzuHiK9bt44XX3yRBx54wHn9oYceIi8vjzZt2qBpGh988AExMTHOsry8vJzn+fmu81Zr1aoFQHx8PACHDh0iLi7OJU9Jz7Gvry8JCQls3bqV3r17X+EWl9W1a1caN26M1WrliSeeAPRe8pKXDiWfEeg9xCWf0caNG5k4cWKZ+eflbae2YcMGBg0axOeff8769euZMGECAOvXry+3Ti1btsTTU181+5tvvsHhcGCxWFiyZAkA4eHhBAdXfG/rSvtlBiTrP6tq6/nesazq2vcH+efsZCXZKMgoDbxz0vS0omzHRe68sEPLCkjcaqHX+NKXV426e9L1KX+2zcph5aRM2j3oQ5shPn+6/kIIIYQQQlyvrovA+/7776d169bY7XY6duxIq1ataNasGQEBAQwZMoSsrCxAnwu8ZcsWAgMD+e2333j88cdxOBwMHz7c2QvevHlzZ7ktWrSgU6dObNq0CYBbb70VgGnTptGzZ086d+5cpje2devWAOTm5tKoUSMaNWrEt99+W9UfAQAPP/wwAOnp+tzgkkXXQF99vU+fPoA+J7x58+a0bNmSgIAA+vXrx4kTJy5Z/kMPPURgYCDNmjXjxhtvZPz48UBpm//I29ubV155BYAlS5bQsGFDGjRowNatWwF94bUqFRSi91xXZ/Ddoe9lV9+/rokX99XnxX31nb3df0xvNdDbmX775BrO9D8eXZ4sf/uvVgO9nXnK6+1u3teLZ7fXI6Ce6+CYzo/78cS6uozZWJdbxwbKkHMhhBBCCCEu4roIvM1mMzExMTz99NPUr1+f+Ph4MjMz6dChA5MnT6Z27drs3LmT119/HYCPPvqIOnXq8O6779KwYUNOnjzJmDFjAD2IHDduHLVr1yYxMZGtW7eSmakHUu+//z79+/fH09OTY8eO8corr9C1a1eXujz66KM8//zz1KxZk9zcXKKjo53PrWoPP/yws6c6JCSEvn1dg74ffviB8ePH07RpU44fP056ejqRkZG89tprtGrV6pLlP/LII7Rs2ZKzZ89y4MABQkJCGDVqFJ988skF73nttdeYNWsW7dq14/Tp02RnZ9O5c2d++OEHhg0b9ucafCm3/x/UCqve4LvdbVXbRiGEEEIIIcQ1T1OXmkArxF/RxEEw+FnwD4Zfv4CMdOg/Sg/EARIPwIq5EBYJtw4DownsNli9QN//u/dwfVV00APvX2boPei3/x+4mUEp2LgEDm6B7vdAc32ldrLOwM+f6ft33zEaCrJhyYcw6j19vrkQQgghhBDib0cCb3F9evNesBVfOt/V4GaGJz+GgCqczy6EEEIIIYS4ZkngLa5PWWegIOfS+a4GLz8JuoUQQgghhPgbk8BbCCGEEEIIIYSoQtfVPt5COF3NHm/p0RZCCCGEEEJchPR4i+vT5PvAark6z6rEHO7fXssgeZeF/DN2ALxqGGjc3ZObn/TD0994wfty021s+U8OKXuKyT1lw2EF/7pGWt7lTfthvhjd9NXsbRbF8okZpO8rJuOEDRTUae3OsIW1XcqL+z6PLf/JoeCcg5BW7vSeEEhQg9LtxL4bfQaHHYbOkBcKQgghhBBC/FnS4y2uT1aLvqp5zXpwaBus/xYiO0PXwaBp+vUrsdr5z5/rZRXkVCjwPrq2ELOvRlBDE4WZDrKT7examEfGSRtD/3Ph+zMTbexdlI+bl0ZgmImsZBtnj9qImZZNdrKNXuOCAD3wPrC0AJ/aRsw+Gpbcsu/Vzh23snxCJi3v9KLb0wHMHpTOb69l8OACPTg/8Es+STssjPg+pHKfuRBCCCGEEKJc18U+3qJyoqOj0TSNBg0a/Omydu3axbBhwwgLC8NsNlO7dm2io6OZOXPmn6/on2Uy61t43XI/3PkkHIyF3asgpCGEt4CRb0FIA/htJjjset5OA+C+lyHpEGz6Xg/I6zeDYeP1PcBXzoO8TD1v255wxz8qVaXRa0IZtSyUh78N4fGVodRt5w5Ayu6L9857+BvoMzGQMRvrMnxxCI8vD8W/nt5DfuCXAmc+d2+N0WtDGb06lFrN3Mst6+wRK8oBoW3N+NQyEtTAxJnDVgAKs+ysfSeLLmP8Cagn7+WEEEIIIYS4EiTwFpdt5syZ3HTTTXz55ZckJydTr149fHx82LBhA2+++WZ1V0/fT/tsin7e7ja48wnYsQJ+nQEOB5g99YC6VhjMnwTJ8XreiA5w77/gyE5YPA1sVjC5wZB/QtP28M07EL9Dz1vSU15BJrPGxo+zWXD/Kf7TO5WUXfqWZ/XamS96X61m7rQe4oPJXR9S7uFvoGYTfWh4SRqAwajhE3zhIesANZu6oRkgdY+FvNN2Mk7YCG6ml7V2ahZ+oSbaD/OpVLuEEEIIIYQQFyaBtyijsLCQV199lSZNmuDu7k5QUBADBw4kLi7OmefQoUP84x//wG63Ex4ezu7duzl27BjHjh3j7NmzvPLKK868GRkZPPnkk9SvXx83Nzdq167NsGHDSExMdOaZOHGisxd+0aJFNG/eHG9vb7p3787hw4cvryHuHjB3fNUH35WUedJKWlwxOan6PO/wTmbunFajUmVkJFhJ3Kr3kre+27tS99Zo5EafSYEk7bAws38awU3d6PdGECe2FHHw1wJ6jQsk5v0spken8PltqWyddY1syyaEEEIIIcRflATeoow777yTt956i+PHj9O4cWOsVis//vgjN998M4cOHQJg1qxZ2O164PjBBx/Qpk0b5/2BgYGMGjUKgKKiInr06MH06dNJT08nIiKCnJwcvvzySzp37syZM2dcnp2SksKDDz6IpmkUFhayYcMGRo4cedH6WiwWcnJyXA6LzQ53jAazV9UG34kHKv35DnivJs/vrsfDi2tTs6kbJ2MtrHwzs8L3p8VZ+GrEaayFiqa3edLlSf9K1+GGQT6MWhbKs9vrcd+cWviGGFkxKYOOI3xJiytmx9w82j3gS5Oenqz/IJuEjYWVfoYQQgghhBBCJ4G3cLF27VpWrVoFwPvvv8/Bgwc5ePAgPj4+5OXlMWXKFAAOHCgNOLt3737B8r766iv27dsHwKJFi9i/fz+bNm3CYDCQmprKJ5984pLfZrPx3XffcfDgQZ599lkANm/eTGHhhQO/KVOm4O/v73JM2RgPXr4w/PWqDb5XzK3U51vC6KZRu7m7s7f6wNICMk5YL3nfkTWFfDPyDAXnHLQe6s2d02pgMGmXvO9SNn6Sg8GkcfNof07GFgHQ7kEf2gzV63diS9GffoYQQgghhBB/VxJ4Cxfbt293nj/wwAMA1KtXj27dugGwY4c+vPr8Xeg07cKBX0l5Xl5eDBw4EIB27drRrFkzl/JK+Pv7M2DAAABatGjhTD99+vQFnzF27Fiys7NdjrFdI/SLvoFVG3yHRV6wXn+UFmchcVtpAGu3KmeQC2AtVM58swakMWtAGmlxpYuu7Zyfy4/PnsVapOj+vD99JgRhMP75oPvUgWJ2fZlLnwmBmMwa/O9Ha3DTrkhQL4QQQgghxN+dBN7isrRs2dJ5vmHDhitWbkBAgPPcZCpdVfti282bzWb8/PxcDrPJCLv0nvsqDb5vHVbhtp07ZuObkWf4+OYU5tydzvToVI6t0wPvWs3dqPW/Bc6shYqMBBsZCTZnMJ6yx8Kad7JQDnD30jiyqpAFD5xyHnn/2xcc4It+aXzRL420OH3httOHip1puadsLnVy2BTLJmTQaqA39Tt6ABDeWV/o7fj6Qo6v10cahEd5VLidQgghhBBCCFcSeP+NKaUoKipyOdq3b++8vnDhQgCSk5OdwXWHDh0AGDlyJEajvnr2c88957LwWkZGhnMIeceOHQEoKCjghx9+APQtyEoWTCspr0rsWAYxi/Tzqgq+jRXfcqtmEzcadvXAaIZzx6zYihQ1GpnoOMKXe2fVQjNcuHfZXlz64qE4X5H2e7HLcf71rCQbWUk2bBb1v3tL0xyucTfb5+WSf9ZO9D8DnGlthvjQ7kEfVkzMZPucXLo+5U+j7p4VbqcQQgghhBDClaYu1pUorkvR0dHExMSUe+2DDz7gl19+YdWqVWiaRvPmzUlOTiY3NxcfHx+2b99O8+bNAX07sZKVzQ0GAw0bNkTTNE6cOEHdunU5ceIERUVFdOzYkX379mEymYiIiOD48eMUFRURGhrKnj17CA4OZuLEiUyaNInw8HBOnDgBwJw5c3jkkUcASEhIqNy+4xMHQYe+evDd8wHoMVRPz83UA29LgR6I16yrp+9aBT9Nhw694fZRYDCApRAWvA6nE+GhCVDvf8PX43foC6s1bQ9dBsGsl2HUe/re3kIIIYQQQgjxB9LjLcr46aefeOWVV2jYsCFHjhzBZDJx1113sXnzZmfQDfDYY4+xdetWHnjgAUJDQ0lMTCQzM5OoqCheffVVADw8PIiJieGJJ54gJCSE+Ph4fH19efDBB9myZQvBwcFV15B2t+lB99qFVdfzvXpB1dVfCCGEEEIIcV2QHm9xfZo4CAY/CzXr6b3ZO5bpPeDtbtOvF+TCz59BcZG+7VjA/14AHNoG67+FyM7QdTBoGlgt8OsXkJEO/UfpgTjoW4ktnwPKIT3eQgghhBBCiAuSwFtcn968R5+DfTWY3GDMp6XBuxBCCCGEEEKcRwJvcX3KOgMFOVfnWV5+EnQLIYQQQgghLkgCbyGEEEIIIYQQogpVfC8kIf5Kyuvxlp5pIYQQQgghRDWQHm9xfZp8n74o2vnczPDkxxUKvu1WRewXOez/KZ/cdDteNYw06+1J16f8cfe68GYAmz7NZvNnFx7iPmp5HfzrmjgWU0jM+1lkp9ip2djEra8GEtra7My38o0MkndaeHhRCEa3C+/vLYQQQgghhLj2yXZifwMjRoxA0zSio6Mvmm/58uW0bt0aDw8PNE1j4sSJTJw4EU3TKreH9rXAatFXNR/1HtzzL/Dw1tMqOO972bgMNk/PISfVTkB9EwXn7Oycn8eSJ86iHBd+V+Vb20id1u4uh4e//mtmdAcPPwNFOQ6WvnAO72Aj/1hdh+ICxY/PnXOWkbzLwu/f5dNnUpAE3UIIIYQQQlwHJPC+St599100TcNkMpGbm+tM79WrF5qm4ebmRn5+fpn0Hj16XJX6ORwO7rvvPuLi4vD19SUqKop69epdlWdXmbOp+hZfLTrBnWMqfNupA8Uc+LkAgFteDuDRpXW468OaACTtsHBkdeEF7209xIdhC2s7j3v/G4zBqF9reac3Zl8DmSetWAsVoTe44+lvpHakO3mn7BRk2rFbFSsmZtD2Hh9C25gv+BwhhBBCCCHEX4cE3ldJt27dALDb7WzevBkAm83Gli1bnOexsbFl0rt3737Zz7Tb7djt9grlTU1NJSsrC4AFCxYQGxvLY489dtnPvias/1bfwxsqNbf7+IYi53lELy8AGnf3wGTWe58TNhWVe1959v9YQEGGAzToONwXgMAwN9w8NVLjiinMtnPqYDE+tY14BRrZ8p8crIWKbs/6V/gZQgghhBBCiGubBN5XSfv27fH09ARgw4YNAOzevZv8/Hxq1arlkr5r1y5n73e3bt3IyMjgySefpH79+ri5uVG7dm2GDRtGYmKis/zzh4TPmzePxo0b4+7uTlJSUpm6nD59msjISDRN46abbuLDDz+kfv36zut9+/ZF0zTmzJlTblvsdjvTpk2jRYsWmM1m/P396dWrl7P+AF27dkXTNJ588knnMzVNQ9M0Dhw4AMD48ePRNI3mzZsDkJeXx+jRo6lfvz5ms5ng4GC6dOnC3LlzK/+BA0R2hp+mlwbfFZSbbnOeewXpvyKaQcMzQD/PSavYywzlUOyYp49uaBLtQVBDNwA8/A0MeK8G+aftfH5rGm6eGne9X4Ozx6xsm5VDr3GB7P4qj89vS2V6dApr383EYZOlGIQQQgghhPirksD7KnFzc6NTp05AaYBd8vX55593+X79+vUAGI1GOnfuTI8ePZg+fTrp6elERESQk5PDl19+SefOnTlz5ozLc1JTUxkxYgQmk4natWuXqUdmZia9e/fm0KFDREVFsXLlSpo2bUrbtm2deSIjI4mKiiI4uPxe4scff5wXXniBgwcPEhYWhslkYtWqVdxyyy3ExMQAOOeTb9q0yeUrwMaNG13aW5J3/PjxfP7555w5c4aWLVvi6+vL1q1bWbt27UU/W4vFQk5Ojsthsdmh62Do0FsPvg9tu2gZFVHZ0PfImkIyT+pBfMdH/FyuNe7hycif6vDcjno8/G0IdW5wZ/mEDL2HXYP1H2TTONqTdg/4smNuHr9/l1/eI4QQQgghhBB/ARJ4X0Ulw823bdtGcXGxM/AcOnQoTZs2JTY2FqvV6gy827Zty+LFi9m3bx8AixYtYv/+/WzatAmDwUBqaiqffPKJyzOsVivTp0/n8OHDpKSkEBYW5ryWl5dHv3792Lt3L506dWLFihX4+/vTv39/vv/+e2e+6dOnExsbS//+/cu04dixY/z3v/8F4JlnnuHIkSMcP36c8PBwbDYb48ePB0qD6bi4OHJycti4cSNGoxEvLy82btyI1Wpl27ZtLnmPHDkCwLhx49i1axfHjx/n9OnTPPfccxf9XKdMmYK/v7/LMWVjPGga3D5KD77Xf3uJn04p35DSXfYKMhyA3ntdlKWf+9UxVqic7XP03u46bdyp1+7i87V3f51H5gkbt7wcwMlYfSh723u8afegDwAntlR8eLsQQgghhBDi2iKB91VUMl+7qKiIbdu2sXHjRkJDQ2nUqBHdu3enoKCAHTt2OHuHu3Xrxvbt2wHw8vJi4MCBALRr145mzZoBsGPHDpdneHp6MmrUKAA0TcNgKP0R79y5k61btxIeHs7y5cvx83Ptha2InTt3UrID3QMPPACAv78/t99+u0t9br75Ztzd3XE4HGzZsoVNmzbRtm1bOnXqxMaNG9m1axcFBfoCZiWB94ABAwA98A4PD6dPnz58/PHH5fbcn2/s2LFkZ2e7HGO7RuirmBsMevAd2bnCbWzY1cN5Hr9Sr+Ox9UXYLHq7G3bRr6fFWZg1II1ZA9JIi3Pduixlt4XUPcUAdBzhe9Hn5abb2PBRNtEvBuAVZKRkgz+jm4bBJKuaCyGEEEII8VcngfdV1KlTJ0wmvTd1xowZnD171tkLXhKUf/7552RkZLikVUZwcLBLsH0+b29vAE6ePMn8+fMrXXZleHl50bFjRwBWrlzJrl276NKlC127diUhIYFvv9V7oJs1a0ZISAgAo0aNIiYmhueff57mzZuzc+dOJk6cyG233XbRZ5nNZvz8/FwOs8kIv34BlkI9+O46uMJ1D2npTuTt+qJqa97OYtaANH589iwA9dqbaXqrPlffWqjISLCRkWDDWug6EL2ktzsgzETE//JfyMo3MwltY6bVXfrPp0EnPbA/vrGIY+v1FdTDO8kK50IIIYQQQvxVSeB9FXl7e9OuXTsAvvrqK4AygffChQud+bt27eoMXgsKCvjhhx8AffG1w4cPA9ChQweXZ2jahXtIO3TowGuvvQbAU0895axDZbRv3975jJK6Zmdn8+uvv5apT0lP9syZM7Farc7AG/QXD+fnAX0IfsuWLXnvvfdYvnw5P//8MwD79+/n3LnSfa4rLCMdFryuB98X+VzK029yEJ3/4YdfHSNZSTa8goy0e9CHu6fXRDNcvKzMRCtH1+oBc4eHfS6a/9CyAhK3Wug1PtCZ1qi7J12f8mfbrBxWTsqk3YM+tBniU6n6CyGEEEIIIa4dpktnEVdSt27d2LZtGzabzfk9QIMGDahfv75zFfLmzZsTHBzM/fffz/vvv8++ffsYOnQoERERHD9+HIfDQWhoKGPGVHx/aoA33niDlJQUZs+ezfDhwwkICKBfv34Vvr9x48aMHDmSWbNm8dFHH/HLL7+QkZFBRkYGJpOJSZMmOfNGR0czefJksrOzAejSpQt+fn4YjUby8vKceUr8+9//5ptvvqFevXoEBQVx9OhRAOrWrUtQUFCl2glA/1Hw20w9+L7toUrdanTT6DrGn65jLrytV9hNHry4r36Z9MAwN174vWx6eZr39aJ5X68y6Z0f96Pz45WfCiCEEEIIIYS49kiP91VWEmgDBAQE0KpVK+f3PXr0KJPPw8ODmJgYnnjiCUJCQoiPj8fX15cHH3yQLVu2XHDl8YuZMWMG/fr1w2q1MmTIEJcVxyviP//5D++++y6RkZEkJiZitVq57bbbWLNmjUsgXTLPGyAsLIy6devi6+tL69atnXnOz9+/f3+6detGYWEhcXFxeHh4MGDAAH799deL9uRfUK0weGgCnE7Uh50LIYQQQgghRDXQVMlKWUJcTyYOgsHPQs16euD98+dgK4ZR70Fo4+qunRBCCCGEEOJvRIaai+uTyQ2WfFg2zUuGbwshhBBCCCGuLunxFtenrDNQkOOa5uUHAZUfmi+EEEIIIYQQf4YE3kIIIYQQQgghRBWSoebi+lTS4y293EIIIYQQQohqdlk93ocPH+att94iNjaWiIgIXn31VVasWMHgwYNdVukWotpMvg+sFnAzw5MfVyj4tlsVsV/ksP+nfHLT7XjVMNKst76ntrvXhTcAyE23seU/OaTsKSb3lA2HFfzrGml5lzfth/lidHNdkf33xXnsXZTHueP6lnL+dY10GO7LDYP0vbpjv8hh99d5FOc5CLvJTK8JQfjUNALgsCnm3XOKOq3d6TPxMrZYE0IIIYQQQlx1ld5ObO/evXTs2JEFCxZw5MgRzp07h4eHBxMnTuTzzz+vijqKcowYMQJN01y249I0DU3TmDNnzgXvW7dunTPfunXrqryeFxMdHV2mDVeM1QJdB+tf/zjX+wKWjctg8/QcclLtBNQ3UXDOzs75eSx54izKceH3U5mJNvYuyic7xYZ/qAnNCGeP2oiZls2atzNd8q56K5PlEzNJ32/FM8BAYLiJggwHKbuLATixuYgNH2VzwyBvhi2szbH1Rax7N8t5/7b/5lKQaSf6nwGV/kiEEEIIIYQQ1aPSgffLL79MXl4e7du3d6a1bduWoKAg1q5de0Urd71799130TQNk8lEbm6uM71Xr15omoabmxv5+fll0nv06EHjxo2JioqiRYsW1VH1v4ajuyuc9dSBYg78XADALS8H8OjSOtz1YU0AknZYOLK68IL3evgb6DMxkDEb6zJ8cQiPLw/Fv57eQ33glwJnvpQ9FnYvzEMzwF0f1uDxlaEMXxzCk+vrcsu/AgA4fUgPwOu1M1OjsRteQQbOHNbTMk9a2fKfHG57JRCzb6V/dYUQQgghhBDVpNL/e9+0aRN169Zly5YtLun169cnKSnpilXs76Bbt24A2O12Nm/eDIDNZnN+tjabjdjY2DLp3bt3Z9y4ccTGxjJ9+vRqqHnFFRcXV9/DT52scNbjG4qc5xG9vABo3N0Dk1kfJp6wqajc+wBqNXOn9RAfTO56Xg9/AzWbuAE40wAOL9eDcJ9aRvZ9n89HnZL5/LZUVr2VScmEj1rN3QFI3mXh3DErBRkOgpu5o5Ri+cRMGnb1cNZPCCGEEEII8ddQ6cDbbrfj4+OD0Wh0ST9z5gwOh+OKVezvoH379nh6egKwYcMGAHbv3k1+fj61atVySd+1a5ez97tbt27lDjUvz7fffkujRo3w9PTk9ttvJyUlpdx8O3bs4K677qJGjRqYzWYaNWrEtGnTACgsLGTgwIE0bNgQb29vzGYzTZs2Zfz48S6BdcnQ8YceeogXX3yRWrVq0axZMwAyMzO555578PLyIiws7ILTEubNm0fbtm3x9fXF19eXyMhIHnrooYp8nGV17FvhrLnpNue5V5D+a6EZNDwD9POcNHuFy8pIsJK41QJA67u9S9NP2P73LDsnt1rwCzGSe8rO7oV5/PKvcwA0uNmDbs/4E7cknwUPnKJRNw+iXwwg7rt8Th8qpvtz/vz22jk+6ZbCF/3S2PdjftkKCCGEEEIIIa4plQ68W7RoQXx8PG+++SYAOTk5vPDCC6SmpsrCapXk5uZGp06dgNIAu+Tr888/7/L9+vXrATAajXTu3LlC5e/Zs4f777+fhIQEzGYz8fHxPP7442Xybd68mS5duvDTTz+Rl5dH06ZNycnJcT7bYrHw448/UlhYSEREBLVq1eLo0aO88cYbvPrqq2XK+/bbb/noo4+oXbs2fn5+ADz22GMsWrSIwsJCvLy8eOGFF9ixY4fLfXv37mXEiBHs3buXkJAQGjRoQHJyMgsWLLhoOy0WCzk5OS6HxWaH2g0q9DldTGVXHkyLs/DViNNYCxVNb/Oky5P+zmsOW2lpQ2cE88gPdejypP75HIspIjtFD8w7/Z8fo9eE8szWegz+RF8Ubt37WfR4PoC4Jfns+6GAHs/7E9zMjWXjMjh71PrnGimEEEIIIYSoUpUOvJ955hmUUkyYMAFN0zh48CAffPABmqYxZsyYqqjjda1kuPm2bdsoLi52BrtDhw6ladOmxMbGYrVanYF3SW9wRUybNg2Hw4G/vz+HDx/m6NGjDB48uEy+1157jeLiYgICAoiLi2Pfvn2cPn2aSZMmAeDt7c3+/ftJT09n9+7dJCUlMWzYMAC+/vrrcp+9fft24uLi2LVrF8eOHWPJkiUA/Otf/+LQoUPs3LkTi8Xics/Ro0dRShEREcHhw4eJi4sjKyuLmJiYi7ZzypQp+Pv7uxxTNsZX6DMq4RtSurNeQYY+ckM5FEVZ+rlfHWO5953vyJpCvhl5hoJzDloP9ebOaTUwmEqHmvvWLi0jpJU+pLzODe7OtJLA+49WT84kOMKd1kO8ORlbhIe/gRsG+dBqoDfKAYlbLzwMXgghhBBCCFH9Kh14Dxs2jLfffhtPT0+UUiil8PDwYPLkyc5gTFRc9+7dASgqKmLbtm1s3LiR0NBQGjVqRPfu3SkoKGDHjh1s2rQJKA3UK2L//v0AdOnShdq1awN6QP9HW7duBWDIkCFEREQAYDAYaNOmjfN8wYIFREREYDab0TTN2QudmppapryePXs67zUajc56ANx9990ANGvWjNatW7vc16VLFwIDA4mPj6dGjRpERUXxxBNPXLKdY8eOJTs72+UY2zUCMk9d8t4SDbt6OM/jV+pzsY+tL8Jm0XupG3bRr6fFWZg1II1ZA9JIiyt9cbBzfi4/PnsWa5Gi+/P+9JkQhMHouo1YeKfSZ6TvK/7f1//1VmsQGGbij46sKeT4+iL6TAxE0zSUAqM+fRxj2exCCCGEEEKIa9Bl/df9pZde4qmnnnIGVC1btnTOVRaV06lTJ0wmEzabjRkzZnD27FnuvfdeQA/KZ82axeeff05GRoYz7Wp7++23mTJlCgDh4eGEhISQnJxMSkpKufP6S4L8ygoJCWH//v3Mnz+fnTt3EhcXx4wZM5g5cyabN28mKiqq3PvMZjNms9k10WSE2KUVf3ZLdyJv9+LgrwWseTuL3V/lkZWk90DXa2+m6a36v29roSIjweY8B3218jXvZAHg7q1xZFUhR1aVroI+8KOa+AQbadbHi53zc0nfb2Xx42fwr2dyDhO/YaC3S687gCXPwarJmXR+3I+gBnq0Hd7Jg+2zc0nfX8zx9UVoBqh/0x/aLoQQQgghhLimXPaeRBs2bCAmJoaYmBg2btx4Jev0t+Lt7U27du0A+Oqrr4DSXu2SIHvhwoXO/F27dq1w2S1btgT0lehPnz4NwOLFi8vkKwlov/vuO44ePQqAUorff/8dwLmyekREBCdOnGDTpk3OHu3yaJprT+/5W559//33AMTHxzvLL5GamsqZM2d46aWX+Oabbzhw4ADNmzfH4XBc3r8x36BKZe83OYjO//DDr46RrCQbXkFG2j3ow93Ta6IZtAveZy8unbtdnK9I+73Y5Si5bnTTGDqjFm2GeuPurZGVaKNmEzdu+VcAvScGlil3/QdZePobuGlk6dSCm//hR2R/L7597DRH1xXSZ2IgwU3dy9wrhBBCCCGEuHZUusc7OTmZQYMGsWvXLpf0G2+8ke+//5769etfscr9XXTr1o1t27Zhs9mc3wM0aNDAZZu25s2bExwcXOFyn3/+eb788kuys7OJiIggODi43C3f3nzzTXr27ElmZiYtW7YkIiKC9PR0unTpwg8//EDr1q35+eefiY+Pp2HDhlitVgoLL7yv9R81adKEgQMH8sMPPzBlyhS+//57kpKSMBqNzjYDHDhwgF69ehEcHExoaCg5OTkkJCQAcMMNN1T4eU6dBsBvX1Q4u9FNo+sYf7qO8b9gnrCbPHhxX/1Lpl2Ih7+B3hOC6D3h0nl7jSv74sDd28Ad79So0LOEEEIIIYQQ14ZK93iPGjWKnTt3Oud3lxy7d+8ud8VscWnnz9sOCAhwWR2+R48e5eariBtvvJGFCxfSoEEDioqKCA8P57PPPiuT7+abb2bTpk0MGDAAHx8fDh8+jI+Pj7N3/ZVXXmH48OEEBASQk5PDfffdV6G51+ebNWsWd999Nx4eHmRnZ/P66687V3Qv0ahRI+677z78/PyIj4/nzJkztGnThhkzZtC7d+9KPQ8Ak1vl7xFCCCGEEEKIK0xTSlVqxyRPT09sNhuffvop999/P6CvbD169Gjc3d0pKCiokooKUSkTB0G3IbBhMYx6D0IbV3eNhBBCCCGEEH9TlQ68w8PD8fX1Zd++fS7prVq1oqCggOPHj1/RCgpxWd68B2xWcDPDkx9DQMWH6AshhBBCCCHElVTpOd7/+te/eOmllzh06BDNmzcH4NChQyQkJPDxxx9f8QoKcVnGfAoFOeDlJ0G3EEIIIYQQolpVuse7Z8+exMbG4nA4nAtexcXFYTab6dChQ2nBmsbq1auvbG2FEEIIIYQQQoi/mEoH3gZDxdZj0zQNu91+WZUS4k/LOiM93kIIIYQQQohrQqWHmj/88MNl9mkW4prz6VNgtVRqjrfdqoj9Iof9P+WTm27Hq4aRZr096fqUP+5eF37hlJtuY8t/ckjZU0zuKRsOK/jXNdLyLm/aD/PF6Kb/vmSn2JjRJ63cMvpMDKT1EB8A4r7PY8t/cig45yCklTu9JwQS1KB0hfbvRp/BYYehM+SFghBCCCGEEH8FlQ6858yZUwXVEH91c+bM4ZFHHgGgkoMoqobVAh36wo5les93BQLvZeMyOPBzAZoBAsNNZCXZ2Dk/j9MHrdz732A0Q/kvnDITbexdlI+bl0ZgmImsZBtnj9qImZZNdrKt3P2467R2d/neq4YRgHPHrSyfkEnLO73o9nQAswel89trGTy4oDYAB37JJ2mHhRHfh1T2ExFCCCGEEEJUk0rv492iRQumTp1KSkpKVdRHVJOMjAxefvllIiMj8fT0xMfHh7Zt2zJ58mSXLeKio6PRNI0RI0ZUX2UrKm59hbOeOlDMgZ/1dt7ycgCPLq3DXR/WBCBph4UjqwsveK+Hv4E+EwMZs7EuwxeH8PjyUPzr6YH0gV/K315v2MLaLkeTnp4AnD1iRTkgtK0Zn1pGghqYOHPYCkBhlp2172TRZYw/AfUq/c5MCCGEEEIIUU0qHXgfOnSIsWPH0qBBA/r06cPChQspLLxwUCKufcnJydx444288847HDp0iJCQEPz9/dm7dy+vvfYaXbp0ITc3t7qrSXFxceVucDNXOOvxDUXO84heXgA07u6Byaz3cidsKir3PoBazdxpPcQHk7ue18PfQM0m+tDwkrQ/+qRbCh92TGbukHT2LspDOfRRAjWbuqEZIHWPhbzTdjJO2Ahuppe1dmoWfqEm2g/zqXC7hBBCCCGEENWv0oH3c889R3h4OHa7nZUrV/LQQw8REhLCo48+yrp166qgiqKqPfHEEyQmJgLw1VdfkZCQQEpKClOmTAFgz549vPrqq2iaRkxMDABz585F0zQ0TePEiRMu5W3evJmOHTvi5eVFu3btiI2Ndbm+detWbr/9dgICAvDw8KBdu3YsXrzYJU9J2VOnTmXw4MH4+PgwatSoyjWs810VzpqbbnOeewXpvxaaQcMzQD/PSav4QoEZCVYSt1oAaH23d5nrXkEGfIL1HvHTh6ysmJTJ+g+zAajRyI0+kwJJ2mFhZv80gpu60e+NIE5sKeLgrwX0GhdIzPtZTI9O4fPbUtk6K6fC9RJCCCGEEEJUj0qval5i165dLF68mCVLlhAfH+9ccK1Bgwb885//5IknnriiFRVVIzMzk5o1a+JwOIiOjmbt2rXOaw6HgyZNmpCQkEBQUBBNmjTh4MGD5ObmUrNmTRo3bgzA999/z/Lly51zvL28vKhfvz7Hjh3DZrMRHh7O0aNHMZlMbNq0iZ49e2K1Wp0964cPHwb0YP7hhx8GcP57cnd3x8PDg7CwMDp37syMGTPKtMFisWCxWFzSzFOHYR70FCydDqPeg9DGF/0cVkzKYO+ifAD+ubceBqP+/M9uTSXvlJ0GXTwY+p9LzxNPi7OwZMxZCs45aHqbJ3e+VwODSS+ruMBBdrKN4Ah9fndhtp2vHj7NuWM2TB4aT2+p61yI7XzWQgezB6XTvK8XviEmVr2ZSben/ck7Y2f3V3kM+bwmDbt6XrJuQgghhBBCiOpR6R7vEu3atePhhx/mzjvvxNtb79VTSpGQkMBTTz3Fc889d8UqKarOkSNHcDgcALRt29blmsFgoHXr1oA+B3zp0qW0a9cOgP79+xMbG0tsbCx16tRxue/tt9/m0KFDTJs2DYCTJ09y9OhRAF577TWsViu9evUiKSmJQ4cO8eyzzwLw6quvlqlfo0aNOHHiBHFxcXz22WfltmHKlCn4+/u7HFM2xlfqc/ANKZ0zXZChfx7KoSjK0s/96hgvWcaRNYV8M/IMBecctB7qzZ3TSoNuAHcvgzPoBvD0NzoDZluRojDTUW65Gz/JwWDSuHm0Pydj9SHv7R70oc1Q/ffuxJYLD4MXQgghhBBCVL9KB955eXnMnDmTm2++mZYtWzJt2jTy8/MJCQlh3LhxfPXVVwQGBjJv3ryqqK+oQuVtE1fRfdvP99BDDwH6QnwlTp06BcC2bdsAWLlyJW5ubmiaxocffgjoc83/uGjf8OHDCQwMBMBoLD/4HTt2LNnZ2S7H2K4REL+jwnVu2NXDeR6/Ul8Q7dj6ImwWfUBIwy769bQ4C7MGpDFrQBppcaW97Dvn5/Ljs2exFim6P+9PnwlBzl7zEkfWFLrMFS/KcXBik74+gpunhmdg2c/61IFidn2ZS58Jgfp88/+NTzG4aS5BvRBCCCGEEOLaVemlkevUqUNBQYFzy6iePXsyevRoBg4ciMmkF/f999+zaNGiK1tTUSWaNGmCwWDA4XCwe/dul2sOh4O9e/cCEBQURHBwxfaNDggIAHD+e4CyW4zVrVuXevXqlbnXZrO5fF+7du1LPs9sNmM2/2EhNZMRDm+rUH0BQlq6E3m7Fwd/LWDN21ns/iqPrCS9LvXam2l6q94zbS1UZCTYnOcAKXssrHknCwB3b40jqwo5sqp0wcGBH9XEJ9jI6YPFbP4sB7Ovhl8dfdsxa4Fexk0jfcsMM3fYFMsmZNBqoDf1O+qBf3hnM0dWF3J8fSHZyXo9wqM8EEIIIYQQQly7Khx433LLLbRo0YL8/Hz8/f0ZPnw4o0ePplmzZmXyjhkzhn79+l3RioqqERQURP/+/Vm6dCnr1q3j66+/5r777gNg6tSpHD9+HIAHH3wQTdPw8tJX/M7Pz7+s53Xs2JGYmBjCw8NZtWoVnp56QJucnMzOnTsJDw93yV9eL3yFNbupUsF3v8lBBISZOLA0n6wkG15BRiJ6edLtaf8L7uENYC8ufalQnK9I+7243OuNoz3JTrWRsruYrCQbJrNGcIQb7Yf50ryvV5lyt8/LJf+sneh/BjjT2gzxISPBxoqJmRhM0PUpfxp1l/ndQgghhBBCXMsqvLiawWCgU6dOPPbYY9x///3OgEn89SUlJdG1a1fnyuYNGjSguLiY1NRUQJ/7HRMTg5+fH88//zwffPABBoOBNm3aUKtWLZYtW8acOXOci6uV/JNat24dPXv2BGDt2rVER0ezfv16br31Vmw2G/7+/jRs2JAzZ86QmppK9+7dnSvjlwTcs2fPvrw9wycOggFPVHhxNSGEEEIIIYSoKpWewDty5EgJuq8z9evXZ9euXbz00ks0a9aMtLQ0MjMzad26NW+++SabNm3Cz88PgBdeeIHbbrsNLy8vdu/ezY4dFZ9HDdC9e3fWr19Pv3790DSNAwcO4Obmxt13380LL7xQFc0TQgghhBBCiGpVqR7vxo0bM27cuIvmK9kOSohqNXEQdLoDYn+WHm8hhBBCCCFEtapU4H2p+baappVZHEuIavHmPWCzgpsZnvwYAiq2MJwQQgghhBBCXGmVDrwvll3TNOx2+xWrnBCXLesMFOSAl58E3UIIIYQQQohqVantxCIjI/n000+rqi5CXDkBwRJwCyGEEEIIIa4JlQq8/fz86NGjR1XVRYgrJ+uM/lWCbyGEEEIIIUQ1q1TgLcRfxidjQNMqPL/bblXEfpHD/p/yyU2341XDSLPennR9yh93rwsv/p+bbmPLf3JI2VNM7ikbDiv41zXS8i5v2g/zxeimr4tgsyiWT8wgfV8xGSdsoKBOa3eGLaztUl7c93ls+U8OBecchLRyp/eEQIIauDmvfzf6DA47DJ0hLxSEEEIIIYT4q6jwdmLDhw/n9ttvr8q6CHHl2IrBatHneVfAsnEZbJ6eQ06qnYD6JgrO2dk5P48lT5xFOS68rkFmoo29i/LJTrHhH2pCM8LZozZipmWz5u3M0upYFAeWFlBcoDD7lL9I4bnjVpZPyKR+BzOP/VKHM/FWfnstw3n9wC/5JO2w0Gt8YAU/BCGEEEIIIcS1oMKB9+zZs3nttdeqsi7iKouOjkbTNOdhNBqpW7cuAwYMYPPmzdVdvavm1IFiDvxcAMAtLwfw6NI63PVhTQCSdlg4srrwgvd6+BvoMzGQMRvrMnxxCI8vD8W/nhGAA78UOPO5e2uMXhvK6NWh1GrmXm5ZZ49YUQ4IbWvGp5aRoAYmzhy2AlCYZWftO1l0GeNPQD0ZqCKEEEIIIcRfSYUDb3H9cnd3JyoqitatW3P69Gl+/vlnevTowbZt26q7alfF8Q1FzvOIXl4ANO7ugcms90wnbCoq9z6AWs3caT3EB5O7ntfD30DNJvrQ8JI0AINRwyfYeNF61GzqhmaA1D0W8k7byThhI7iZXtbaqVn4hZpoP8znMloohBBCCCGEqE4SeAvq1KlDbGwsu3fv5ocffgDAZrOxcOFCAH766Se6du2Kj48PHh4e3HjjjcyaNculjHnz5tG2bVt8fX3x9fUlMjKShx56yCXPggUL6NixI15eXvj6+tK3b1/27NnjvG632xk7diyNGjXCw8ODoKAgOnTowLvvvlul7c9NL9173itI/5XQDBqeAfp5TlrFt8jLSLCSuNUCQOu7vStVjxqN3OgzKZCkHRZm9k8juKkb/d4I4sSWIg7+WkCvcYHEvJ/F9OgUPr8tla2zKjaMXgghhBBCCFG9ZMyquKgFCxY4A+jatWvj4eHBnj17eOyxx0hPT+fVV19l7969jBgxAqUUTZo0wcPDgxMnTnDo0CHmz58PwNSpU/nXv/4FQEREBHl5eSxfvpyNGzeyfft251Z1b7/9NkajkZYtW1JQUEBcXBw+Pj68+OKLF6yjxWLBYrG4pJltdsymi/cwX0qFNrg/T1qchSVjzmItVDS9zZMuT/pX+pk3DPLhhkGlvdrWQgeLR5+h4whf0uKK2TE3j25P+5N3xs76D7Kp1cyNhl09K/0cIYQQQgghxNUjPd6CtLQ0OnXqxI033sjAgQMBMJlM3H///bz66qsAREVFcfLkSRISEhg0aBAAkydPpqCggKNHj6KUIiIigsOHDxMXF0dWVhYxMTEAFBQUMGnSJAAmTZrE4cOHOXnyJB06dCA/P5+33noLgCNHjgDwyCOPsHfvXo4cOcK5c+cu2eM9ZcoU/P39XY4pG+Mr3H7fkNL3TwUZDgCUQ1GUpZ/71bl0AH9kTSHfjDxDwTkHrYd6c+e0GhhM5S+iVhkbP8nBYNK4ebQ/J2P1Ie/tHvShzVC9N/3ElgsPgxdCCCGEEEJcGyodeBuNRrp06VImfeTIkURFRV2RSomrq7i4mK1bt/L7778THBxM//79iYmJoWHDhiQmJgIwePBgzGYzmqZx3333AVBYWMj+/fvp0qULgYGBxMfHU6NGDaKionjiiSec5e/fv5+CAn2hsQkTJqBpGm5ubuzYsQOA2NhYAO644w40TWPmzJnUrVuXnj178uabbxIUFHTR+o8dO5bs7GyXY2zXiAq3v2FXD+d5/Eq9nsfWF2Gz6H3eDbvo19PiLMwakMasAWmkxZX2sO+cn8uPz57FWqTo/rw/fSYEYTD++aD71IFidn2ZS58Jgfp88/91wRvctCsS1AshhBBCCCGujkoPNVdKoVTZQbj79u1j586dV6RS4uoKDw/nxIkTZdJPnz5doftDQkLYv38/8+fPZ+fOncTFxTFjxgxmzpzJ5s2bMRhK3+9ERkbi5+fncn+NGjUA6NOnD7t27WLRokXs3buX3bt3s27dOubMmcPRo0fx8Sl/YTGz2YzZbHZNrMQw85CW7kTe7sXBXwtY83YWu7/KIytJn/ddr72ZprfqQ7mthYqMBJvzHCBlj4U172QB+srlR1YVcmRV6SroAz+q6VxU7Yt+aQDkndbnjJ8+VOxMu29OML61S38dHTbFsgkZtBroTf2OeuAf3tnMkdWFHF9fSHayXo/wqNKXBkIIIYQQQohrU4UD79dff915npyc7PJ9fn4+v//+Ox4eEgRcT2rVqkVYWBiJiYksWbKEZ555Bnd3d77++msAPD09admyJampqZw9e5aXXnrJeW9kZCSHDh1i48aNjB49Gk9PTwoLC+nbty/Tpk1D0/Qe2927d1NYqAeqJT3ukydPBiA9PZ06depw6tQpDh8+TPv27ausrf0mBxEQZuLA0nyykmx4BRmJ6OVJt6f90QwX7l22F5e+hCrOV6T9XnzB6yXBfOm10jSH6yW2z8sl/6yd6H8GONPaDPEhI8HGiomZGEzQ9Sl/GnWX+d1CCCGEEEJc6zRVXvd1OQwGA5qmoZRyBk3nU0rRuXNnNm3adMUrKapGdHQ0MTExF+zxhvIXVzt58iQAb775Jq+++iqrVq2iV69eBAcHExoaSk5ODgkJCQAsX76c3r17M2XKFF555RUAQkNDCQ4OJikpiYyMDCZMmMDEiRN57bXXeOutt6hXrx7BwcEkJiZy9uxZvLy8SElJISAgoOKNm6jPQ2fUexDa+LI+HyGEEEIIIYS4Eirc4x0WFoamaSQmJuLu7k5ISIjzmpeXF82bN+fNN9+skkqK6jNs2DD8/PyYOnUqu3fvJisri7Zt2zJmzBgeffRRABo1asR9993H9u3biY+Px2g00qZNG5588kl69+4N6POw69atyyeffMK+ffvIzs6mfv363HPPPQwePBiA7t27s2vXLn7//Xf27duHr68vt9xyCxMmTKhc0C2EEEIIIYQQ15AK93iXMBgMdOrUic2bN1dVnYT486THWwghhBBCCHGNqPTiagkJCS4LWdlsNkwm2Q5cXGNM7oACL79LZhVCCCGEEEKIqlTp7cTCw8M5fPgwPXr0wMPDgx49erB69WpGjhwpveDi2jHmExjzKQQEV3dNhBBCCCGEEH9zle6qXrduHb1798Zm05dhVkoRFhbGnDlzALj55puvaAWFuCwScAshhBBCCCGuEZXu8R4/fjx2u51BgwY505o2bUrt2rVlRXNx7cg6ox9CCCGEEEIIUc0q3eO9Y8cOGjZsyHfffYfBUBq316lTh/j4+CtaOSEu2ydjQNPgyY8r1Ptttypiv8hh/0/55Kbb8aphpFlvT7o+5Y+714XfT+Wm29jynxxS9hSTe8qGwwr+dY20vMub9sN8MbrpW++l7rWw+q1Mzh6z4V/XSI/nA2jco3QP7q3/zWHH3Fwe/akOHv6Vfh8mhBBCCCGEuIZV+n/4JpOJPy6E7nA4SElJwWg0XrGKCfGn2IrBaoGCnAplXzYug83Tc8hJtRNQ30TBOTs75+ex5ImzKMeFF/7PTLSxd1E+2Sk2/ENNaEY4e9RGzLRs1rydCejTMX58/hzWQsU/VtfBu4aRpS+coyjH4Sxj8/Qcbh0bKEG3EEIIIYQQ16FK/y//xhtv5MSJE/zf//0fAGfOnOH+++/nzJkztG/f/opXUFwd0dHRaJqGpmkYjUZ8fX1p1qwZjzzyCLt27ap0eSNGjHCWV1JmzZo1uf3229m9e3e590ycONGZ39/fn4KCgj/brAo5daCYAz/rz7rl5QAeXVqHuz6sCUDSDgtHVhde8F4PfwN9JgYyZmNdhi8O4fHlofjX019AHfhFL7Mw00HeKTu1It3x9DdSp7U71kJFVqK+TsKK1zMI72SmeV+vqmymEEIIIYQQoppUOvB++eWXAfjvf/+LpmkcP36cxYsXo2kaL7744hWvoLi63N3d6dixI/7+/hw5coQ5c+YQFRXFzJkzL7vMqKgoIiMjOXfuHL/99ht9+vShsNA1mFVKMW/ePOf3OTk5LFmy5LKfWRnHNxQ5zyN66cFv4+4emMz6MPGETUXl3gdQq5k7rYf4YHLX83r4G6jZxA3AmeYZaMCntpHTB4spzLaT9nsxbp4aAWEm4r7PI31fMb1eC6yStgkhhBBCCCGqX6UD7379+rFw4ULCwsJQSjlXNV+wYAH9+vWrijqKq6hOnTrExsaSnJzMtm3bCA8Px2azMXr0aA4dOgTAvn37GDx4MDVq1MDd3Z1GjRoxduzYMsF0idjYWPbt28e4ceMAfZTEgQMHXPLExMSQkJAAQIcOHQCcK+VXtdx0m/PcK0j/ldAMGp4B+nlOmr3CZWUkWEncagGg9d3eelmaxl3v18DkofH5rWnkn7Uz4L0a2K2Kde9l0/1Zf05utfBFvzQ+6ZbCb6+do7jAcaWaJ4QQQgghhKhmlV5cDeDee+/l3nvv5ezZswDUrFnzilZKXBs6dOjARx99xMCBA7HZbMyaNYuRI0fSuXNn8vLy8PHxoUmTJhw6dIi3336bnTt3smLFikuWazKZCA0NdUkrCbI7duzIuHHjuPPOO1m7di1JSUnUr1//ouVZLBYsFotLmtlmx2z6c2sOXHhmd/nS4iwsGXMWa6Gi6W2edHnS33kttI2Zh78Nccm/9IWz1Ghkol47M3OHnKJJT08aR3uwbFwmXjWM9Hgu4E/VXwghhBBCCHFtqHSPd3Z2NomJiRQWFlKzZk1iYmJ45pln+O9//1sV9RPVrFu3bs7zAwcO8PbbbzuD7gMHDnDgwAHef/99AFauXMnatWvLlNGpUydatWrFG2+8gbe3Nx999BF16tRxXs/Ly2Px4sUAPPTQQ/Tt25eaNWvicDiYO3fuJes4ZcoU/P39XY4pGyu+wr5vSOn7p4IMvadZORRFWfq5X51LB/BH1hTyzcgzFJxz0HqoN3dOq4HBpF0w/7GYQo6sKaTPxCCStltQDmg1yJsbBvng4W/g5JYLD28XQgghhBBC/LVUOvB+/PHHadiwIQcOHGDp0qUMHTqUTz75hP/7v/9j6tSpVVFHUY0cDtchz9u3bwf0gLykJ/qBBx5wXt+xY0eZMrZu3cr+/fsBaNCgAbfddpvL9cWLF5Ofn4+bmxv33Xcfbm5u3HvvvQAVCrzHjh1Ldna2yzG2a0SF29iwq4fzPH6lviDasfVF2Cx6n3fDLvr1tDgLswakMWtAGmlxpT3sO+fn8uOzZ7EWKbo/70+fCUEYjBcOuosLHKx8I5NO/+dHjcZulGwSYNSnhmO4rHEoQgghhBBCiGtVpQPvnTt3EhAQQPv27fnuu+/QNI3evXujlKpQkCT+WjZs2OA8b9GixWWV4XA42L59OzVq1GD//v3ce++9LlvSlQwzt9vtNG3alICAAOcIiqNHj7Jx48aLlm82m/Hz83M5KjPMPKSlO5G364uqrXk7i1kD0vjxWX0aRb32Zprequ+3bS1UZCTYyEiwYS3U65+yx8Kad7JQDnD30jiyqpAFD5xyHnlnys4PX/9hNmYfjajH/AAIizKjGSBhYxFpcRYKzjkIi/Ioc58QQgghhBDir6nSgXdqaiphYWEAxMXFceONN/Lbb7/RrFkzEhMTr3gFRfXZsWMHzz33HABGo5FHHnmEjh07AnpAnpycDMDChQud95QsjHY+TdPo0KEDEyZMAGDPnj3OoeUJCQmsX78e0AP0kh7r8xdquxqLrPWbHETnf/jhV8dIVpINryAj7R704e7pNdEMF+69theXvkAozlek/V7scpx/HSB1r4W93+bRZ1IQRje93OCm7vSZGMiR1YUsGnWGyP5e3PwPv6ppqBBCCCGEEOKq09T5XY8VEBgYSEBAAIcOHSI4OJiBAwcyb9482rRpw4kTJ8jOzq6quooqFB0dTUxMDO7u7tx4442kpKSQkpKCUgqTycRnn33GY489xsGDB7npppuc87zr16/PoUOHUErRq1cv5+JqI0aMcI6AKPknVlhYSHh4OGfOnOHGG29k165dTJw4kUmTJuHm5sbp06cJCAhw1um5557jww8/xM/Pj7S0NLy8KrHP9cRB+tdR70Fo4yvyGQkhhBBCCCHE5ah0j3dkZCSJiYnUrl2b/Px8oqKiAEhOTqZevXpXvILi6iouLmbbtm1kZWXRpEkThg8fztatW3nssccA/ee/ZcsWBg0ahLu7O0eOHKFBgwa8/PLL/Pjjjxct29PTk6effhqA3bt388svvzj37u7Zs6dL0A0wePBg4Oru6S2EEEIIIYQQV1qle7x//fVXBg8eTHFxMY0bN2bnzp0cOHCAm2++mUceeYRZs2ZVVV2FqDjp8RZCCCGEEEJcIyq9fvLtt99OcnIyiYmJtGzZErPZTMuWLTly5Ag1atSoijoKUXkmd0CBl8yVFkIIIYQQQlSvSvd4/1FycjLbt2+nRYsWNGvW7ErVS4g/J+uM/jUguHrrIYQQQgghhPjbq/Qc75deeolGjRoRGxvL3r17iYyMZMiQIdxwww389NNPVVFHISovIFiCbiGEEEIIIcQ1odKB94oVKzh9+jTt27dn9uzZ5Ofn4+vri81m45133qmKOgpReVlnSnu9hRBCCCGEEKIaVXqO94kTJwgPD8fNzY2dO3fSqFEjDh48SMOGDTl48GBV1FGIyvvkSdAM8OTHFer5tlsVsV/ksP+nfHLT7XjVMNKstyddn/LH3evi76e2/Cebo2uLOHO4GLtVT3tuZz1M5tL9v5N3Wdi9MJe0fcUUnHNgdNeo0chExxG+NL21dJu0uO/z2PKfHArOOQhp5U7vCYEENXBzXv9u9Bkcdhg6Q3rzhRBCCCGE+KuodI+31WrFaDQCcPjwYdq0aYObmxu1a9emqKjoildQ/H2MGDECTdOIjo7+84XZrGC1QEFOhbIvG5fB5uk55KTaCahvouCcnZ3z81jyxFmU4+LLIBxeUUjmSSueQcYL5jm5pYhDywqxFigC6psozneQsruYH545x6FlBQCcO25l+YRM6ncw89gvdTgTb+W31zKcZRz4JZ+kHRZ6jQ+sUJuEEEIIIYQQ14ZKB95hYWHs37+fPn36cO7cOW688UYA0tPTCQkJueIVrCrR0dFomkaDBg1c0tetW4emaWiaxpw5c6qlblfLnDlznG39Ozt1oJgDP+vB7y0vB/Do0jrc9WFNAJJ2WDiyuvCi99/9aU2e2lyX1oO9L5inZlM3hs4I5sn1dRmxJIRhC2uj/e+37+Av+QCcPWJFOSC0rRmfWkaCGpg4c1jvQi/MsrP2nSy6jPEnoF6lB6oIIYQQQgghqlGlA+/HHnsMpRQrV67E3d2dBx54gOPHj5OWlka7du2qoo4CKC4uru4qVBm73Y7dbq+25x/fUDpSI6KXPuy7cXcP51DxhE0XH8nhG2K65MuLZr29aHCzh/P7WpFuuHvr9xjd9a81m7qhGSB1j4W803YyTtgIbqYPM187NQu/UBPth/lUsnVCCCGEEEKI6lbpwPuf//wnP/74I++99x47duygUaNGOBwOvvjiC1599dWqqGO1ycnJwcfHB03TmDlzpjM9Li7O2VMcGxvr0kv+008/0a1bNzw8PGjSpAmLFy92KfPQoUMMHTqU4OBg3N3diYyM5LPPPnPJ06BBAzRN48UXX2TkyJEEBATQp08fAOdzpk2bxrBhw/D19aVu3bq8+eabLmVkZ2fzzDPPEB4ejru7O/Xq1eP555+noEDv2R0xYgSPPPKIM39JuRMnTuS1115D0zRatmzpvN6iRQs0TWP69OkArFmzxnlPeno6ABkZGTz55JPUr1/fOf1g2LBhJCYmOsuZOHGic6TBvHnzaNy4Me7u7iQlJZX5/E+fPk1kZCSapnHTTTeRlZVV4Z9dZeSm25znXkH6r4Rm0PAM0M9z0q78S4EDPxdgyVWgwQ2D9WC6RiM3+kwKJGmHhZn90whu6ka/N4I4saWIg78W0GtcIDHvZzE9OoXPb0tl66yKDaMXQgghhBBCVK9KB94AAwYM4Pnnn3cGZk2aNKFv37789ttvV7Ry1c3Pz48HHngAgP/+97/O9O+++w6AiIgIOnXq5HLPPffcw+nTpzGbzRw7dox7772X3bt3A3DkyBE6derE4sWLcTgcNGvWjMOHD/PEE0/w+uuvl3n+v//9b77++mvCwsLw9PR0uTZ27FjWrFmDh4cHqampjBs3jpUrVwJ673h0dDT//ve/ncHruXPn+OCDDxgwYABKKRo3bkyjRo2c5UVFRREVFUW9evWcc6wPHjxIZmYmGRkZHDp0CICNGzcCsGHDBgCaNWtGSEgIRUVF9OjRg+nTp5Oenk5ERAQ5OTl8+eWXdO7cmTNnXFcYT01NZcSIEZhMJmrXrl2m7ZmZmfTu3ZtDhw4RFRXFypUrCQgIKPfnZLFYyMnJcTkstj8fLP+pDe4vIm5JHsvG6XO3o18IoGGX0p7wGwb5MGpZKM9ur8d9c2rhG2JkxaQMOo7wJS2umB1z82j3gC9Nenqy/oNsEjZefBi8EEIIIYQQovpdVuBdoqioiIULF9K7d28aNGjA+PHjr1S9rpqTJ086e241TaNnz54u10ePHg3Ali1bnMFnSeD98MMPlynvueee4/Dhwxw+fJiAgAAcDodzm7W33nqL7OxsWrVqRVJSEnFxcXzwwQcAvP322+Tm5rqU5efnx+HDh/n9999ZunSpy7UOHTpw4sQJDh48iJubPhx59erVAHz11Vfs2bMHd3d3fv/9d/bu3UtsbCyg91SvWbOGcePGMW7cOGd5sbGxxMbG8thjj3HzzTfj7u6OUopNmzaxadMmlFL4+fk5A++SryVB+ldffcW+ffsAWLRoEfv372fTpk0YDAZSU1P55JNPXOpvtVqZPn06hw8fJiUlhbCwMOe1vLw8+vXrx969e+nUqRMrVqzA39+/vB8fAFOmTMHf39/lmLIx/oL5/8g3pHTOdEGGAwDlUBRl6ed+dS68aFplKKXY8O9slo3PBKDvG4F0HO570Xs2fpKDwaRx82h/TsbqQ97bPehDm6H6fPITW2RBQyGEEEIIIa51lxV4b968mVGjRlGnTh0eeughVq9ejd1uR6mq6iOsOu7u7s7e3qioKCIjI12u33jjjURFRQF6r3d8fDz79u1D0zQeeuihMuXdf//9AISEhDiD+Li4OAC2bdsGwL59+/D29kbTNJ599lkACgsL+f33313Kuvvuu6lfvz6AcyX5Evfccw/u7u7UrFmTWrVqAXDq1CmX5xQXFxMREYGmabRt29Z5b0kQfiFeXl507NgRwBl4GwwG/u///o+kpCQSEhKcZZQE3tu3b3feO3DgQADatWtHs2bNANixY4fLMzw9PRk1ahSgD3M3GEr/Ke7cuZOtW7cSHh7O8uXL8fPzu2h9x44dS3Z2tssxtmvERe85X8OupT3O8Sv1ofjH1hdhs+j/nkt6pNPiLMwakMasAWmkxVkqXD7o25X98nIGsTNyMPtq3P1ZMDcMuvh87VMHitn1ZS59JgTq883/9+tlcNMwmP7eC+IJIYQQQgjxV1Lh5ZFTUlKYO3cuc+fO5ejRowDOQFvTND788EMGDx5cNbWsQnXq1HEJRNetW1em1/uJJ55g69atzJ8/H19fvYeyZ8+eLr20lVGzZk0aN25cJv2PwXV5Q7BLnD/s2mTSf4x/fPHh7u7uXHX+fIGBl96OKjo6mk2bNrFx40aUUrRq1Yr+/fszbdo0PvnkE/Ly8pz5LkdwcLBLsH0+b29v8vPzOXnyJPPnz+fJJ5+8aFlmsxmz2eyaaKp4L3VIS3cib/fi4K8FrHk7i91f5ZGVpM/7rtfeTNNb9WH+1kJFRoLNeV7i53+dI+33YoqyHc60/96VhqZp9Hjen4heXmyfk8vBX/Sg3s3LwMaPs9n4cbbe3ppGBv27pkudHDbFsgkZtBroTf2OeuAf3tnMkdWFHF9fSHayXo/wKA+EEEIIIYQQ17YKB97h4eEopZzBXevWrXnooYeYOHEiBQUFPP3001VWyep2zz338Nxzz5Genu4cNl7eMHOAb775htatW3P69GnWrVsHwA033ABAx44dOXDgAP7+/vz6668EBQUBcPbsWVavXl1mvvjlbvNV0lttt9uZPn26c7X5oqIifvnlF2699VZA750ukZ+fj7d36XZY0dHRTJ482dmTPXLkSKKiojCZTMyYMQMond9d8szPPvuMgoICfvjhBwYOHMiuXbs4fPgwoA+Nr2jbOnToQLdu3XjzzTd56qmnCAoKco4kqCr9JgcREGbiwNJ8spJseAUZiejlSben/dEMF/855J2yOwP1EtnJ+hzz4nz998VerFzy550qnYPuF1r2JcH2ebnkn7UT/c8AZ1qbIT5kJNhYMTETgwm6PuVPo+6eZe4VQgghhBBCXGNUBWmapgwGg7rpppvU3r17nekBAQHKYDBUtJhrRo8ePRSgwsPDXdLXrl2r0Af1qtmzZzvT//nPfzrTvb29VW5ubrn3eHt7q2bNmil/f38FKIPBoHbu3KmUUurQoUPKz89PAcrLy0u1bdtWhYWFKaPR6FKP8PBwBagJEyaUqXd5dSvJP3z4cKWUUkVFRap169bO57ds2VJFREQos9msAJWQkKCUUmrv3r3O8sLCwlRUVJTauHGjUkqp/Px85e7u7ry+YMECpZRSHTt2dKY9/vjjzjoUFhaqVq1aKUCZTCbVokUL5eHhoQAVGhqqTp8+rZRSasKECeV+7kopNXz4cAWoHj16KKWUeuSRRxSg3Nzc1K+//nqhH2X5JgzUj5SjlbtPCCGEEEIIIa6wSs/x3rFjB/369eOll14qMyf5evaPf/zD2Us7ePBgfHzKn5+7ePFiateuTVFREY0aNeKrr75y9jg3a9aMLVu2MHToULy8vNi/fz8Oh4O+ffvyxhtvXLG6ms1mYmJiePrpp6lfvz7x8fFkZmbSoUMHJk+e7BzC3rp1a8aNG0ft2rVJTExk69atZGbqC3+dP88b4Oabbwaga9euzrTzh5l7eHgQExPDE088QUhICPHx8fj6+vLggw+yZcsWgoODK92OGTNm0K9fP6xWK0OGDGHTpk2X83EIIYQQQgghRLXSlKrYimhz5sxh7ty5rF+/HqWUMwgtOd+/fz/Nmzev0spWJ4vFQu3atcnOzmb16tXccsstzmvnzwtPSEigQYMG1VRL4TRxkP511HsQWnY+vRBCCCGEEEJcLRXu8R4xYgRr167l2LFjjB8/ngYNGrgs5tWyZUtatGhRJZWsbsOGDaNTp05kZ2fTvn17l6BbXKPczGByA6+Lr4guhBBCCCGEEFWtwj3e5YmJiWH27Nl899135Ofno2kadrv90jf+xWiahpubGx07dmTu3Lk0adLE5br0eF+Dss7oXwMqP8RdCCGEEEIIIa6kPxV4l8jPz2fRokXMnTuXtWvXXol6CSGEEEIIIYQQ14UrEngLIYQQQgghhBCifBXex1uIv5S1X0OH3uAbdMmsdqsi9osc9v+UT266Ha8aRpr19qTrU/64e118GYTifAcbP8nm8IpCCs7Z8Q0x0vJObzqP8sNg0hcgPLGliM2fZZORYMOS68DDz0CNRm60f9iXprfo+3A77Ip172Zx8NcCHDZo1MODXuMCnc+35DqYdWca7e73pdMombcuhBBCCCHEX0mltxMT4i8h5hvIzaxQ1mXjMtg8PYecVDsB9U0UnLOzc34eS544i3JceECIciiWPHmWnfPzKDin35uTamfz9Bx+G5fhzHf2qJWzR6141zRSs4kbxfmKpB0Wfnz2LCm7LQDEfZ/PzgV5dH/WnwHv1eDA0gK2fpHjLGPdtCy8Ao3cNNL3Mj8QIYQQQgghRHWRwFtclujoaDRNo3Hjslt1HTt2DE3T0DSNN998E9BXxS9Ja9q0aZl71q1b57z+x6Nt27ZV1o5TB4o58HMBALe8HMCjS+tw14c1AUjaYeHI6sIL3ntkdSFJO/TAeeCHNXl0aR1u+VcAAAeWFnDqQDEAbe/14enN9Xjk+xCGLw5h8Kd6+coBqXv1+08fsgJQr72Z+h3Netph6//qUcS+H/LpMynQ2YsuhBBCCCGE+OuQwFtclhEjRgBw/PhxNm3a5HJtwYIFABgMBh5++GHy8vJYvHix8/rRo0fZuHHjBctu1KgRUVFRzqN169ZXvgH/c3xDkfM8opcXAI27e2Ay6wFuwqaicu8DSNioXzN5aDTq7uFShst1d43sVBsLHjjF3CHpLBlzFgDNAKFt9SC7VnM3AJJ3WkjargfjtZq5YStWrJiUyY0P+FDnBvOfb7AQQgghhBDiqpM53uKyDBkyhDFjxpCfn8/8+fPp0qWL81pJ4N2zZ0/CwsKYM2cO+fn5mM1mmjZtyr59+5gzZw5du3Ytt+xx48Y5A/uqlptuc557BenvoTSDhmeAgdxTdnLSLrw9Xk66fs3T34Bm0AN1rxql77Jy0krLthUp0n4vdn7v5qnR780g6v4v8L5hkDfnjlqJ+SAbh13RYoAXUf/nx5bPc7AXK9rd78v3T58leYcFn9pGov/pT8OunlfgExBCCCGEEEJUNenxFpfFx8eHIUOGAPDtt99isei9tFu2bOHo0aNAaa/4nDlzALjzzjsZNWoUAIsWLaKgoOCK1MVisZCTk+NyWGx/bj/5y17q/wI31mjkxov76jNmUyjdn/PHWqhYPinDORzdYNS45eVAxmyoy9Ob69F/Sg2yk21s+28OvcYHsf7DLI6vL6Tf5CDcvTV+fP4cBRl/ro1CCCGEEEKIq0MCb3HZSgLrzMxMfv75Z6C0t9vPz4/BgweTkJDA+vXrAXjooYe47777MJlM5OTksGTJknLLfeSRR1zmeE+cOPGi9ZgyZQr+/v4ux5SN8RVqg29I6aCPggwHoC+aVpSln/vVMV7wXr8Q/VphlsO5CFtJGfq9ZQeUePobiXrUDw8/A5YcxfY5ueWWrRyK5RMyad7Pi4ZdPDgZayE4wo0mPT2J7OeFtUCRel4PuhBCCCGEEOLaJYG3uGw9evSgYcOGAMyfPx+r1co333wDwNChQ/Hy8mLu3LkopQgODqZv374EBwfTp08foLQn/I/+OMe7Xr16F63H2LFjyc7OdjnGdo2oUBsadvVwnsev1Hvgj60vwmbRA+mGXfTraXEWZg1IY9aANNLi9N79Bv+712ZRHF9f5FLG+WX/vjiPwuzS3umU3RaKcvUA3VpYGqifb9eXeWSn2JyLtaHA4KYPZ5cF1oQQQgghhPhrkTne4rJpmsbDDz/MpEmT+PXXX5k3bx7nzp0D9N5wpRTz5s0D9F7x4OBgAIqK9CB17dq1JCUlUb9+fZdyKzvH22w2Yzb/YeEx04V7qs8X0tKdyNu9OPhrAWvezmL3V3lkJelzs+u1N9P0Vn0etbVQkZFgc54DNL3Fk7rt3EnZVcwPz54loL6JzJN6nsj+XtRu4Q7Alhk5rHgjE/+6JowmOJdgcw5Jb3Gnd5k65aTZ2PBxNr0nBOIZoLcjvLOZ4+uLyE6xkbCpCDdPjTo3uFf4MxJCCCGEEEJUH+nxFn/K8OHD0TQNq9XKs88+C0CTJk3o2rUrMTExJCQkAGCz2Zy90SXzwR0OB3Pnzq2uqjv1mxxE53/44VfHSFaSDa8gI+0e9OHu6TWdi6aVx2DUuHt6MO0e9MErSL/Xr46Rzv/wo9+bQc58zft5UaORGwUZdjJO2vAMMNCgiwd3f1aTZuetgl5ixeuZ1G9vpkX/0qD8lrGB1O9oZvagdDISrAx4rwbeNSr2ckEIIYQQQghRvTSl1GWvIyUE6Ht6x8TEOL9/4403eO211xgxYgRz586ldu3apKamYjCUvucZNGgQP/zwA02aNOHIkSOsW7eOnj17AvpQ85LecQBfX19WrlxZuUpNHASj3oPQsvuMCyGEEEIIIcTVJEPNxZ82YsQIZ+Bd3t7dd911l0vQDTB48GB++OGHcvf0Pn78OMePH3d+7+/vX8UtEEIIIYQQQoiqIz3e4vokPd5CCCGEEEKIa4TM8RbXpx73gm9gdddCCCGEEEIIIaTHWwghhBBCCCGEqErS4y2EEEIIIYQQQlQhCbyFEEIIIYQQQogqJKuai+vT2q+hQ2/wDbpkVrtVEftFDvt/yic33Y5XDSPNenvS9Sl/3L0u/m6qON/Bxk+yObyikIJzdnxDjLS805vOo/wwmPQ9wE9sKWLzZ9lkJNiw5Drw8DNQo5Eb7R/2pektngA47Ip172Zx8NcCHDZo1MODXuMCnc+35DqYdWca7e73pdMovz/54QghhBBCCCGuJunxFpdF0zQ0TWPOnDnVXZXyxXwDuZkVyrpsXAabp+eQk2r///buPK6qam3g+G8fhsNhHgUUBRxQQFJRI3NMs66Zc7fULK281q1suvXezEq7N1/tLcubad3Kq41WDmneyiwTccIRRxQcQBFEAhGQ4cCB9f5xYusR0KMJODzfz+d83Oy99trPXm4O5zlr7bXxbu5ISV4l2z87w9LHc1FVdU+BoKoUS5/IZftnZyjJs+5bmFXJxrmF/PjKKb1c7qEKcg9V4ObvgH9rJ8qLFRnbzCx/JpfMJDMAe74tZvvnZ+j1jBeD3vIjeUUJmz8q1OuIn3kaVx8Hbn7Y4zIbRAghhBBCCNFYJPFuQGVlZbz99tvExcXh6emJq6srERERPProozbPrb5SwsLC0DSNqVOnXvG661ufPn3QNI1x48bV63FOJpeT/N8SAPq+6M0jK4IZMssfgIxtZg6uLq1z34OrS8nYZk2ch87y55EVwfT9uzcAyStKOJlcDkDH+9x5amMID30bxNjFQQyfY61fVUHWLuv+OQcqAAjpbKR5V6N1XUrF73GUsXdZMXe+5qP3ogshhBBCCCGuHTLUvIHk5+fTr18/kpKSAPDw8KBVq1YcO3aMDz/8kG7dutGyZctGi6+yshIABweHRouhMRxZV6YvR/R3BaBVLxccjRoWsyJtQ5m+/nxp6637OrpotOzlotexevppfXtglDOOzhoFWRZWPJ9HZbniVLoFAM0ATTtak+wm7ZwAOL7djEem9f+iSVsnLOWKVa/l02m0O8Exxit89kIIIYQQQoiGID3eDeTJJ5/Uk+4XXniBU6dOsWfPHgoKCli7di1t27YF4LvvvqNHjx64u7vj4uJCp06dmDdvnk1d1cO8Z86cyZgxY/Dw8KBZs2a8/vrrAKSnp6NpGkePHgXgtdde0/cBmDp1KpqmERYWxqeffkqrVq1wdnYmIyPD7hjOt2DBAv0Ya9asITY2FpPJRGxsLImJiTZlN2/ezF133YW3tzcuLi7ExsayePFim/Nbu3YtAJ988oleb3p6+uU0/QUVZVv0ZVdf66+DZtAweVuXC09U1rlvYbZ1m8nLgGawtq2r39lfqcITZ+u2lClO7C4n50AFljKFk0lj0Jt+NPs98Y4Z5kbnMe6sfaeA757PJWqQK3F/8WTTB4VUlitiR3nw7VO5zL41k/nDsklbX3dPvBBCCCGEEOLqIj3eDaCgoIBvvvkGgA4dOvDGG2/oSTBAr169APj888954IEHAAgMDMTFxYWdO3cyfvx4srOzmTx5sk29kyZNwt/fHxcXF7KysnjllVeIi4ujffv2xMXFkZSURHl5Oc2aNSMkJKRGXFlZWYwbN442bdoQGBh4WTHUZsCAAYSFhWGxWEhKSmLkyJEcOnQIR0dHNmzYwG233UZFRQVBQUEEBQWRlJTEn//8Zz755BMefPBB4uLiSE5OpqioCH9/f1q1agWA0Vh7j6/ZbMZsNtusM1oq+SP9w5f9cPs6dvRr6cQLe5tTWlDJ7sXFJLxTwE+vncK7uSOBUc4YHDT6vuhD3xd99H1+Sy1ny38KGT4ngIRZpzmSUMqQd/zZPK+Q5c/lMWFlMK6+N9YIBSGEEEIIIa5F0uPdAFJTU7FYrL2fPXv2tEm6z1Wd1MbFxXH06FHS0tIYNmwYANOmTaOkpMSmfJcuXUhPT2f//v04OVmHKq9evZrg4GASExMJDg4GYPz48SQmJtboea6oqGDu3LmkpKSQmZlJixYtLjmG2rz55pscOHCAmTNnAnD06FEOHToEwMsvv0xFRQX9+/cnIyODAwcO8Mwzz9icf2JiIrGxsQAMHDhQj736fM43ffp0vLy8bF7T16deNE4Aj6Cz3z2VnKoCrJOmlZ22LnsG153YegZZt5WertInYauuw7pvze+1TF4OxD3iiYunAXOhYuuColrrVlWKn6bk026AK+HdXTiaaCYgwonWt5mIHOBKRYkia3e5XecohBBCCCGEaFySeDcApc52g9aVdOfk5HDs2DEAhg8fjtFoRNM0Ro4cCUBpaSn79u2z2efee+/F2dkZf39/mjRpAsDJkyftjstkMjFhwgQ9rtzc3EuOoTbVPeZRUVH6uuq4tmzZAsDPP/+Mk5MTmqYxa9YsAI4fP05mZqbd8VebNGkSBQUFNq9JPSLs2je8h4u+nPqz9UuFwwllWMzW/7Pw7tbtJ/aYmTfoBPMGneDEHmvvetjv+1rMiiMJZTZ1nFv37sVnKC04O2Q9M8lMWZE1Qa8oPZuon2vHF2coyLTok7WhwOBkvXZkgjUhhBBCCCGuLTLUvAG0bdsWR0dHLBYL69evRylVZwJ+Kby9vfVlR0frf+W5Sf7FBAQEYDBc+e9equOqjglqxlXX8PfqkQGXwmg01hyG7mjfEOygaGci73Jl/w8l/DrjNEkLz3A6wxpDSGcjbfpZn7NdUao4lWbRlwHa9DXRLNaZzB3lLHsmF+/mjuQftZaJHOhKYJQzAJs+LGTVP/PxauaIgyPkpVn0IelRg91qxFR4wsK62QXcMcUHk7f1PEK7GTmSUEZBpoW0DWU4mTSCY5wvoZWEEEIIIYQQjUV6vBuAl5cX9957LwBJSUm89NJLNgnmL7/8wqFDh2jRogUAS5cuxWw2o5Tiq6++Aqy909HR0Zd0XFdX62zcxcXFtW4/P/lv0qTJFY/hfF27dgUgNDSUNWvW6MPIFy9ezKRJkwgNDbUr9itpwDRfuj3miWewA6czLLj6OhB7vzsj5vrrk6bVxuCgMWJuALH3u+Pqa93XM9iBbo95MuB1X71cuwGu+LV0ouRUJaeOWjB5Gwjr7sKI9/1pW8uM6av+kU/zzkaiBp5NyvtO8qF5VyPzh2VzKq2CQW/54eYn93cLIYQQQghxLZAe7wYye/ZskpOT2blzJzNmzGDu3LmEhYWRkZFBfn4+8+fPZ9q0aTzwwANs3ryZ0NBQXFxc9JnJJ0+erCej9mrXrh379+/n3XffJT4+nvbt2zN//vwL7nOlYzjfP/7xD/r168fGjRsJDg4mPDyc3377jaysLHr16sWQIUP02H/88UeWLl1KbGwsTZo0YeXKlX/o2HVxcNLo8aQXPZ70qrNMi5tdeGFv8xrrje4G+k3yod8kn1r2sur9rDe9n7U/nnveD6ixzt3fgRFza64XQgghhBBCXP2kx7uB+Pr6smnTJt566y26du1KVVUVKSkp+Pj4MH78eHr16sWYMWNYvnw53bt3p6ioiOzsbDp27MjHH39s12zi53v99de55ZZbMBgMbNu2jT179lx0nysdw/l69epFQkICAwYMQNM0kpOTcXJyYsSIETz//PN6ueeff57bb78dV1dXkpKS2LZt2x8+thBCCCGEEEI0Bk1dyk3BQlwrpg6DCW9B01aNHYkQQgghhBDiBic93uL61Ps+8Kh7+LcQQgghhBBCNBTp8RZCCCGEEEIIIeqR9HgLIYQQQgghhBD1SBJvIYQQQgghhBCiHsnjxMT1ac1X0OUO8PC9aNHKCkXiR4Xs+66YouxKXP0caHuHiR4TvXB2vfB3U+XFVax/r4CUVaWU5FXiEeRA9GA3uk3wxOBofQZ4+qYyNr5fwKk0C+aiKlw8Dfi1dKLzgx606WsCoKpSEf/mafb/UEKVBVr2dqH/Kz768c1FVcwbfILYUR7cMsHzDzaOEEIIIYQQoiFJjzfQp08fNE1j3Lhx18VxrjRN09A0jalTpwIQHx+vr0tPT2/U2Oq09msoyrer6MpXTrFxbiGFWZV4N3ekJK+S7Z+dYenjuaiquqdAUFWKpU/ksv2zM5TkWfctzKpk49xCfnzllF4u91AFuYcqcPN3wL+1E+XFioxtZpY/k0tmkhmAPd8Ws/3zM/R6xotBb/mRvKKEzR8V6nXEzzyNq48DNz/scZkNIoQQQgghhGgsV0XivWDBAj2Rc3BwICMj44ofIz09XT9GfHz8Fa/fnuNERUURFxdHq1aX9oirsrIy3nnnHW699Va8vb0xGo20aNGC22+/nbfffvsKRm8fT09P4uLiiIuLw2g01ssxqttwwYIF9VJ/tZPJ5ST/twSAvi9688iKYIbM8gcgY5uZg6tL69z34OpSMrZZE+ehs/x5ZEUwff/uDUDyihJOJpcD0PE+d57aGMJD3wYxdnEQw+dY61dVkLXLun/OgQoAQjobad7V2qY5KRW/x1HG3mXF3Pmaj96LLoQQQgghhLh2XBVDzc9Nrqqqqvjkk094+eWXGy+gejJ37txL3icvL49+/fqxa9cuAFxdXYmIiKCoqIi1a9eyevVqnnvuuTr3r6ysBMDBweHygq5FbGwsiYmJV6y+xnRkXZm+HNHfFYBWvVxwNGpYzIq0DWX6+vOlrbfu6+ii0bKXi17H6umn9e2BUc44OmsUZFlY8XweleWKU+kWADQDNO1oTbKbtHMC4Ph2Mx6Z1v+zJm2dsJQrVr2WT6fR7gTH1M+XHEIIIYQQQoj61eg93mlpaSQkJADQpUsXAD755BObMgUFBTz99NOEhobi7OxMSEgIzz33HCUlJXqZlJQUBg8eTJMmTTAajYSEhDBgwAC2bNnCggULCA8P18vedtttaJpGnz59bI6jlOJ///d/adq0KT4+PowZM4aioiJ9e1VVFf/6179o3749Li4u+Pj48Oc//5m0tDSAix6ntqHmRUVFPP/887Rq1QpnZ2f8/Pz405/+RGmptaf1ySef1JPup59+mry8PPbs2UN6ejq5ubnMnz9fr2vq1KlomkZYWBiffvqpXmdGRgY///wzPXv2pEmTJjg7O+Pp6UnPnj358ccfbdpg9+7d3HLLLbi4uNChQwfWr19f4/+srqHmP/74I71798bDwwOTyUTPnj1Zs2aNvv3c0QALFizg7rvvxtXVlfDwcObNm2dTd7WHHnpIP6f6UJRt0Zddfa2/DppBw+RtXS48UVnnvoXZ1m0mLwOawRqzq9/ZX6nCE2frtpQpTuwuJ+dABZYyhZNJY9CbfjT7PfGOGeZG5zHurH2ngO+ezyVqkCtxf/Fk0weFVJYrYkd58O1Tucy+NZP5w7JJW193T7wQQgghhBDi6tLoifcnn3yCUoqgoCA++ugjAA4dOqQnfOXl5fTp04d3332XnJwcIiMjycvL45133mHQoEFUP4Z81KhRrFixAovFQnR0NFVVVaxcuZLk5GQCAgLo2LGjfszIyEji4uKIioqyiWXRokXMmDEDFxcXTp8+zRdffMGMGTP07U8++STPPPMM+/bto3Xr1jg4OLB48WJuvfVWcnJy7D5OtepzmzlzJkeOHKFp06b4+vqyatUqzGYzp0+fZtGiRQB06NCBt99+GxcXF31/Ly+vWu8Xz8rKYty4cTg6OhIYGAjAvn372Lx5Mx4eHrRv3x6lFOvXr2fw4MF6Yl9aWspdd93F5s2bqaqqoqKigoEDB9rz38jXX3/NwIEDSUhIwM/Pj+DgYNavX0///v1tku9qEyZMYN++fTg5OZGens6ECRM4cOCAPoy9WsuWLYmLi6NTp051HttsNlNYWGjzMlvqTpjtcdkPt69jR7+WTrywtzlPbmhKr2e9qChV/PTaKX04usFBo++LPjy5rhlPbQxh4HQ/Co5b2PKfQvq/6kvCrNMcSShlwDRfnN00lj+XR8mpP3aOQgghhBBCiIbRqIm3UopPP/0UgNGjR9OxY0duuukm4Ozw84ULF7Jz506cnZ3ZvXs3u3bt0oc5//rrr/z6668AHDx4EIAVK1awY8cOsrKyOHLkCH369GHgwIF8++23+nHnzp1LYmJijaHfjo6O7N+/n0OHDtG5c2cAVq9eDVh75j/44APA+mXB3r17SU9PJyQkhOzsbGbPnm33cap99dVX7NixA4D/+7//Iz09nYMHD7Jnzx5cXV1JTU3Vh4r37NkTg8H63zV06FC957i2+6ArKiqYO3cuKSkpZGZm0qJFC4YNG0ZOTg6HDx9mx44dHDt2DA8PDywWC4sXLwbgyy+/JDMzE4DvvvuO5ORku+8hf/HFF1FK8fDDD5OWlsbhw4cZNmwYlZWVvPrqqzXKDxkyhCNHjrBu3TrAOpogPj6+xjD2V155hcTERJt2Pd/06dPx8vKyeU1fn2pX3B5BZ++2KDlVBVgnTSs7bV32DK57iL5nkHVb6ekqfRK26jqs+9a8k8Pk5UDcI564eBowFyq2LiiqUaY6hp+m5NNugCvh3V04mmgmIMKJ1reZiBzgSkWJImt3uV3nKIQQQgghhGhcjZp4r127Vh+m/cADD9j8u2jRIkpKStiyZQtg7R2OiIhA0zSbXuXqJG3QoEGAdXh3ZGQkI0aMYOXKlQQHB9sdT9++fWnWrBkGg4F27doBcPLkSQC2bdum966PHTsWTdPw8PDg+PHjNnFcis2bNwNgNBpt7tOOjo7G2dnZpmx10g3Qtm1bOnToUGe9JpOJCRMmANZJygwGA2azmXHjxtGkSRMcHBzw9fXVh9FnZWUB1l5xsN5H/qc//QmAe++996Ln8dtvv+lDzv/zn/9gMBgwGAx6slx9nue6//770TTNZjRAdVtfqkmTJlFQUGDzmtQjwq59w3ucHUGQ+rP11oXDCWVYzNb/6/Du1u0n9piZN+gE8wad4MQe64RoYb/vazErjiSU2dRxbt27F5+htOBs73RmkpmyImuCXlF6NlE/144vzlCQadEna0OBwck6nF0mWBNCCCGEEOLa0qiTq53bU1t9H7TFYr0vtrCwkKVLl+rbnZ2dax1u7OPjA8Cnn37K4MGDiY+PJzk5mR9++IGlS5eyd+9e5syZY1c83t7e+rKjo7VpqpPtc3Xs2LHGbN6hoaF2HaMu597XXK1t27Y4ODhQWVnJxo0b9fVvvPEGDz30EJGRkbXWFRAQYJOoAwwcOJBDhw7h6OhITEwMLi4uJCUlUV5erveqXygWe7Vs2ZKAgIAa68vLbXtnq9u6up2h9ra2h9ForDm7uqN9k8kFRTsTeZcr+38o4dcZp0laeIbTGdZrMKSzkTb9rM/ZrihVnEqz6MsAbfqaaBbrTOaOcpY9k4t3c0fyj1rLRA50JTDK+uXJpg8LWfXPfLyaOeLgCHlpFn1IetRgtxoxFZ6wsG52AXdM8cHkbT2P0G5GjiSUUZBpIW1DGU4mjeAY5xr7CiGEEEIIIa4+jZZ4nzlzRh/iDNYJ1M63YMECxowZA1hn5547dy6xsbGA9RFb33//Pf369QNg3bp1DBs2jJEjRwIwY8YMJk2apE/c5up6dmbq4uLiS463c+fOaJqGUopx48bx9NNPA+j3Snt5eV3yceLi4pg7dy5ms5lZs2bpvd779++nVatWeHl5ce+997Jw4UK2bdvGlClTePXVVy86Q/n5iXNeXh6HDh0C4B//+AeTJk0iPT1d79WvFh0drce9atUq7rjjDpv/o7oEBAQQGhrK0aNHiY2NZeHChXpCnZqaytGjR2v04F+MyWSitLT0sv6vLtWAab54t3AkeUUxpzMsuPo6ENHfRM+nvPRJ02pjcNAYMTeA9bMLSP25lNMZFjyDHYga5Ea3Rz31cu0GuHIkoYzCExYqShUmbwOBUc50HuNOy56mGvWu+kc+zTsbiRp4NinvO8mHitJTzB+WjUegA4Pe8sPN78rNVC+EEEIIIYSoR6qRzJ8/X2Ht91N79+612TZr1iwFKIPBoNLT09VNN92k/xwdHa0iIiKU0WhUgEpLS1NKKdWsWTNlMplURESE6tixo3JyclKAGj16tFJKqaqqKuXn56cA5ePjo26++Wb17rvvKqWU6t27twLU2LFj9RjGjh2rABUaGqqvmzBhgh5zeHi4iomJUZ6engpQ8+fPv+TjmM1mFRsbq9cZFham2rRpowwGg8rPz1dKKZWbm6ufP6A8PT1Vx44dVWBgoL6u+thTpkypEXN1TCEhIQpQTk5Oqn379srHx0e5ubnZxFNSUqKaNm2qAOXs7KyioqL0MoCaMmWKUkqpNWvW6Ouq2/+LL77Q1wUEBNjEWF1/WlqaXmbNmjV6fOfXr5RSnTp1UoByd3dXXbt2VZMmTbrwBXW+KUOVyjx0afsIIYQQQgghRD1otHu8q4eZR0RE6D2t1YYPHw5YJ9z67LPPWLt2LU899RTNmzcnNTWV/Px8unTpwrRp0/RZux966CGio6PJzc0lOTmZoKAgJkyYwHvvvQdYe4E/+ugjWrduTWFhIVu2bOHo0aOXFPP777/PO++8Q0xMDFlZWRw9epSwsDCee+45faj8pRzH2dmZNWvW8Le//Y3w8HAyMzPJy8vj9ttv14dO+/n5kZiYyBtvvEHnzp2pqqriwIEDmEwm7rzzTj744AOGDh16wbg1TWPJkiV07dpVH7r+xRdf4O/vb1POZDLx/fff07VrV33dhSY1O9fo0aP573//S+/evSktLSUlJQUPDw8efPBBxo8fb1cd53r33XeJiYmhvLycrVu3kppq32RpQgghhBBCCHG10ZS6zBtrhbiaTR0GE96Cpq0aOxIhhBBCCCHEDa7Rn+MtRL3ofR94+DR2FEIIIYQQQgghPd5CCCGEEEIIIUR9kh5vIYQQQgghhBCiHkniLYQQQgghhBBC1KNGe463EPVqzVfQ5Q7w8L1o0coKReJHhez7rpii7Epc/Rxoe4eJHhO9cHa98HdT5cVVrH+vgJRVpZTkVeIR5ED0YDe6TfDE4Gh9Bnj6pjI2vl/AqTQL5qIqXDwN+LV0ovODHrTpa32Od1WlIv7N0+z/oYQqC7Ts7UL/V3z045uLqpg3+ASxozy4ZYJnnfEIIYQQQgghrj7S492AwsLC0DSNqVOnNnYoV0R8fDyapqFpGvHx8Ze077hx49A0TX8M2xW39msoyrer6MpXTrFxbiGFWZV4N3ekJK+S7Z+dYenjuaiquqdAUFWKpU/ksv2zM5TkWfctzKpk49xCfnzllF4u91AFuYcqcPN3wL+1E+XFioxtZpY/k0tmkhmAPd8Ws/3zM/R6xotBb/mRvKKEzR8V6nXEzzyNq48DNz/scZkNIoQQQgghhGgsN2zi3adPHz1p7NChg822vLw8TCaTvv3FF1+0u95zk9H09HSbbZ06dSIuLo6QkJArcQo1VCezmqbRpEkTzGazvs1isdCsWTN9+8iRI+slhmvNyeRykv9bAkDfF715ZEUwQ2ZZn2+esc3MwdWlde57cHUpGdusbTx0lj+PrAim79+9AUheUcLJ5HIAOt7nzlMbQ3jo2yDGLg5i+Bxr/aoKsnZZ9885UAFASGcjzbtan+Gek1Lxexxl7F1WzJ2v+ei96EIIIYQQQohrxw2beJ9r9+7dJCQk6D9//PHHlJWVXfHjfPvttyQmJjJ+/PgrXvf5fvvtN77++mv95yVLlpCVlVXvx73WHFl39v85or8rAK16ueBotCa4aRvqvg7S1lu3ObpotOzlYlOHzXZnjYIsC5+PPskn92Sz9MlcADQDNO1oTbKbtHMC4Ph2Mxlbrcl4k7ZOWMoVq17Lp9Nod4JjjH/8hIUQQgghhBAN7oZPvJ2crAnP7NmzAaisrGTu3Ln6+nOdOnWKJ554gubNm+Pk5ERgYCBjxozh2LFjAEydOpXbbrtNLx8eHo6maYwbNw6ofaj5sWPHePDBBwkKCsLJyYmQkBAef/xxTp06O1T53GHZc+bMISwsDA8PD+6++26ys7NrxOno6GhzTucu13ZepaWlTJ48mdatW+Ps7Iyvry9Dhw5lz549NuW++eYbWrZsiclk4q677iIzM7NGXVOnTkXTNMLCwvR1FxoFcC6z2cyUKVNo06YNzs7ONGnShIcffpjc3Nw69/mjirIt+rKrr/XXQTNomLyty4UnKuvctzDbus3kZUAzWBN1V7+zv1KFJ87WbSlTnNhdTs6BCixlCieTxqA3/Wj2e+IdM8yNzmPcWftOAd89n0vUIFfi/uLJpg8KqSxXxI7y4Nuncpl9aybzh2WTtr7unnghhBBCCCHE1eWGn1ytY8eO5OXlsWzZMo4fP87WrVs5duwYo0aNYuHChXq5srIyevfuzd69e3F0dCQiIoIjR47wxRdfsGbNGnbu3ElISAiRkZHs379fr9toNNKqVataj52Tk0O3bt3IysrCaDQSERFBamoq77//PuvWrWPr1q24uLjo5Tdu3MjmzZtp3rw5Z86c4fvvv+dvf/sbX3zxhU29fn5+REZGEh8fT2JiIkajkQ0bNujHOnr0qE35wYMH88svv6BpGm3btuX48eMsX76c1atXs3XrVtq1a8fOnTsZNWoUVVVVeHl5kZqayqOPPnql/hsAGD58OD/88AMODg5ER0eTnp7O/Pnz2bx5M9u2bcNkMtW6n9lsthlWD2C0VPJH+ocv++H2dezo19KJF/Y2p7Sgkt2Li0l4p4CfXjuFd3NHAqOcMTho9H3Rh74v+uj7/JZazpb/FDJ8TgAJs05zJKGUIe/4s3leIcufy2PCymBcfR0uN1IhhBBCCCFEA7nhe7wNBgNPPPEEFouF999/X+8Znjhxok25hQsXsnfvXgAWLVrEvn372LBhAwaDgaysLN577z3Gjx/P3Llz9X2qh5a/8sortR57zpw5ZGVlYTAY2LhxI/v27WPRokUA7N271ybxB2tvfGJiIqmpqQwbNgyA1atX11p3dfyzZ8+u85wA1qxZwy+//ALA22+/zf79+9m/fz/u7u6cOXOG6dOnAzBz5kw96U5JSeHQoUMMHz68rma9ZGvXruWHH34A4Ndff2XXrl0cOHAAk8lEcnIyX375ZZ37Tp8+HS8vL5vX9PWpdh3XI+jsd08lp6oA66RpZaety57BdSe2nkHWbaWnq/RJ2KrrsO5b83stk5cDcY944uJpwFyo2LqgqNa6VZXipyn5tBvgSnh3F44mmgmIcKL1bSYiB7hSUaLI2l1u1zkKIYQQQgghGtcNn3gDPPzww7i5uTF79mzWrFlD586d6datm02ZrVu3AuDq6srQoUMBiI2NpW3btgBs27btko9bXWfbtm2JjY0FYOjQobi6utZaZ0xMjD4RXFRUFAAnT56ste4hQ4bQokULFi1axMKFCwkODuaee+6pMwaA0aNHAxASEkLPnj1tYti3bx8A3bt3JzAwEIA///nPl3rKddqyZYu+3Lt3bzRNo2nTppSWWodUJyYm1rnvpEmTKCgosHlN6hFh13HDe5wdUZD6s3WStcMJZVjM1kQ6vLt1+4k9ZuYNOsG8QSc4scfaux72+74Ws+JIQplNHefWvXvxGUoLzg5Zz0wyU1ZkTdArSs8m6ufa8cUZCjIt+mRtKDA4WYezywRrQgghhBBCXFtu+KHmAN7e3owZM4Z///vfQO09w1cDb29vfbn6Pu66ODg48Ne//pVJkyZRUVHBo48+Wuv93VeaplmTwsrKs4lmQUHBJdURFxdXY11QUFCd5Y1GI0bjeQPLHe0bgh0U7UzkXa7s/6GEX2ecJmnhGU5nWO/NDulspE0/6/D2ilLFqTSLvgzQpq+JZrHOZO4oZ9kzuXg3dyT/qLVM5EBXAqOcAdj0YSGr/pmPVzNHHBwhL82iD0mPGuxWI6bCExbWzS7gjik+mLyt5xHazciRhDIKMi2kbSjDyaQRHONs1zkKIYQQQgghGpf0eP/uySefBCAgIKDWR2117doVgJKSEpYtWwbAjh07SElJAaBLly4Aem81QHFx8QWPWV1nSkoKO3bsAGDZsmWUlJTY1Hm5xo8fj4uLC05OTnXej10dA6AP5z5+/Djr1q2ziSE6OhqADRs2kJOTA8DixYtr1NekSRPAev96dcJdW7kLxTFp0iQSExNJTExk/fr1TJ06lUceeeSidVyuAdN86faYJ57BDpzOsODq60Ds/e6MmOuvT5pWG4ODxoi5AcTe746rr3Vfz2AHuj3myYDXffVy7Qa44tfSiZJTlZw6asHkbSCsuwsj3ven7TmzoFdb9Y98mnc2EjXwbFLed5IPzbsamT8sm1NpFQx6yw83P7m/WwghhBBCiGuCukH17t1bASouLk5fl5eXpwoKCvSfsfZLqr///e+qtLRUtW/fXgHK0dFRRUVFKRcXFwWopk2bqpycHKWUUrm5ucrJyUkBKigoSMXFxalFixYppZQKDQ1VgJoyZYpSSqmTJ0+q4OBgBSij0aiio6OVo6OjAlT79u1VaWmpUkqpsWPHKkD17t1bj23KlCl6fNWqywUGBurr8vPzVX5+vv5zdQz33Xefvu72229XgNI0TUVGRioPDw8FKHd3d7V//36llFI7duxQmqYpQHl5eanWrVsro9Gox7BmzRqllFL79+9XBoNBASo8PFx17txZ/xlQaWlpdZ7TnXfeqZdr27atioqKUm5ubjb1223KUKUyD13aPkIIIYQQQghRD6TH+xy+vr54enrWus3FxYW1a9fy+OOPExQURGpqKh4eHtx///1s2rSJgIAAwDqj+Lvvvkvz5s05efIkmzdvrvWRX2DtHU5MTOSBBx7A29ublJQUAgMDeeyxx1i7dq3NjOaXy9vb22aIem2+++47XnrpJcLDwzl48CCOjo4MGTKEjRs30q5dOwA6derEl19+SVhYGGVlZYSGhvL+++/XqKtdu3Z8+OGHhIWFceLECfz9/W0mnLuQZcuW8eqrr9KmTRuOHDlCdnY2kZGRvPzyy7Rv3/6Sz10IIYQQQgghrgaaUuqyn5wkxFVr6jCY8BY0rf1RbkIIIYQQQgjRUKTHW1yfet8HHj4XLyeEEEIIIYQQ9Ux6vIUQQgghhBBCiHokPd5CCCGEEEIIIUQ9ksRbCCGEEEIIIYSoR46NHYAQ9WLNV9DlDvDwvWjRygpF4keF7PuumKLsSlz9HGh7h4keE71wdr3wd1PlxVWsf6+AlFWllORV4hHkQPRgN7pN8MTgaH0G+JnfKvl1Rj7Ze8spyKwEoN2fTAx6y9+mrsSPCkn66gzlZ6pocbOR/lN8cfe3Pqu7yqL49N6TBN/kzJ1TL35OQgghhBBCiKuH9HiLetOnTx80TWPcuHENf/C1X0NRvl1FV75yio1zCynMqsS7uSMleZVs/+wMSx/PRVXVPQWCqlIsfSKX7Z+doSTPum9hViUb5xby4yun9HLFeZWk/FQKGjgatVrrSt9Yxrp/FRAzzI0xXwZyOKGM+DdP69u3/KeIkvxK+vzN265zEkIIIYQQQlw9JPG+BlQnsGFhYTbrx40bh6ZpaFrtyZy4uJPJ5ST/twSAvi9688iKYIbMsvZEZ2wzc3B1aZ37HlxdSsY2MwBDZ/nzyIpg+v7dG4DkFSWcTC4HwDfMkSfXN2XCyqa4+tX+K5dzwFo2JNaIXysnXH0N/JZiXZd/tIJN/y7k9pd8MHrIr6wQQgghhBDXGvkUL64J5eXl9VLvkXVl+nJEf1cAWvVy0Xum0zaU1bofQNp66zZHF42WvVxs6jh3u5OLAZO3wwXjaNLOGYDjO8zkHa6g5FQVAW2dUUrx09R8wnu42NQthBBCCCGEuHZI4n0dqe4B79OnD3PmzCE0NBQXFxcGDBhARkaGXi4xMZF+/frh5+eHi4sLYWFhDB06lMOHD+tltm3bxpAhQ/Dz88NoNNKyZUtmzpwJQGlpKUOHDiU8PBw3NzeMRiNt2rTh1VdfvWiCXFBQwNNPP01oaCjOzs6EhITw3HPPUVJSUut5/N///R8hISG4uLhc4dayKsq26MuuvtZfB82gYfK2LheeqKxz38Js6zaTlwHNYE3Uz+3RLjxhqXW/2oTd6kLPp73Ys7SYz0efpGVPF/q84M2eJcXkHCin17Ne/PhyHu/1zOSjASfYu7zY/pMUQgghhBBCNCqZXO06lJiYyJYtWwgLC6OiooKVK1cydOhQtm3bhlKKu+++m7y8PAIDA4mMjCQzM5Ply5fzzDPP0KpVKzZu3Mhtt91GeXk5zs7OtGnThuzsbNatW8ff/vY3zGYzy5cvJzAwkIiICHJzczl06BD//Oc/KS0t5c0336w1rvLycvr06cPOnTtxcXEhMjKS1NRU3nnnHXbt2sUvv/xiM2x+06ZNrFu3jrZt21JWVnfPs9lsxmw226wzWiox/oE2vOyH21/2jnDLXzy55S+e+s9nciuJf/s0vZ/zZs/SYvYuK+FP//Th8NoyVr5yiqBoZ/xbO13+AYUQQgghhBANQnq8r0OVlZVs3bqV5ORk5s6dC8COHTv46aefyM/PJy8vD4Dt27eTlJRETk4Oe/fuJSoqCoCXX36Z8vJyvL292bNnD3v37iUnJ4fXXnsNADc3N/bt20d2djZJSUlkZGQwZswYAL766qs641q4cCE7d+7E2dmZ3bt3s2vXLhITEwH49ddf+fXXX23Kl5eX89///pfk5GROnjxZZ73Tp0/Hy8vL5jV9fapdbeURdPa7p5JTVYB10rSy09Zlz+C6h4h7Blm3lZ6u0idhq67Duu8f+15r9bR8AiKcuekeN44mluHiZSBmmDvth7qhquDY5rq/jBBCCCGEEEJcPSTxvgZc6uRpMTExREdHAzBq1Ch9/Z49e/Dz86Nbt24AtG7dmpiYGEaNGkVSUhL+/tZJxTZv3gzAPffcQ0REBAAGg4EOHTroy59//jkREREYjUY0TePzzz8HICsrq864tmzZAlgT6oiICDRNo2PHjvr26iS8Wtu2bRkwYAAADg51J8CTJk2ioKDA5jWpR8RFWskqvMfZIeypP1uHux9OKMNitibS4d2t20/sMTNv0AnmDTrBiT3W3vWw3/e1mBVHEsps6ji/7kt18NdSjiSUcedUHzRNQylw+L1z20HGqQghhBBCCHFNkY/w1wA3NzcATp06ZbO+uufa3d39kupbvXo1X375JRs2bCA5OZnFixfz1VdfceLECV544YWL7j9jxgymT58OQGhoKEFBQRw/fpzMzEyqqqousjc4OzvTqVOnGut9fHxsfg4MDLTrfIxGI0bjeQPLHS88mVm1oGhnIu9yZf8PJfw64zRJC89wOsN6b3ZIZyNt+pkAqChVnEqz6MsAbfqaaBbrTOaOcpY9k4t3c0fyj1rLRA50JTDKOmFa0UkLX437DYAzOdb7wg8nlPHRgBMA/OXHYJuYzGeq+GVaPt0e9cQ3zJpth97iwtb5RWTvK+dIQhmaAZrf/EcG0wshhBBCCCEaivR4XwOqe4WLior4+OOPsVgsbN++nTVr1gDoPdHV9uzZw/79+wH4+uuv9fUxMTEopdi4cSPjxo3jP//5D4mJiTzyyCMAJCQkABAXFwfAkiVLOHToEABKKXbv3g2c7ZmOiIggPT2dDRs21IihNl27dgWsQ+Hnzp1LYmIiiYmJxMfH88ILLzB69Gib8g31mLQB03zp9pgnnsEOnM6w4OrrQOz97oyY669PmlYbg4PGiLkBxN7vjquvdV/PYAe6PebJgNd99XJVFjidYeF0hoWq3+dbqyhR+rrzJbxzGpOXgZsf9tDX3fqYJ5EDXflmfA6H4ku5c6oPAW2cr1wjCCGEEEIIIeqNppT6A9NBiYZw/PhxOnTooPd4W4ceK315xYoVDBw4kHHjxvHJJ5/g5uZGVVUV4eHhHDhwgKqqKjp27MiOHTuorKzEyckJDw8PmjdvjsFgIDk5maqqKl566SWmTZtWY3K1iIgIsrOz6d69O8uWLWPy5Mn87//+L4A+gVtpaakeX3Vsffr0Ye3atYwdO5YFCxZgNpu5+eab2b17NwaDgcjISCoqKjh69Chms5m0tDTCwsL08+jduzfx8fGX12hTh8GEt6Bpqz/W+EIIIYQQQgjxB0mP9zUgJCSEjRs3ct999xEYGIjBYMDLy4u+ffvy448/MnDgQJvyXbp04d1336W4uBgnJyfuuOMOli1bhqZpODg48NhjjxEeHk5mZiaHDh0iLCyM559/nldffRWAW2+9lQ0bNjBo0CDc3d1JSUnB3d2dHj16APDSSy8xduxYvL29KSwsZOTIkTz++OMXPQ+j0cjatWt56qmnaN68OampqeTn59OlSxemTZtm99ByIYQQQgghhLiWSI/3deSK9BRfL6THWwghhBBCCHGVkB5vcX3qfR94+Fy8nBBCCCGEEELUM5nVXFyfbhvZ2BEIIYQQQgghBCBDzYUQQgghhBBCiHolQ82FEEIIIYQQQoh6JIm3EEIIIYQQQghRjyTxFkIIIYQQQggh6pEk3kIIIYQQQgghRD2SxFsIIYQQQgghhKhHkngLIYQQQgghhBD1SBJvIYQQQgghhBCiHkniLYQQQgghhBBC1CNJvIUQQgghhBBCiHokibcQQgghhBBCCFGPJPEW1x2z2czUqVMxm82NHcp1T9q64UhbNxxp64Yjbd1wpK0bjrR1w5G2bjjS1n+cppRSjR2EEFdSYWEhXl5eFBQU4Onp2djhXNekrRuOtHXDkbZuONLWDUfauuFIWzccaeuGI239x0mPtxBCCCGEEEIIUY8k8RZCCCGEEEIIIeqRJN5CCCGEEEIIIUQ9ksRbXHeMRiNTpkzBaDQ2dijXPWnrhiNt3XCkrRuOtHXDkbZuONLWDUfauuFIW/9xMrmaEEIIIYQQQghRj6THWwghhBBCCCGEqEeSeAshhBBCCCGEEPVIEm8hRL2aOnUqmqYRFhbW2KFc98aNG4emafTp06exQ7nuSVs3Lk3T0DSNBQsWNHYo170+ffqgaRrjxo1r7FCue2FhYWiaxtSpUxs7lOuetHXDkffrsyTxFteNr776itjYWEwmE76+vtxzzz0cPny4scO66s2cOZM+ffoQHByM0WgkNDSUsWPHcuTIEZty1R++zn/16NGjkSK/9lR/CVHby2Kx6OUqKip47bXXaNmyJc7OzoSEhPDss89y5syZRoz+2pGenl5nO5//Qav6w9f5rzFjxjTeCVzFEhISuOuuuwgICNDb6oMPPqhRzt5r+NChQ9xzzz34+vpiMpmIjY3l66+/bqjTuarZ09ZFRUU888wzdO7cGX9/f0wmExEREbzyyisUFRXZlK3r9+Hll19uyNO6Ktl7Xdv7d/DkyZM8/PDDNGnSBKPRSFRUFO+9915Dnc5VzZ62XrBgwQXfw+Pj44ELv9d//PHHjXB2Vxd7P9/J+3XDcWzsAIS4EubNm8f48eMBCA8PJy8vjyVLlrBu3Tp27dpFUFBQI0d49Zo9ezbHjh2jbdu2mEwm0tLS+PTTT1m1ahUpKSl4enralG/ZsiUBAQH6z9HR0Q0d8jXP39+fVq1a2azTNE1ffvjhh/n8888xGAy0adOGI0eOMGvWLJKSkvj1118xGOQ70wsxGo3ExcXZrDt9+jQpKSkABAcH19gnMjLS5lpv3bp1/QZ5jdqxYwc///wzLVu2JDc3t85y9lzDJ06coHv37uTk5ODp6UlwcDBJSUmMHDmS4uJiHn744QY8s6uPPW2dl5fHv/71L4xGI+3atSMzM5ODBw/y+uuvs337dn744Yca+3Ts2NFmVuLmzZvX2zlcK+y9rqtd6O9gcXExvXv3JiUlBZPJRGhoKPv372fixInk5OTwj3/8o17O4VphT1sHBATUeA8/duwYJ06cAKj1M9355Zs0aXKFIr522fv5Tt6vG5AS4hpnNpuVv7+/AtSIESOUUkplZmYqDw8PBaiJEyc2coRXt9dff10dPXpU//mZZ55RgALU0qVL9fW9e/dWgJo/f/4l1T9lyhQFqNDQUKWUUsXFxapHjx4KUK1atVLHjh27EqdxTahui7Fjx9ZZZvv27Xr7z549Wyml1HfffaevW7JkSZ37jh07VgGqd+/eSimlTp48qdq1a6cA1bVrV5Wfn38Fz+ba8sQTTyhA+fj4qKKiIn19aGioAtSaNWsuqb4bta1zc3NVSUmJSktL06/J999/36aMvdfwxIkTFaA8PDxUZmamUkqpESNGKED5+/srs9lcZxzVdVW/Hy1dulQ5OjoqQL3++uv1cOYNz562PnHihHrzzTdVYWGhUkqp0tJSdcstt+jlT506pZetXpeWlnZJcVS/91e/b6WmpqqgoCAFqLvvvluVlZX9ofO8GtjT1krZ93dw5syZClCapqldu3YppZR67rnnFKCcnJxUdnZ2nftWvx9NmTJFKaXUxo0blZubmwLUhAkTVFVV1R86z6uBvW19vpiYGAWo/v376+vOreNS3Qhtbc/nO3m/bljSbSKueVu3btW/NR0xYgQATZs25ZZbbgFg5cqVjRbbtWDy5Mm0aNFC/7lnz576cm3Panz22WcxGo20bNmSCRMmcPLkSbuPZTabGTp0KOvXr6d169bEx8ffkL0tS5YswWQyERwczN13301SUpK+7ccff9SXq6/ngQMH4uLiAth/Pefn53PHHXdw4MAB4uLi+Pnnn/H29r5yJ3ENycvLY/78+QD89a9/xd3dvUaZESNG4OLiQkREBP/zP/9DYWGh3fXfSG3t5+eHyWS6YBl7r+Hqct26daNp06YADB8+HIDc3Fy2bdtmV0wrV65k5MiRWCwWpk+fzuTJky/hjK5e9rR1UFAQzz//PB4eHgC4uLjQtWtXAAwGA46ONQc2dunSBVdXV6Kjo5kxYwZms9numI4ePUq/fv3Izs5m8ODBLFmy5Lp4pq89bX2uC/0drL6u27Rpw0033QSc/T2oqKhg9erVdh0jKSmJAQMGUFxczF//+lc++OADm5FR16pLbWuw/o7v2bMHgBdeeKHWMgEBAbi7u9OpUyc+/PBDqqqq7K7/em1rez7fyft1w5LEW1zzMjIy9OVzhxYFBgYC1uFJwj6VlZV8+OGHgHUoXb9+/Wy2m0wmmjVrRkBAAGlpaXz00Ud069aN4uLii9ZtsVi47777+Pnnn2nTpg3x8fGEhITUy3lczRwcHAgKCiIsLIzs7Gy+//57unXrpifftV3PBoMBf39/wL7r+cyZMwwYMIBdu3Zxyy23sGrVKry8vOrhbK4Nc+fOpaSkBKPRyMSJE2ts9/DwoFmzZnh5eXHw4EHefPNN7rzzTrs+uElb12TvNVxdrrb37XPLXUhCQgLDhw+nvLycN954gxdffPGPn8A1LCcnhyVLlgAwcuRIPSGv5uPjQ0hICEajkeTkZCZNmsSDDz5oV93Z2dn069ePjIwMhgwZwuLFi3F2dr7i53C1u9jfwStxXe/fv5877riDgoICHn/8cebMmXNdJIKX68033wSgQ4cO9O/fv8b2Jk2a6Mngzp07efTRR5k0aZJddd8obV3X5zt5v25YkniL65ZSqrFDuKYUFxczbNgwfvrpJ4KCglixYoVNT8Y777xDfn4+e/fuJSMjQ/+jlpaWxrfffnvR+jMzM1m+fDnu7u6sWbOGZs2a1du5XK1Gjx5NTk4OBw8eZP/+/fo3yWazmTlz5lxw30u5nrdv387mzZsJDQ3lp59+qnGf/o3k3LYdM2ZMjXsDFy9eTH5+Prt37yYzM5MHHngAgMTERDZu3HjR+qWt7WfPNXyp79vz58+ntLSUZ599lv/5n/+53NCuC4cPH6ZHjx5kZWXRvXv3GhNWJSYmkpeXx86dO8nMzKRv374AfPPNNzYfvuvy008/cfjwYW6++WYWLVqEk5NTvZzH1exy/w5e6nX9zTffkJuby/Dhw6/bRNBe1fcZAzz//PM22wICAti9ezcnT55k165dHDt2jKioKMB6f3N5eflF678R2vpin+9qI+/X9UMSb3HNO3eock5OTo3lc4fZiNplZ2fTu3dvVqxYQUREBBs2bND/eFXr1KmT/kataRqjR4/Wt9nzTaeLiwsODg6cOXOGWbNmXdH4rxURERH4+vrqP9955534+fkBZ9uwtuu5qqqKvLw8wL7r2c3NDbAOC/3ss8+uTPDXqE8//ZSTJ0+iaRp/+9vfamzv0qULDg4OADg6OnLvvffq2+y5rqWta7L3Gq4uV9v79rnlLqT6toGFCxfe0E+x2LRpE7fccgsHDx5k0KBBrFq1qkZvd1xcnJ5UuLq6MmzYMH2bPYl3dVvXNWnbjcCev4NX8rpetWoVmzdv/uOBX8PeeustwNquI0eOtNnm5uZGTEyM/rOvry8DBgwAoLS01K6J8q73tr7Y5zt5v25YkniLa17Xrl315KV6iF1WVhaJiYkA/OlPf2q02K4F+/bt45ZbbmH79u307NmTTZs20bJlS5syOTk5vP322zaPpzn3ERL2PKM7MDBQ74F56623mDFjxpU5gWvIG2+8YZPM/fzzz/oft+o2PPd6rb6ev//+e8rKympsr0uXLl30RwRNnDiRhQsXXpH4rzVKKWbOnAlY71mLjIy02b5v3z7mzZun3+NaWVnJ4sWL9e32XNfS1jXZew1X/7tp0yaysrIAWLp0KWCd+b9Lly4XPdY///lP2rdvT3Z2Nv3799dnPb6RLF68mL59+5Kbm8vEiRNZtmwZrq6uNmUSEhJYvHgxlZWVAJSVlbF8+XJ9e2ho6EWPM3z4cB544AEqKysZOXIka9euvbIncpWz9+9g9XV98OBBdu/eDZz9PXBycqpxC1dtJk6cSN++fTlz5gx33XUXycnJV+o0rinHjh3jm2++AeDpp5+uMWfB8uXLWbVqlf7z6dOn9ZFkbm5uNjPP1+V6bmt7Pt/J+3UDa7x53YS4cv7973/rsyaGh4crT09PfabF6tkXRe0iIiL0tuvYsaOKi4vTXx999JFS6uzMoY6Ojqpdu3aqefPm+j6RkZGqtLS0zvrPn9V86tSp+r7V9d8oQkNDlaZpqkWLFioyMlJpmqYA5ebmpvbt26eXGzVqlAKUwWBQ7dq1U05OTgpQPXv2VJWVlXXWf/5M2w899JA+k+4PP/xQ36d31Vm+fLl+ra1du7bG9jVr1ihAGY1GFR0drQIDA/Xyffv2veCstjdqWy9ZskS1atVKnxEYUAEBAapVq1Zq9OjRejl7ruHjx4/rT6Tw9PRU4eHhep0ffvjhBeOoLjd//nyVkZGhQkJCFKBiYmJsZvK+ltnT1pmZmfr7iLOzs837d1xcnNq+fbtSSqn58+fr7zUxMTHKx8dHr/Ohhx66YBznzmpeXl6u+vfvr/+fVdd/rbOnre39O1hUVKTatGmjAGUymWz+xr700ksXjOPcmbYLCgpUhw4dFKCaNWum0tPT670dGoK97yFKKfXss88qQHl5eekz95+r+vOFl5eXuummm5S7u7te52uvvXbBOG6Etrbn851S8n7dkCTxFteNzz//XHXs2FEZjUbl5eWlhg8frlJTUxs7rKveuX/8zn9VP2bjzJkzavLkyapr167K19dXmUwm1a5dO/Xiiy9e9E3z/MRbKaX+8pe/6G/yixYtqsezu7r8+9//VrfffrsKDg5WRqNRhYWFqfvvv18dOHDAplx5ebl69dVXVVhYmHJyclJNmzZVTz31VK0fPM51fjJYUVGhBgwYoADl6uqq1q9fX1+ndlXq2bOnAtTNN99c6/bs7Gz13HPPqZtuukl5eXkpd3d3FRMTo6ZPn65KSkouWPeN2tbVCVxtr+q2UMr+azglJUUNHz5ceXl5KaPRqDp27Ki++OKLi8Zx7gc5pZTau3ev8vb2VoDq1q2bKi4uvpKn3SjsaetzH6dU26v6MXkHDx5Ujz32mIqMjFTu7u7Ky8tLde7cWX3wwQeqoqLignGc/zixwsJCFRsbqydMKSkp9dgKDcOetr6Uv4NZWVlq7Nixyt/fXzk5Oal27dqpWbNmXTSO8x9xlZWVpa9r06aNOnny5JU+9QZn73vI6dOn9cfCvvDCC7XWtW3bNjV27FjVunVr5erqqvz8/NStt96qvv7664vGcSO0tT2f75SS9+uGpCklM1AJIYQQQgghhBD1Re7xFkIIIYQQQggh6pEk3kIIIYQQQgghRD2SxFsIIYQQQgghhKhHkngLIYQQQgghhBD1SBJvIYQQQgghhBCiHkniLYQQQgghhBBC1CNJvIUQQgghhBBCiHokibcQQgghhBBCCFGPJPEWQgghhBBCCCHqkSTeQgghhGgQZWVlvP3228TFxeHp6YmrqysRERE8+uijHDlypNHi0jQNTdNYsGBBo8UghBDi+ubY2AEIIYQQ4vqXn59Pv379SEpKAsDDw4NWrVpx7NgxPvzwQ7p160bLli0bOUohhBCifkiPtxBCCCHq3ZNPPqkn3S+88AKnTp1iz549FBQUsHbtWtq2bQvAd999R48ePXB3d8fFxYVOnToxb948m7pq66Hu06cPmqYxbtw4ANLT023K3X333bi6uhIeHq7XFx8fj6Zpeh0PPfQQmqYRFhYGQEpKCoMHD6ZJkyYYjUZCQkIYMGAAW7ZsqadWEkIIcb2SxFsIIYQQ9aqgoIBvvvkGgA4dOvDGG2/g6Hh20F2vXr3o1q0bn3/+OUOGDGHDhg24u7sTFBTEzp07GT9+PNOmTbvs40+YMIF9+/bh5OREeno6EyZM4MCBA3h6ehIXF6eXa9myJXFxcXTq1AmAUaNGsWLFCiwWC9HR0VRVVbFy5UqSk5MvOxYhhBA3Jkm8hRBCCFGvUlNTsVgsAPTs2dOml/lckydPBiAuLo6jR4+SlpbGsGHDAJg2bRolJSWXdfwhQ4Zw5MgR1q1bB0BVVRXx8fHExsaSmJiol3vllVdITEzk22+/BeDgwYMArFixgh07dpCVlcWRI0fo06fPZcUhhBDixiWJtxBCCCHqlVJKX64r6c7JyeHYsWMADB8+HKPRiKZpjBw5EoDS0lL27dt3Wce///770TSNqKgofd3Jkycvut+gQYMAuO2224iMjGTEiBGsXLmS4ODgy4pDCCHEjUsmVxNCCCFEvWrbti2Ojo5YLBbWr1+PUqrOBPxSVFZW6ssFBQV1lvP29gawGd5+7pcBdfn0008ZPHgw8fHxJCcn88MPP7B06VL27t3LnDlzLj9wIYQQNxzp8RZCCCFEvfLy8uLee+8FICkpiZdeekkfeg7wyy+/cOjQIVq0aAHA0qVLMZvNKKX46quvADCZTERHRwPQpEkTwDqEHeDAgQPs2bPnsuMzmUwAFBcX26xft24dw4YN44MPPiAhIYEpU6YAkJCQcNnHEkIIcWOSxFsIIYQQ9W727Nl07NgRgBkzZuDn50eHDh3w9fWlf//+pKam6hOobd68mdDQUMLDw/X7rSdPnoyrqysA/fr1A2DmzJncdtttdOvWza4e7Lq0a9cOgBdffJGbb76Zl156CYAHHngAHx8f2rZtS6dOnXj11VcBuOmmmy77WEIIIW5MkngLIYQQot75+vqyadMm3nrrLbp27UpVVRUpKSn4+Pgwfvx4evXqxZgxY1i+fDndu3enqKiI7OxsOnbsyMcff6xPvAbw9ttvM3DgQEwmE4cPH+all16iR48elx3bu+++S0xMDOXl5WzdulXvSX/ooYeIjo4mNzeX5ORkgoKCmDBhAu+9994fbg8hhBA3Fk39ka+IhRBCCCGEEEIIcUHS4y2EEEIIIYQQQtQjSbyFEEIIIYQQQoh6JIm3EEIIIYQQQghRjyTxFkIIIYQQQggh6pEk3kIIIYQQQgghRD2SxFsIIYQQQgghhKhHkngLIYQQQgghhBD1SBJvIYQQQgghhBCiHkniLYQQQgghhBBC1CNJvIUQQgghhBBCiHokibcQQgghhBBCCFGPJPEWQgghhBBCCCHq0f8DlvnWh8YdVxYAAAAASUVORK5CYII=",
+      "text/plain": [
+       "<Figure size 1000x350 with 1 Axes>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "csv_file = current_dir.parent / \"data/CSV/Models/Civi_models.csv\"\n",
+    "out_file = current_dir.parent / \"plots/Figure_12.svg\"\n",
+    "sortByFrequency_model_types_csv(csv_file, out_file)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e91e903f",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/SuppM_Figure_S13_Danbooru_Taxonomy.ipynb

@@ -0,0 +1,1848 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "68a6e559-6ef4-4c7f-8736-93bc8588e3bc",
+   "metadata": {},
+   "source": [
+    "# \"Danbooru\" Taxonomy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "36aff030",
+   "metadata": {},
+   "source": [
+    "### Getting Tags and Categories from Danbooru.donmai.us"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "a97e3a64-5990-44a9-809d-45030e85f161",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:07.782534Z",
+     "iopub.status.busy": "2025-03-17T12:48:07.781679Z",
+     "iopub.status.idle": "2025-03-17T12:48:07.786016Z",
+     "shell.execute_reply": "2025-03-17T12:48:07.785465Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:07.782509Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "import subprocess\n",
+    "import json\n",
+    "import urllib.parse"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97917f01-8f5b-4aa7-8869-9a7d853843d0",
+   "metadata": {},
+   "source": [
+    "#### Paths credentials"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "1f50a910-66c2-472a-8027-dd589b6ae755",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:08.799790Z",
+     "iopub.status.busy": "2025-03-17T12:48:08.798840Z",
+     "iopub.status.idle": "2025-03-17T12:48:08.905544Z",
+     "shell.execute_reply": "2025-03-17T12:48:08.904995Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:08.799770Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "current_dir = Path.cwd()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "a0f846e3-1fcc-4839-a459-c974f9c46d06",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:09.490977Z",
+     "iopub.status.busy": "2025-03-17T12:48:09.490799Z",
+     "iopub.status.idle": "2025-03-17T12:48:09.701470Z",
+     "shell.execute_reply": "2025-03-17T12:48:09.700912Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:09.490961Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "api_key = (current_dir.parent / \"misc/credentials/api-key_danbooru\").read_text().strip()\n",
+    "username = (current_dir.parent / \"misc/credentials/username_danbooru\").read_text().strip()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4c2290f-7705-4061-9b29-e1b74da732b8",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-12T20:39:34.470218Z",
+     "iopub.status.busy": "2025-03-12T20:39:34.469746Z",
+     "iopub.status.idle": "2025-03-12T20:39:34.472498Z",
+     "shell.execute_reply": "2025-03-12T20:39:34.472084Z",
+     "shell.execute_reply.started": "2025-03-12T20:39:34.470199Z"
+    }
+   },
+   "source": [
+    "### Query danbooru.donmai.us for a single Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "00ffaf5f-a4ae-408a-8a7c-e6b174f091f8",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:11.712064Z",
+     "iopub.status.busy": "2025-03-17T12:48:11.711866Z",
+     "iopub.status.idle": "2025-03-17T12:48:20.143088Z",
+     "shell.execute_reply": "2025-03-17T12:48:20.142318Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:11.712047Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "🚀 Fetching tag details for 'wombat'...\n",
+      "🔎 Running cURL: curl -s -L --user parodyofsomething:FkzGApb17bfJayJMqKTzeyTw -X GET https://danbooru.donmai.us/tags.json?search%5Bname%5D=wombat&only=id,name,category,post_count,is_deprecated,created_at,updated_at,wiki_page,artist,antecedent_alias,consequent_aliases,antecedent_implications,consequent_implications,dtext_links\n",
+      "✅ Test complete! Data saved to `/home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/misc/danbooru/tag_info_test/wombat_tag_info.json`.\n"
+     ]
+    }
+   ],
+   "source": [
+    "output_dir = current_dir.parent / \"misc\" / \"danbooru\" / \"tag_info_test\"\n",
+    "output_dir.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "# Get the tag to test\n",
+    "tag_name = input(\"Enter a single tag to test: \").strip()\n",
+    "if not tag_name:\n",
+    "    print(\"❌ ERROR: No tag entered.\")\n",
+    "    exit()\n",
+    "\n",
+    "# Encode the tag name properly\n",
+    "encoded_tag = urllib.parse.quote(tag_name, safe=\"\")  # Encode everything\n",
+    "\n",
+    "# API endpoints with correct encoding\n",
+    "BASE_URL = \"https://danbooru.donmai.us\"\n",
+    "#TAGS_API = f\"{BASE_URL}/tags.json?search%5Bname%5D={encoded_tag}\"\n",
+    "TAGS_API = f\"{BASE_URL}/tags.json?search%5Bname%5D={encoded_tag}&only=id,name,category,post_count,is_deprecated,created_at,updated_at,wiki_page,artist,antecedent_alias,consequent_aliases,antecedent_implications,consequent_implications,dtext_links\"\n",
+    "IMPLICATIONS_API = f\"{BASE_URL}/tag_implications.json?search%5Bantecedent_name%5D={encoded_tag}\"\n",
+    "ALIASES_API = f\"{BASE_URL}/tag_aliases.json?search%5Bantecedent_name%5D={encoded_tag}\"\n",
+    "WIKI_API = f\"{BASE_URL}/wiki_pages.json?search%5Btitle%5D={encoded_tag}\"\n",
+    "\n",
+    "# Function to run cURL command properly\n",
+    "def fetch_data(api_url, description):\n",
+    "    print(f\"🚀 Fetching {description} for '{tag_name}'...\")\n",
+    "\n",
+    "    # Build the cURL command\n",
+    "    curl_command = [\n",
+    "        \"curl\", \"-s\", \"-L\",  # Silent & follow redirects\n",
+    "        \"--user\", f\"{username}:{api_key}\",  # Auth\n",
+    "        \"-X\", \"GET\",\n",
+    "        api_url\n",
+    "    ]\n",
+    "\n",
+    "    print(f\"🔎 Running cURL: {' '.join(curl_command)}\")  # Show the command\n",
+    "\n",
+    "    try:\n",
+    "        result = subprocess.run(curl_command, capture_output=True, text=True, check=True)\n",
+    "        if result.returncode == 0:\n",
+    "            return json.loads(result.stdout) if result.stdout else None\n",
+    "        else:\n",
+    "            print(f\"⚠️ cURL failed (status {result.returncode}): {result.stderr}\")\n",
+    "            return None\n",
+    "    except subprocess.CalledProcessError as e:\n",
+    "        print(f\"⚠️ cURL error: {e}\")\n",
+    "        return None\n",
+    "    except json.JSONDecodeError:\n",
+    "        print(f\"⚠️ Failed to parse JSON: {result.stdout}\")\n",
+    "        return None\n",
+    "\n",
+    "# Fetch tag details\n",
+    "tag_info = {\n",
+    "    \"tag_details\": fetch_data(TAGS_API, \"tag details\"),\n",
+    "    #\"implications\": fetch_data(IMPLICATIONS_API, \"tag implications\"),\n",
+    "    #\"aliases\": fetch_data(ALIASES_API, \"tag aliases\"),\n",
+    "    #\"wiki\": fetch_data(WIKI_API, \"wiki information\"),\n",
+    "}\n",
+    "\n",
+    "# Save to JSON\n",
+    "output_file = output_dir / f\"{tag_name}_tag_info.json\"\n",
+    "with open(output_file, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(tag_info, f, indent=4, ensure_ascii=False)\n",
+    "\n",
+    "print(f\"✅ Test complete! Data saved to `{output_file}`.\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "75a28e4c-e06f-4578-a338-4f0014953302",
+   "metadata": {},
+   "source": [
+    "## Hierarchy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "1a58f2d1-909b-41c0-bcb8-0a55e6835f53",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:42.365809Z",
+     "iopub.status.busy": "2025-03-17T12:48:42.365447Z",
+     "iopub.status.idle": "2025-03-17T12:48:42.376305Z",
+     "shell.execute_reply": "2025-03-17T12:48:42.375752Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:42.365788Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "manual_hierarchy = {\n",
+    "    \"subject\": {\n",
+    "        \"female\": {\n",
+    "            \"female_general\": [\"woman\", \"girl\", \"1girl\", \"2girls\", \"3girls\", \"4girls\", \"5girls\", \"6+girls\", \"multiple girls\"]\n",
+    "        },\n",
+    "        \"male\": {\n",
+    "            \"male_general\": [\"man\", \"boy\", \"1boy\", \"2boys\", \"3boys\", \"4boys\", \"5boys\", \"6+boys\", \"multiple boys\"]\n",
+    "        },\n",
+    "        \"koma\": {\n",
+    "            \"koma_general\": [\"1koma\", \"2koma\"]\n",
+    "        },\n",
+    "        \"anthro\": {\n",
+    "            \"anthro_general\": [\"cat girl\", \"fox girl\", \"dog girl\", \"cat boy\", \"furry\"]\n",
+    "        }\n",
+    "    },\n",
+    "\n",
+    "    \"visual_characteristics\": {\n",
+    "        \"image_composition_and_style\": {\n",
+    "            \"artistic_license\": [],\n",
+    "            \"image_composition\": {\n",
+    "                \"backgrounds\": [],\n",
+    "                \"censorship\": [],\n",
+    "                \"colors\": [],\n",
+    "                \"focus_tags\": [],\n",
+    "                \"lighting\": [],\n",
+    "                \"prints\": []\n",
+    "            },\n",
+    "            \"patterns\": [],\n",
+    "            \"symbols\": [],\n",
+    "            \"text\": [],\n",
+    "            \"year_tags\": []\n",
+    "        }\n",
+    "    },\n",
+    "    \"body\": {\n",
+    "        \"body_parts\": {\n",
+    "            \"ass\": [],\n",
+    "            \"breasts_tags\": [],\n",
+    "            \"face_tags\": {\n",
+    "                \"eyes_tags\": []\n",
+    "            },\n",
+    "            \"ears_tags\": [],\n",
+    "            \"hair\": {\n",
+    "                \"hair_color\": [],\n",
+    "                \"hair_styles\": []\n",
+    "            },\n",
+    "            \"hands\": {\n",
+    "                \"gestures\": []\n",
+    "            },\n",
+    "            \"neck_and_neckwear\": [],\n",
+    "            \"posture\": [],\n",
+    "            \"pussy\": [],\n",
+    "            \"penis\": [],\n",
+    "            \"shoulders\": [],\n",
+    "            \"skin_color\": [],\n",
+    "            \"tail\": [],\n",
+    "            \"wings\": []\n",
+    "        },\n",
+    "        \"injury\": []\n",
+    "    },\n",
+    "    \"attire_and_body_accessories\": {\n",
+    "        \"attire\": {\n",
+    "            \"dress\": [],\n",
+    "            \"handwear\": [],\n",
+    "            \"headwear\": [],\n",
+    "            \"legwear\": [],\n",
+    "            \"neck_and_neckwear\": [],\n",
+    "            \"sexual_attire\": {\n",
+    "                \"bra\": [],\n",
+    "                \"panties\": []\n",
+    "            },\n",
+    "            \"sleeves\": [],\n",
+    "            \"swimsuit\": []\n",
+    "        },\n",
+    "        \"embellishment\": [],\n",
+    "        \"eyewear\": [],\n",
+    "        \"fashion_style\": [],\n",
+    "        \"nudity\": []\n",
+    "    },\n",
+    "    \"sex\": {\n",
+    "        \"sex_acts\": {\n",
+    "            \"simulated_sex_acts\": []\n",
+    "        },\n",
+    "        \"sexual_positions\": []\n",
+    "    },\n",
+    "    \"objects\": {\n",
+    "        \"computer\": [],\n",
+    "        \"airplanes\": [],\n",
+    "        \"armor\": [],\n",
+    "        \"ground_vehicles\": [],\n",
+    "        \"helicopters\": [],\n",
+    "        \"pokemon_objects\": [],\n",
+    "        \"ships\": [],\n",
+    "        \"weapons\": [],\n",
+    "        \"audio_tags\": [],\n",
+    "        \"cards\": {\n",
+    "            \"playing_card_faces\": []\n",
+    "        },\n",
+    "        \"eyewear\": [],\n",
+    "        \"piercings\": [],\n",
+    "        \"sex_objects\": []\n",
+    "    },\n",
+    "    \"creatures\": {\n",
+    "        \"animals\": {\n",
+    "            \"birds\": [],\n",
+    "            \"cats\": [],\n",
+    "            \"dogs\": []\n",
+    "        },\n",
+    "        \"legendary_creatures\": []\n",
+    "    },\n",
+    "    \"plants\": {\n",
+    "        \"plant\": {\n",
+    "            \"tree\": []\n",
+    "        },\n",
+    "        \"flowers\": []\n",
+    "    },\n",
+    "    \"games\": {\n",
+    "        \"game_activities\": [],\n",
+    "        \"board_games\": [],\n",
+    "        \"sports\": [],\n",
+    "        \"video_game\": []\n",
+    "    },\n",
+    "    \"real_world\": {\n",
+    "        \"companies_and_brand_names\": [],\n",
+    "        \"holidays_and_celebrations\": [],\n",
+    "        \"jobs\": [],\n",
+    "        \"locations\": [],\n",
+    "        \"people\": [],\n",
+    "        \"real_world_locations\": []\n",
+    "    },\n",
+    "    \"more\": {\n",
+    "        \"dances\": [],\n",
+    "        \"family_relationships\": [],\n",
+    "        \"food_tags\": [],\n",
+    "        \"fire\": [],\n",
+    "        \"groups\": [],\n",
+    "        \"phrases\": [],\n",
+    "        \"scan\": [],\n",
+    "        \"subjective\": [],\n",
+    "        \"technology\": [],\n",
+    "        \"verbs_and_gerunds\": [],\n",
+    "        \"water\": [],\n",
+    "        \"disambiguation_pages\": [],\n",
+    "        \"magazine_publications\": [],\n",
+    "        \"special_moves\": [],\n",
+    "        \"uniforms\": [],\n",
+    "        \"pokemon_media\": [],\n",
+    "        \"tagged_songs\": [],\n",
+    "        \"vocaloid_derivatives\": [],\n",
+    "        \"vocaloid_songs\": [],\n",
+    "        \"vocal_synthesizers\": [],\n",
+    "        \"vocal_synth_derivatives\": [],\n",
+    "        \"vocal_synth_songs\": [],\n",
+    "        \"deemo_songs\": []\n",
+    "    },\n",
+    "    \"copyrights_artists_projects_and_media\": {\n",
+    "        \"genres_of_video_games\": {\n",
+    "            \"fighting_games\": [],\n",
+    "            \"platform_games\": [],\n",
+    "            \"role-playing_games\": [],\n",
+    "            \"shooter_games\": [],\n",
+    "            \"visual_novel_games\": []\n",
+    "        },\n",
+    "        \"artists\": {\n",
+    "            \"named_drawfags\": [],\n",
+    "            \"pixiv_projects\": []\n",
+    "        }\n",
+    "    },\n",
+    "    \"characters\": {\n",
+    "        \"ace_attorney\": [],\n",
+    "        \"arknights\": [],\n",
+    "        \"atelier\": [],\n",
+    "        \"azur_lane\": [],\n",
+    "        \"bleach\": [],\n",
+    "        \"bokujou_monogatari\": [],\n",
+    "        \"brave_girl_ravens\": [],\n",
+    "        \"cardcaptor_sakura\": [],\n",
+    "        \"danganronpa\": [],\n",
+    "        \"digimon\": {\n",
+    "            \"digimon_characters\": []\n",
+    "        },\n",
+    "        \"dragon_ball\": [],\n",
+    "        \"dragon_quest\": [],\n",
+    "        \"fate_series\": [],\n",
+    "        \"final_fantasy\": [],\n",
+    "        \"fire_emblem\": [],\n",
+    "        \"flower_knight_girl\": [],\n",
+    "        \"gensou_suikoden\": [],\n",
+    "        \"girls_frontline\": [],\n",
+    "        \"girls_und_panzer\": [],\n",
+    "        \"gundam_mechas\": [],\n",
+    "        \"hunter_x_hunter\": [],\n",
+    "        \"jojo_no_kimyou_na_bouken\": [],\n",
+    "        \"kamen_rider\": [],\n",
+    "        \"kantai_collection\": [],\n",
+    "        \"kingdom_hearts\": [],\n",
+    "        \"mahou_sensei_negima\": [],\n",
+    "        \"meitantei_conan\": [],\n",
+    "        \"minecraft\": [],\n",
+    "        \"naruto\": [],\n",
+    "        \"nippon_ichi\": [],\n",
+    "        \"one_piece\": [],\n",
+    "        \"oshiro_project\": [],\n",
+    "        \"pokemon\": {\n",
+    "            \"elite_four_members\": [],\n",
+    "            \"gym_leaders\": [],\n",
+    "            \"families_of_pokemon_main_characters\": [],\n",
+    "            \"pokemon_ranger_characters\": [],\n",
+    "            \"pokemon_trainer_classes\": []\n",
+    "        },\n",
+    "        \"pretty_cure\": [],\n",
+    "        \"ragnarok_online\": [],\n",
+    "        \"rosenkreuzstilette\": [],\n",
+    "        \"sailor_moon\": [],\n",
+    "        \"street_fighter\": [],\n",
+    "        \"super_smash_bros\": [],\n",
+    "        \"world_witches_series\": [],\n",
+    "        \"tales_of\": [],\n",
+    "        \"toaru_majutsu_no_index\": [],\n",
+    "        \"touhou\": [],\n",
+    "        \"touken_ranbu\": [],\n",
+    "        \"ultra_series\": [],\n",
+    "        \"umamusume\": [],\n",
+    "        \"vocaloid\": [],\n",
+    "        \"yu_gi_oh\": [],\n",
+    "        \"genderswap_characters\": [],\n",
+    "        \"official_mascots\": [],\n",
+    "        \"real_life_racehorses\": []\n",
+    "    },\n",
+    "    \"metatags\": {\n",
+    "        \"tag_group_metatags\": [],\n",
+    "        \"drawing_software\": []\n",
+    "    }\n",
+    "}\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "824760af-a568-47f8-b076-b30e8606b9ac",
+   "metadata": {},
+   "source": [
+    "## Tag Groups"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "557cdd69-f6b1-4c33-9f67-a13b00c0146b",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:43.777297Z",
+     "iopub.status.busy": "2025-03-17T12:48:43.776934Z",
+     "iopub.status.idle": "2025-03-17T12:48:43.787862Z",
+     "shell.execute_reply": "2025-03-17T12:48:43.787288Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:43.777279Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "tag_groups = {\n",
+    "    # **Subjects (Humans, Anthro, etc.)**\n",
+    "    \"subject\": \"subject\",\n",
+    "    \"subject.female\": \"female\",\n",
+    "    \"subject.female_general\": \"female_general\",  # Added\n",
+    "    \"subject.female.1girl\": \"1girl\",\n",
+    "    \"subject.female.2girls\": \"2girls\",\n",
+    "\n",
+    "\n",
+    "    # **Male Subjects**\n",
+    "    \"subject.male\": \"male\",\n",
+    "    \"subject.male.1boy\": \"1boy\",\n",
+    "    \"subject.male.2boys\": \"2boys\",\n",
+    "    \"subject.male.3boys\": \"3boys\",\n",
+    "    \"subject.male.4boys\": \"4boys\",\n",
+    "    \"subject.male.5boys\": \"5boys\",\n",
+    "    \"subject.male.6+boys\": \"6+boys\",\n",
+    "    \"subject.male.multiple_boys\": \"multiple_boys\",\n",
+    "    \"subject.male.man\": \"man\",\n",
+    "    \"subject.male.boy\": \"boy\",\n",
+    "\n",
+    "    # **Koma (Manga Panel Counts)**\n",
+    "    \"subject.koma\": \"koma\",\n",
+    "    \"subject.koma.1koma\": \"1koma\",\n",
+    "    \"subject.koma.2koma\": \"2koma\",\n",
+    "\n",
+    "    # **Anthropomorphic Characters**\n",
+    "    \"subject.anthro\": \"anthro\",\n",
+    "    \"subject.anthro.cat_girl\": \"cat_girl\",\n",
+    "    \"subject.anthro.fox_girl\": \"fox_girl\",\n",
+    "    \"subject.anthro.dog_girl\": \"dog_girl\",\n",
+    "    \"subject.anthro.cat_boy\": \"cat_boy\",\n",
+    "    \"subject.anthro.furry\": \"furry\",\n",
+    "\n",
+    "    # **Visual Characteristics**\n",
+    "    \"visual_characteristics\": \"visual_characteristics\",\n",
+    "    \"visual_characteristics.image_composition_and_style\": \"image_composition_and_style\",\n",
+    "    \"visual_characteristics.image_composition_and_style.artistic_license\": \"artistic_license\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition\": \"image_composition\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.backgrounds\": \"backgrounds\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.censorship\": \"censorship\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.colors\": \"colors\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.focus_tags\": \"focus_tags\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.lighting\": \"lighting\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.prints\": \"prints\",\n",
+    "    \"visual_characteristics.image_composition_and_style.image_composition.style_parodies\": \"style_parodies\",\n",
+    "    \"visual_characteristics.image_composition_and_style.patterns\": \"patterns\",\n",
+    "    \"visual_characteristics.image_composition_and_style.symbols\": \"symbols\",\n",
+    "    \"visual_characteristics.image_composition_and_style.text\": \"text\",\n",
+    "    \"visual_characteristics.image_composition_and_style.year_tags\": \"year_tags\",\n",
+    "\n",
+    "\n",
+    "    # Body\n",
+    "    \"body\": \"body\",\n",
+    "    \"body.body_parts\": \"body_parts\",\n",
+    "    \"body.body_parts.ass\": \"ass\",\n",
+    "    \"body.body_parts.breasts_tags\": \"breasts_tags\",\n",
+    "    \"body.body_parts.face_tags\": \"face_tags\",\n",
+    "    \"body.body_parts.face_tags.eyes_tags\": \"eyes_tags\",\n",
+    "    \"body.body_parts.ears_tags\": \"ears_tags\",\n",
+    "    \"body.body_parts.hair\": \"hair\",\n",
+    "    \"body.body_parts.hair.hair_color\": \"hair_color\",\n",
+    "    \"body.body_parts.hair.hair_styles\": \"hair_styles\",\n",
+    "    \"body.body_parts.hands\": \"hands\",\n",
+    "    \"body.body_parts.hands.gestures\": \"gestures\",\n",
+    "    \"body.body_parts.neck_and_neckwear\": \"neck_and_neckwear\",\n",
+    "    \"body.body_parts.posture\": \"posture\",\n",
+    "    \"body.body_parts.pussy\": \"pussy\",\n",
+    "    \"body.body_parts.penis\": \"penis\",\n",
+    "    \"body.body_parts.shoulders\": \"shoulders\",\n",
+    "    \"body.body_parts.skin_color\": \"skin_color\",\n",
+    "    \"body.body_parts.tail\": \"tail\",\n",
+    "    \"body.body_parts.wings\": \"wings\",\n",
+    "    \"body.injury\": \"injury\",\n",
+    "\n",
+    "    # Attire & Accessories\n",
+    "    \"attire_and_body_accessories\": \"attire_and_body_accessories\",\n",
+    "    \"attire_and_body_accessories.attire\": \"attire\",\n",
+    "    \"attire_and_body_accessories.attire.dress\": \"dress\",\n",
+    "    \"attire_and_body_accessories.attire.handwear\": \"handwear\",\n",
+    "    \"attire_and_body_accessories.attire.headwear\": \"headwear\",\n",
+    "    \"attire_and_body_accessories.attire.legwear\": \"legwear\",\n",
+    "    \"attire_and_body_accessories.attire.mask\": \"mask\",\n",
+    "    \"attire_and_body_accessories.attire.neck_and_neckwear\": \"neck_and_neckwear\",\n",
+    "    \"attire_and_body_accessories.attire.sexual_attire\": \"sexual_attire\",\n",
+    "    \"attire_and_body_accessories.attire.sexual_attire.bra\": \"bra\",\n",
+    "    \"attire_and_body_accessories.attire.sexual_attire.panties\": \"panties\",\n",
+    "    \"attire_and_body_accessories.attire.sleeves\": \"sleeves\",\n",
+    "    \"attire_and_body_accessories.attire.swimsuit\": \"swimsuit\",\n",
+    "    \"attire_and_body_accessories.embellishment\": \"embellishment\",\n",
+    "    \"attire_and_body_accessories.eyewear\": \"eyewear\",\n",
+    "    \"attire_and_body_accessories.fashion_style\": \"fashion_style\",\n",
+    "    \"attire_and_body_accessories.nudity\": \"nudity\",\n",
+    "\n",
+    "    # Sex\n",
+    "    \"sex\": \"sex\",\n",
+    "    \"sex.sex_acts\": \"sex_acts\",\n",
+    "    \"sex.sex_acts.simulated_sex_acts\": \"simulated_sex_acts\",\n",
+    "    \"sex.sexual_positions\": \"sexual_positions\",\n",
+    "\n",
+    "    # Objects\n",
+    "    \"objects\": \"objects\",\n",
+    "    \"objects.computer\": \"computer\",\n",
+    "    \"objects.airplanes\": \"airplanes\",\n",
+    "    \"objects.armor\": \"armor\",\n",
+    "    \"objects.ground_vehicles\": \"ground_vehicles\",\n",
+    "    \"objects.helicopters\": \"helicopters\",\n",
+    "    \"objects.pokemon_objects\": \"pokemon_objects\",\n",
+    "    \"objects.ships\": \"ships\",\n",
+    "    \"objects.weapons\": \"weapons\",\n",
+    "    \"objects.audio_tags\": \"audio_tags\",\n",
+    "    \"objects.cards\": \"cards\",\n",
+    "    \"objects.cards.playing_card_faces\": \"playing_card_faces\",\n",
+    "    \"objects.eyewear\": \"eyewear\",\n",
+    "    \"objects.piercings\": \"piercings\",\n",
+    "    \"objects.sex_objects\": \"sex_objects\",\n",
+    "\n",
+    "    # Creatures\n",
+    "    \"creatures\": \"creatures\",\n",
+    "    \"creatures.animals\": \"animals\",\n",
+    "    \"creatures.animals.birds\": \"birds\",\n",
+    "    \"creatures.animals.cats\": \"cats\",\n",
+    "    \"creatures.animals.dogs\": \"dogs\",\n",
+    "    \"creatures.legendary_creatures\": \"legendary_creatures\",\n",
+    "\n",
+    "    # Plants\n",
+    "    \"plants\": \"plant\",\n",
+    "    \"plant.plant\": \"plant\",\n",
+    "    \"plant.tree\": \"tree\",\n",
+    "    \"plant.flowers\": \"flowers\",\n",
+    "\n",
+    "    # Games\n",
+    "    \"games\": \"games\",\n",
+    "    \"games.game_activities\": \"game_activities\",\n",
+    "    \"games.board_games\": \"board_games\",\n",
+    "    \"games.sports\": \"sports\",\n",
+    "    \"games.video_game\": \"video_game\",\n",
+    "    \"games.fighting_games\": \"fighting_games\",\n",
+    "\n",
+    "    # Real World\n",
+    "    \"real_world\": \"real_world\",\n",
+    "    \"real_world.companies_and_brand_names\": \"companies_and_brand_names\",\n",
+    "    \"real_world.holidays_and_celebrations\": \"holidays_and_celebrations\",\n",
+    "    \"real_world.jobs\": \"jobs\",\n",
+    "    \"real_world.locations\": \"locations\",\n",
+    "    \"real_world.people\": \"people\",\n",
+    "    \"real_world.real_world_locations\": \"real_world_locations\",\n",
+    "\n",
+    "    # More Categories\n",
+    "    \"more\": \"more\",\n",
+    "    \"more.dances\": \"dances\",\n",
+    "    \"more.family_relationships\": \"family_relationships\",\n",
+    "    \"more.food_tags\": \"food_tags\",\n",
+    "    \"more.fire\": \"fire\",\n",
+    "    \"more.groups\": \"groups\",\n",
+    "    \"more.phrases\": \"phrases\",\n",
+    "    \"more.scan\": \"scan\",\n",
+    "    \"more.subjective\": \"subjective\",\n",
+    "    \"more.technology\": \"technology\",\n",
+    "    \"more.verbs_and_gerunds\": \"verbs_and_gerunds\",\n",
+    "    \"more.water\": \"water\",\n",
+    "\n",
+    "    # Genres of Video Games\n",
+    "    \"copyrights_artists_projects_and_media\": \"copyrights_artists_projects_and_media\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games\": \"genres_of_video_games\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games.fighting_games\": \"fighting_games\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games.platform_games\": \"platform_games\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games.role-playing_games\": \"role-playing_games\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games.shooter_games\": \"shooter_games\",\n",
+    "    \"copyrights_artists_projects_and_media.genres_of_video_games.visual_novel_games\": \"visual_novel_games\",\n",
+    "\n",
+    "    \"characters\": \"characters\",\n",
+    "    \"characters.ace_attorney\": \"ace_attorney_characters\",\n",
+    "    \"characters.arknights\": \"arknights_characters\",\n",
+    "    \"characters.atelier\": \"atelier_characters\",\n",
+    "    \"characters.azur_lane\": \"azur_lane_characters\",\n",
+    "    \"characters.bleach\": \"bleach_characters\",\n",
+    "    \"characters.bokujou_monogatari\": \"bokujou_monogatari_characters\",\n",
+    "    \"characters.brave_girl_ravens\": \"brave_girl_ravens_characters\",\n",
+    "    \"characters.cardcaptor_sakura\": \"cardcaptor_sakura_characters\",\n",
+    "    \"characters.danganronpa\": \"danganronpa_characters\",\n",
+    "    \"characters.digimon\": \"digimon\",\n",
+    "    \"characters.digimon.digimon_characters\": \"digimon_characters\",\n",
+    "    \"characters.dragon_ball\": \"dragon_ball_characters\",\n",
+    "    \"characters.dragon_quest\": \"dragon_quest_characters\",\n",
+    "    \"characters.fate_series\": \"fate_series_characters\",\n",
+    "    \"characters.final_fantasy\": \"final_fantasy_characters\",\n",
+    "    \"characters.fire_emblem\": \"fire_emblem_characters\",\n",
+    "    \"characters.flower_knight_girl\": \"flower_knight_girl_characters\",\n",
+    "    \"characters.gensou_suikoden\": \"gensou_suikoden_characters\",\n",
+    "    \"characters.girls_frontline\": \"girls_frontline_characters\",\n",
+    "    \"characters.girls_und_panzer\": \"girls_und_panzer_characters\",\n",
+    "    \"characters.gundam_mechas\": \"gundam_mechas\",\n",
+    "    \"characters.hunter_x_hunter\": \"hunter_x_hunter_characters\",\n",
+    "    \"characters.jojo_no_kimyou_na_bouken\": \"jojo_no_kimyou_na_bouken_characters\",\n",
+    "    \"characters.kamen_rider\": \"kamen_rider_characters\",\n",
+    "    \"characters.kantai_collection\": \"kantai_collection_characters\",\n",
+    "    \"characters.kingdom_hearts\": \"kingdom_hearts_characters\",\n",
+    "    \"characters.mahou_sensei_negima\": \"mahou_sensei_negima_characters\",\n",
+    "    \"characters.meitantei_conan\": \"meitantei_conan_characters\",\n",
+    "    \"characters.minecraft\": \"minecraft_characters\",\n",
+    "    \"characters.naruto\": \"naruto_characters\",\n",
+    "    \"characters.nippon_ichi\": \"nippon_ichi_characters\",\n",
+    "    \"characters.one_piece\": \"one_piece_characters\",\n",
+    "    \"characters.oshiro_project\": \"oshiro_project_characters\",\n",
+    "    \"characters.pokemon\": \"pokemon\",\n",
+    "    \"characters.pokemon.elite_four_members\": \"elite_four_members\",\n",
+    "    \"characters.pokemon.gym_leaders\": \"gym_leaders\",\n",
+    "    \"characters.pokemon.families_of_pokemon_main_characters\": \"families_of_pokemon_main_characters\",\n",
+    "    \"characters.pokemon.pokemon_ranger_characters\": \"pokemon_ranger_characters\",\n",
+    "    \"characters.pokemon.pokemon_trainer_classes\": \"pokemon_trainer_classes\",\n",
+    "    \"characters.pretty_cure\": \"pretty_cure_characters\",\n",
+    "    \"characters.ragnarok_online\": \"ragnarok_online_characters\",\n",
+    "    \"characters.rosenkreuzstilette\": \"rosenkreuzstilette_characters\",\n",
+    "    \"characters.sailor_moon\": \"sailor_moon_characters\",\n",
+    "    \"characters.street_fighter\": \"street_fighter_characters\",\n",
+    "    \"characters.super_smash_bros\": \"super_smash_bros_characters\",\n",
+    "    \"characters.world_witches_series\": \"world_witches_series_characters\",\n",
+    "    \"characters.tales_of\": \"tales_of_characters\",\n",
+    "    \"characters.toaru_majutsu_no_index\": \"toaru_majutsu_no_index_characters\",\n",
+    "    \"characters.touhou\": \"touhou_characters\",\n",
+    "    \"characters.touken_ranbu\": \"touken_ranbu_characters\",\n",
+    "    \"characters.ultra_series\": \"ultra_series_characters\",\n",
+    "    \"characters.umamusume\": \"umamusume_characters\",\n",
+    "    \"characters.vocaloid\": \"vocaloid_characters\",\n",
+    "    \"characters.yu_gi_oh\": \"yu_gi_oh_characters\",\n",
+    "    \"characters.genderswap\": \"genderswap_characters\",\n",
+    "    \"characters.official_mascots\": \"official_mascots\",\n",
+    "    \"characters.real_life_racehorses\": \"real_life_racehorses\",\n",
+    "\n",
+    "\n",
+    "    # metatags\n",
+    "    \"metatags\": \"metatags\",\n",
+    "    \"drawing_software\": \"drawing_software\",\n",
+    "    \n",
+    "}\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "c4759995-396a-4ec1-9f35-e75b3875e13c",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:44.138013Z",
+     "iopub.status.busy": "2025-03-17T12:48:44.137852Z",
+     "iopub.status.idle": "2025-03-17T12:48:44.491853Z",
+     "shell.execute_reply": "2025-03-17T12:48:44.491271Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:44.137999Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "list_based_categories = [\n",
+    "    # Image composition style\n",
+    "    \"style_parodies\",\n",
+    "    # Objects\n",
+    "    \"computer\",\n",
+    "    \"airplanes\",\n",
+    "    \"armor\",\n",
+    "    \"ground_vehicles\",\n",
+    "    \"helicopters\",\n",
+    "    \"pokemon_objects\",\n",
+    "    \"ships\",\n",
+    "    \"weapons\",\n",
+    "    \"playing_card_faces\",\n",
+    "    \n",
+    "    # Creatures\n",
+    "    \"animals\",\n",
+    "    \"birds\",\n",
+    "    \"cats\",\n",
+    "    \"dogs\",\n",
+    "    \"legendary_creatures\",\n",
+    "\n",
+    "    # Plants\n",
+    "    \"plant\",\n",
+    "    \"tree\",\n",
+    "    \"flowers\",\n",
+    "\n",
+    "    # Games\n",
+    "    \"game_activities\",\n",
+    "    \"board_games\",\n",
+    "    \"sports\",\n",
+    "    \"video_game\",\n",
+    "    \"fighting_games\",\n",
+    "    \"platform_games\",\n",
+    "    #\"role-playing_games\",\n",
+    "    \"shooter_games\",\n",
+    "    \"visual_novel_games\",\n",
+    "\n",
+    "    # Real World\n",
+    "    \"companies_and_brand_names\",\n",
+    "    \"holidays_and_celebrations\",\n",
+    "    \"jobs\",\n",
+    "    \"locations\",\n",
+    "    \"people\",\n",
+    "    \"real_world_locations\",\n",
+    "\n",
+    "    # More Categories\n",
+    "    \"dances\",\n",
+    "    \"family_relationships\",\n",
+    "    \"food_tags\",\n",
+    "    \"fire\",\n",
+    "    \"groups\",\n",
+    "    \"phrases\",\n",
+    "    \"scan\",\n",
+    "    \"subjective\",\n",
+    "    \"technology\",\n",
+    "    \"verbs_and_gerunds\",\n",
+    "    \"water\",\n",
+    "    \"airplanes\",\n",
+    "\n",
+    "    # Artists\n",
+    "    \"named_drawfags\",\n",
+    "    \"pixiv_projects\",\n",
+    "\n",
+    "    # Characters\n",
+    "    \"ace_attorney_characters\",\n",
+    "    \"arknights_characters\",\n",
+    "    \"atelier_characters\",\n",
+    "    \"azur_lane_characters\",\n",
+    "    \"bleach_characters\",\n",
+    "    \"bokujou_monogatari_characters\",\n",
+    "    \"brave_girl_ravens_characters\",\n",
+    "    \"cardcaptor_sakura_characters\",\n",
+    "    \"danganronpa_characters\",\n",
+    "    \"digimon\",\n",
+    "    \"digimon_characters\",\n",
+    "    \"dragon_ball_characters\",\n",
+    "    \"dragon_quest_characters\",\n",
+    "    \"fate_series_characters\",\n",
+    "    \"final_fantasy_characters\",\n",
+    "    \"fire_emblem_characters\",\n",
+    "    \"flower_knight_girl_characters\",\n",
+    "    \"gensou_suikoden_characters\",\n",
+    "    \"girls_frontline_characters\",\n",
+    "    \"girls_und_panzer_characters\",\n",
+    "    \"gundam_mechas\",\n",
+    "    \"hunter_x_hunter_characters\",\n",
+    "    \"jojo_no_kimyou_na_bouken_characters\",\n",
+    "    \"kamen_rider_characters\",\n",
+    "    \"kantai_collection_characters\",\n",
+    "    \"kingdom_hearts_characters\",\n",
+    "    \"mahou_sensei_negima_characters\",\n",
+    "    \"meitantei_conan_characters\",\n",
+    "    \"minecraft_characters\",\n",
+    "    \"naruto_characters\",\n",
+    "    \"nippon_ichi_characters\",\n",
+    "    \"one_piece_characters\",\n",
+    "    \"oshiro_project_characters\",\n",
+    "    \"pokemon_characters\",\n",
+    "    \"elite_four_members\",\n",
+    "    \"gym_leaders\",\n",
+    "    \"families_of_pokemon_main_characters\",\n",
+    "    \"pokemon_ranger_characters\",\n",
+    "    \"pokemon_trainer_classes\",\n",
+    "    \"pokemon\",\n",
+    "    \"pretty_cure_characters\",\n",
+    "    \"ragnarok_online_characters\",\n",
+    "    \"rosenkreuzstilette_characters\",\n",
+    "    \"sailor_moon_characters\",\n",
+    "    \"street_fighter_characters\",\n",
+    "    \"super_smash_bros_characters\",\n",
+    "    \"world_witches_series_characters\",\n",
+    "    \"tales_of_characters\",\n",
+    "    \"toaru_majutsu_no_index_characters\",\n",
+    "    \"touhou_characters\",\n",
+    "    \"touken_ranbu_characters\",\n",
+    "    \"ultra_series_characters\",\n",
+    "    \"umamusume_characters\",\n",
+    "    \"vocaloid_characters\",\n",
+    "    \"yu_gi_oh_characters\",\n",
+    "    \"genderswap_characters\",\n",
+    "    \"official_mascots\",\n",
+    "    \"real_life_racehorses\",\n",
+    "\n",
+    "\n",
+    "    \n",
+    "    # Other Lists\n",
+    "    \"disambiguation_pages\",\n",
+    "    \"magazine_publications\",\n",
+    "    \"special_moves\",\n",
+    "    \"uniforms\",\n",
+    "    \"pokemon_media\",\n",
+    "    \"tagged_songs\",\n",
+    "    \"vocaloid_derivatives\",\n",
+    "    \"vocaloid_songs\",\n",
+    "    \"vocal_synthesizers\",\n",
+    "    \"vocal_synth_derivatives\",\n",
+    "    \"vocal_synth_songs\",\n",
+    "    \"deemo_songs\",\n",
+    "    #\"role_playing_games\",\n",
+    "\n",
+    "    # Metatags\n",
+    "    #\"metatags\",\n",
+    "    #\"drawing_software\",\n",
+    "\n",
+    "    # Pool Groups & Meta-Wikis\n",
+    "    \"meta_wikis\"\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "8f8b6b29-e7bd-4814-8705-2b7a68f2d660",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:52.253272Z",
+     "iopub.status.busy": "2025-03-17T12:48:52.252077Z",
+     "iopub.status.idle": "2025-03-17T12:48:52.256578Z",
+     "shell.execute_reply": "2025-03-17T12:48:52.256016Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:52.253248Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "special_wiki_pages = [\n",
+    "    \"plant\", \"tree\", \"computer\", \"on_object\", \"injury\", \"swimsuit\", \"on\" , \"mask\" # Add more here if needed\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "79234bc5-b287-4306-85cd-a27ddba769ea",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:48:55.161786Z",
+     "iopub.status.busy": "2025-03-17T12:48:55.160800Z",
+     "iopub.status.idle": "2025-03-17T12:48:55.165137Z",
+     "shell.execute_reply": "2025-03-17T12:48:55.164568Z",
+     "shell.execute_reply.started": "2025-03-17T12:48:55.161760Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "HEADERS = {\n",
+    "    \"Authorization\": f\"Basic {base64.b64encode(f'{username}:{api_key}'.encode()).decode()}\"\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "id": "64d47678-78a4-4d55-a3a0-1c1b251b03b6",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:40:12.358876Z",
+     "iopub.status.busy": "2025-03-17T12:40:12.358573Z",
+     "iopub.status.idle": "2025-03-17T12:40:43.547891Z",
+     "shell.execute_reply": "2025-03-17T12:40:43.547451Z",
+     "shell.execute_reply.started": "2025-03-17T12:40:12.358851Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "🚀 Starting Danbooru Tag Hierarchy API Fetcher...\n",
+      "❌ No data found for attire_and_body_accessories in any format\n",
+      "✅ Added 91 tags directly under attire_and_body_accessories.attire.dress\n",
+      "✅ Added 100 tags directly under attire_and_body_accessories.attire.handwear\n",
+      "✅ Added 251 tags directly under attire_and_body_accessories.attire.headwear\n",
+      "✅ Added 69 tags directly under attire_and_body_accessories.attire.legwear\n",
+      "✅ Added 64 tags directly under attire_and_body_accessories.attire.mask\n",
+      "✅ Added 264 tags directly under attire_and_body_accessories.attire.neck_and_neckwear\n",
+      "✅ Added 54 tags directly under attire_and_body_accessories.attire.sexual_attire.bra\n",
+      "✅ Added 83 tags directly under attire_and_body_accessories.attire.sexual_attire.panties\n",
+      "✅ Added 87 tags directly under attire_and_body_accessories.attire.sleeves\n",
+      "✅ Added 83 tags directly under attire_and_body_accessories.attire.swimsuit\n",
+      "✅ Added 4 tags directly under attire_and_body_accessories.embellishment\n",
+      "✅ Added 94 tags directly under attire_and_body_accessories.eyewear\n",
+      "✅ Added 62 tags directly under attire_and_body_accessories.fashion_style\n",
+      "✅ Added 189 tags directly under attire_and_body_accessories.nudity\n",
+      "❌ No data found for body in any format\n",
+      "✅ Added 40 tags directly under body.body_parts.ass\n",
+      "✅ Added 213 tags directly under body.body_parts.breasts_tags\n",
+      "✅ Added 54 tags directly under body.body_parts.ears_tags\n",
+      "✅ Added 200 tags directly under body.body_parts.face_tags.eyes_tags\n",
+      "✅ Added 31 tags directly under body.body_parts.hair.hair_color\n",
+      "✅ Added 157 tags directly under body.body_parts.hair.hair_styles\n",
+      "✅ Added 117 tags directly under body.body_parts.hands.gestures\n",
+      "✅ Added 264 tags directly under body.body_parts.neck_and_neckwear\n",
+      "✅ Added 74 tags directly under body.body_parts.penis\n",
+      "✅ Added 230 tags directly under body.body_parts.posture\n",
+      "✅ Added 50 tags directly under body.body_parts.pussy\n",
+      "✅ Added 64 tags directly under body.body_parts.shoulders\n",
+      "✅ Added 26 tags directly under body.body_parts.skin_color\n",
+      "✅ Added 79 tags directly under body.body_parts.tail\n",
+      "✅ Added 90 tags directly under body.body_parts.wings\n",
+      "✅ Added 55 tags directly under body.injury\n",
+      "❌ No data found for characters in any format\n",
+      "✅ Added 300 tags directly under characters.ace_attorney\n",
+      "✅ Added 579 tags directly under characters.arknights\n",
+      "✅ Added 296 tags directly under characters.atelier\n",
+      "✅ Added 706 tags directly under characters.azur_lane\n",
+      "✅ Added 223 tags directly under characters.bleach\n",
+      "✅ Added 98 tags directly under characters.bokujou_monogatari\n",
+      "✅ Added 46 tags directly under characters.brave_girl_ravens\n",
+      "✅ Added 144 tags directly under characters.cardcaptor_sakura\n",
+      "✅ Added 128 tags directly under characters.danganronpa\n",
+      "✅ Added 240 tags directly under characters.digimon.digimon_characters\n",
+      "✅ Added 613 tags directly under characters.dragon_ball\n",
+      "✅ Added 331 tags directly under characters.dragon_quest\n",
+      "✅ Added 814 tags directly under characters.fate_series\n",
+      "✅ Added 566 tags directly under characters.final_fantasy\n",
+      "✅ Added 727 tags directly under characters.fire_emblem\n",
+      "✅ Added 441 tags directly under characters.flower_knight_girl\n",
+      "✅ Added 221 tags directly under characters.genderswap\n",
+      "✅ Added 431 tags directly under characters.gensou_suikoden\n",
+      "❌ No data found for girls_frontline_characters in any format\n",
+      "✅ Added 0 tags directly under characters.girls_frontline\n",
+      "✅ Added 305 tags directly under characters.girls_und_panzer\n",
+      "✅ Added 267 tags directly under characters.gundam_mechas\n",
+      "✅ Added 313 tags directly under characters.hunter_x_hunter\n",
+      "✅ Added 348 tags directly under characters.jojo_no_kimyou_na_bouken\n",
+      "✅ Added 308 tags directly under characters.kamen_rider\n",
+      "✅ Added 465 tags directly under characters.kantai_collection\n",
+      "✅ Added 64 tags directly under characters.kingdom_hearts\n",
+      "✅ Added 72 tags directly under characters.mahou_sensei_negima\n",
+      "✅ Added 158 tags directly under characters.meitantei_conan\n",
+      "✅ Added 153 tags directly under characters.minecraft\n",
+      "✅ Added 198 tags directly under characters.naruto\n",
+      "✅ Added 336 tags directly under characters.nippon_ichi\n",
+      "✅ Added 375 tags directly under characters.official_mascots\n",
+      "✅ Added 464 tags directly under characters.one_piece\n",
+      "✅ Added 305 tags directly under characters.oshiro_project\n",
+      "✅ Added 72 tags directly under characters.pokemon.elite_four_members\n",
+      "✅ Added 165 tags directly under characters.pokemon.families_of_pokemon_main_characters\n",
+      "✅ Added 100 tags directly under characters.pokemon.gym_leaders\n",
+      "✅ Added 37 tags directly under characters.pokemon.pokemon_ranger_characters\n",
+      "✅ Added 149 tags directly under characters.pokemon.pokemon_trainer_classes\n",
+      "✅ Added 925 tags directly under characters.pretty_cure\n",
+      "✅ Added 1214 tags directly under characters.ragnarok_online\n",
+      "✅ Added 434 tags directly under characters.real_life_racehorses\n",
+      "✅ Added 23 tags directly under characters.rosenkreuzstilette\n",
+      "✅ Added 320 tags directly under characters.sailor_moon\n",
+      "✅ Added 170 tags directly under characters.street_fighter\n",
+      "❌ No data found for super_smash_bros_characters in any format\n",
+      "✅ Added 0 tags directly under characters.super_smash_bros\n",
+      "❌ No data found for tales_of_characters in any format\n",
+      "✅ Added 0 tags directly under characters.tales_of\n",
+      "✅ Added 189 tags directly under characters.toaru_majutsu_no_index\n",
+      "✅ Added 381 tags directly under characters.touhou\n",
+      "✅ Added 128 tags directly under characters.touken_ranbu\n",
+      "✅ Added 540 tags directly under characters.ultra_series\n",
+      "✅ Added 250 tags directly under characters.umamusume\n",
+      "✅ Added 140 tags directly under characters.vocaloid\n",
+      "✅ Added 184 tags directly under characters.world_witches_series\n",
+      "❌ No data found for yu_gi_oh_characters in any format\n",
+      "✅ Added 0 tags directly under characters.yu_gi_oh\n",
+      "❌ No data found for copyrights_artists_projects_and_media in any format\n",
+      "❌ No data found for genres_of_video_games in any format\n",
+      "✅ Added 175 tags directly under copyrights_artists_projects_and_media.genres_of_video_games.fighting_games\n",
+      "✅ Added 42 tags directly under copyrights_artists_projects_and_media.genres_of_video_games.platform_games\n",
+      "✅ Added 515 tags directly under copyrights_artists_projects_and_media.genres_of_video_games.role-playing_games\n",
+      "✅ Added 114 tags directly under copyrights_artists_projects_and_media.genres_of_video_games.shooter_games\n",
+      "✅ Added 185 tags directly under copyrights_artists_projects_and_media.genres_of_video_games.visual_novel_games\n",
+      "❌ No data found for creatures in any format\n",
+      "✅ Added 232 tags directly under creatures.animals.birds\n",
+      "✅ Added 224 tags directly under creatures.animals.cats\n",
+      "✅ Added 237 tags directly under creatures.animals.dogs\n",
+      "✅ Added 285 tags directly under creatures.legendary_creatures\n",
+      "✅ Added 55 tags directly under drawing_software\n",
+      "❌ No data found for games in any format\n",
+      "✅ Added 21 tags directly under games.board_games\n",
+      "✅ Added 175 tags directly under games.fighting_games\n",
+      "✅ Added 61 tags directly under games.game_activities\n",
+      "✅ Added 420 tags directly under games.sports\n",
+      "✅ Added 144 tags directly under games.video_game\n",
+      "✅ Added 213 tags directly under metatags\n",
+      "❌ No data found for more in any format\n",
+      "✅ Added 110 tags directly under more.dances\n",
+      "✅ Added 29 tags directly under more.family_relationships\n",
+      "✅ Added 68 tags directly under more.fire\n",
+      "✅ Added 1051 tags directly under more.food_tags\n",
+      "✅ Added 31 tags directly under more.groups\n",
+      "✅ Added 76 tags directly under more.phrases\n",
+      "✅ Added 35 tags directly under more.scan\n",
+      "✅ Added 45 tags directly under more.subjective\n",
+      "✅ Added 237 tags directly under more.technology\n",
+      "✅ Added 446 tags directly under more.verbs_and_gerunds\n",
+      "✅ Added 54 tags directly under more.water\n",
+      "❌ No data found for objects in any format\n",
+      "✅ Added 410 tags directly under objects.airplanes\n",
+      "✅ Added 145 tags directly under objects.armor\n",
+      "✅ Added 377 tags directly under objects.audio_tags\n",
+      "✅ Added 66 tags directly under objects.cards.playing_card_faces\n",
+      "✅ Added 80 tags directly under objects.computer\n",
+      "✅ Added 94 tags directly under objects.eyewear\n",
+      "✅ Added 407 tags directly under objects.ground_vehicles\n",
+      "✅ Added 25 tags directly under objects.helicopters\n",
+      "✅ Added 48 tags directly under objects.piercings\n",
+      "✅ Added 94 tags directly under objects.pokemon_objects\n",
+      "✅ Added 105 tags directly under objects.sex_objects\n",
+      "✅ Added 256 tags directly under objects.ships\n",
+      "✅ Added 917 tags directly under objects.weapons\n",
+      "✅ Added 21 tags directly under plant.flowers\n",
+      "✅ Added 59 tags directly under plant.plant\n",
+      "✅ Added 44 tags directly under plant.tree\n",
+      "✅ Added 59 tags directly under plants\n",
+      "❌ No data found for real_world in any format\n",
+      "✅ Added 363 tags directly under real_world.companies_and_brand_names\n",
+      "✅ Added 138 tags directly under real_world.holidays_and_celebrations\n",
+      "✅ Added 75 tags directly under real_world.jobs\n",
+      "✅ Added 263 tags directly under real_world.locations\n",
+      "✅ Added 1526 tags directly under real_world.people\n",
+      "✅ Added 463 tags directly under real_world.real_world_locations\n",
+      "❌ No data found for sex in any format\n",
+      "✅ Added 16 tags directly under sex.sex_acts.simulated_sex_acts\n",
+      "✅ Added 56 tags directly under sex.sexual_positions\n",
+      "❌ No data found for subject in any format\n",
+      "❌ No data found for anthro in any format\n",
+      "❌ No data found for cat_boy in any format\n",
+      "✅ Added 0 tags directly under subject.anthro.cat_boy\n",
+      "❌ No data found for cat_girl in any format\n",
+      "✅ Added 0 tags directly under subject.anthro.cat_girl\n",
+      "❌ No data found for dog_girl in any format\n",
+      "✅ Added 0 tags directly under subject.anthro.dog_girl\n",
+      "❌ No data found for fox_girl in any format\n",
+      "✅ Added 0 tags directly under subject.anthro.fox_girl\n",
+      "❌ No data found for furry in any format\n",
+      "✅ Added 0 tags directly under subject.anthro.furry\n",
+      "❌ No data found for female in any format\n",
+      "❌ No data found for 1girl in any format\n",
+      "✅ Added 0 tags directly under subject.female.1girl\n",
+      "❌ No data found for 2girls in any format\n",
+      "✅ Added 0 tags directly under subject.female.2girls\n",
+      "❌ No data found for female_general in any format\n",
+      "✅ Added 0 tags directly under subject.female_general\n",
+      "❌ No data found for koma in any format\n",
+      "❌ No data found for 1koma in any format\n",
+      "✅ Added 0 tags directly under subject.koma.1koma\n",
+      "❌ No data found for 2koma in any format\n",
+      "✅ Added 0 tags directly under subject.koma.2koma\n",
+      "❌ No data found for male in any format\n",
+      "❌ No data found for 1boy in any format\n",
+      "✅ Added 0 tags directly under subject.male.1boy\n",
+      "❌ No data found for 2boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.2boys\n",
+      "❌ No data found for 3boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.3boys\n",
+      "❌ No data found for 4boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.4boys\n",
+      "❌ No data found for 5boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.5boys\n",
+      "❌ No data found for 6+boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.6+boys\n",
+      "❌ No data found for boy in any format\n",
+      "✅ Added 0 tags directly under subject.male.boy\n",
+      "❌ No data found for man in any format\n",
+      "✅ Added 0 tags directly under subject.male.man\n",
+      "❌ No data found for multiple_boys in any format\n",
+      "✅ Added 0 tags directly under subject.male.multiple_boys\n",
+      "❌ No data found for visual_characteristics in any format\n",
+      "❌ No data found for image_composition_and_style in any format\n",
+      "✅ Added 73 tags directly under visual_characteristics.image_composition_and_style.artistic_license\n",
+      "✅ Added 115 tags directly under visual_characteristics.image_composition_and_style.image_composition.backgrounds\n",
+      "✅ Added 90 tags directly under visual_characteristics.image_composition_and_style.image_composition.censorship\n",
+      "✅ Added 54 tags directly under visual_characteristics.image_composition_and_style.image_composition.colors\n",
+      "✅ Added 28 tags directly under visual_characteristics.image_composition_and_style.image_composition.focus_tags\n",
+      "✅ Added 55 tags directly under visual_characteristics.image_composition_and_style.image_composition.lighting\n",
+      "✅ Added 76 tags directly under visual_characteristics.image_composition_and_style.image_composition.prints\n",
+      "✅ Added 495 tags directly under visual_characteristics.image_composition_and_style.image_composition.style_parodies\n",
+      "✅ Added 41 tags directly under visual_characteristics.image_composition_and_style.patterns\n",
+      "✅ Added 310 tags directly under visual_characteristics.image_composition_and_style.symbols\n",
+      "✅ Added 242 tags directly under visual_characteristics.image_composition_and_style.text\n",
+      "✅ Added 62 tags directly under visual_characteristics.image_composition_and_style.year_tags\n",
+      "✅ Finished building hierarchy.\n"
+     ]
+    },
+    {
+     "ename": "FileNotFoundError",
+     "evalue": "[Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/misc/danbooru_donmai/danbooru_tags_step_01.json'",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[13], line 198\u001b[0m\n\u001b[1;32m    196\u001b[0m \u001b[38;5;66;03m# Save cleaned JSON\u001b[39;00m\n\u001b[1;32m    197\u001b[0m output_file \u001b[38;5;241m=\u001b[39m current_dir\u001b[38;5;241m.\u001b[39mparent \u001b[38;5;241m/\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmisc/danbooru_donmai/danbooru_tags_step_01.json\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m--> 198\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43moutput_file\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mw\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mencoding\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mutf-8\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m    199\u001b[0m     json\u001b[38;5;241m.\u001b[39mdump(manual_hierarchy, f, indent\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m4\u001b[39m, ensure_ascii\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m)\n\u001b[1;32m    201\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m✅ Hierarchy saved to `\u001b[39m\u001b[38;5;132;01m{\u001b[39;00moutput_file\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m`\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
+      "File \u001b[0;32m~/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    322\u001b[0m     )\n\u001b[0;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
+      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/home/lauhp/000_PHD/000_010_PUBLICATION/2025_SAGE/CODE/pm-paper_uzh_gitlab/pm-paper/misc/danbooru_donmai/danbooru_tags_step_01.json'"
+     ]
+    }
+   ],
+   "source": [
+    "import requests\n",
+    "import json\n",
+    "import time\n",
+    "import re\n",
+    "from urllib.parse import quote\n",
+    "\n",
+    "# Base API URL\n",
+    "BASE_URL = \"https://danbooru.donmai.us\"\n",
+    "\n",
+    "# Authentication (Replace with your credentials)\n",
+    "AUTH = (username, api_key)\n",
+    "\n",
+    "# Storage for missing tags\n",
+    "missing_tags = []\n",
+    "\n",
+    "# ✅ Initialize manual_hierarchy\n",
+    "manual_hierarchy = {}\n",
+    "\n",
+    "def clean_hierarchy_and_count_tags(hierarchy):\n",
+    "    \"\"\"\n",
+    "    Recursively removes empty categories from the JSON hierarchy and counts total tags.\n",
+    "    \"\"\"\n",
+    "    total_tag_count = 0\n",
+    "\n",
+    "    def clean_dict(d):\n",
+    "        nonlocal total_tag_count\n",
+    "        keys_to_delete = []\n",
+    "\n",
+    "        for key, value in d.items():\n",
+    "            if isinstance(value, dict):\n",
+    "                clean_dict(value)  # Recursively clean subcategories\n",
+    "                \n",
+    "                # ✅ If the subcategory is empty after cleaning, mark it for removal\n",
+    "                if not value:\n",
+    "                    keys_to_delete.append(key)\n",
+    "            elif isinstance(value, list):\n",
+    "                # ✅ Count the total number of tags\n",
+    "                total_tag_count += len(value)\n",
+    "\n",
+    "                # ✅ If the list is empty, mark it for deletion\n",
+    "                if not value:\n",
+    "                    keys_to_delete.append(key)\n",
+    "\n",
+    "        # ✅ Remove empty keys\n",
+    "        for key in keys_to_delete:\n",
+    "            del d[key]\n",
+    "\n",
+    "    clean_dict(hierarchy)\n",
+    "\n",
+    "    return hierarchy, total_tag_count\n",
+    "\n",
+    "\n",
+    "def fetch_wiki_page(tag, is_list=False):\n",
+    "    \"\"\"\n",
+    "    Fetches the wiki page of a tag group, list category, or special wiki page using the API.\n",
+    "    Uses list_of_ for lists, tag_group: for regular groups, \n",
+    "    and direct queries for special cases.\n",
+    "    \"\"\"\n",
+    "    if tag in special_wiki_pages:\n",
+    "        prefixes = [\"\"]  # Query directly with no prefix\n",
+    "        #print(f\"🔍 {tag} is a special wiki page, querying directly...\")\n",
+    "    elif is_list:\n",
+    "        prefixes = [\"list_of_\", \"tag_group:\"]\n",
+    "    else:\n",
+    "        prefixes = [\"tag_group:\"]\n",
+    "\n",
+    "  #  print(f\"🚀 Fetching {prefixes} {tag}\")  # Debugging print\n",
+    "\n",
+    "    for prefix in prefixes:\n",
+    "        query_tag = f\"{prefix}{tag}\".strip()  # Avoid unnecessary :\n",
+    "        encoded_query = quote(query_tag, safe=\"\")  # Proper URL encoding\n",
+    "        url = f\"{BASE_URL}/wiki_pages.json?search[title]={encoded_query}\"\n",
+    "\n",
+    "      #  print(f\"🚀 Fetching: {query_tag}\")\n",
+    "\n",
+    "        try:\n",
+    "            response = requests.get(url, auth=AUTH)  # Use authentication\n",
+    "            response.raise_for_status()  # Raise error for bad responses (4xx, 5xx)\n",
+    "            wiki_data = response.json()\n",
+    "\n",
+    "            if wiki_data and isinstance(wiki_data, list) and len(wiki_data) > 0:\n",
+    "                #print(f\"✅ Data found using {query_tag}\")\n",
+    "                return wiki_data[0].get(\"body\", \"\")  # Extract the \"body\" text\n",
+    "\n",
+    "        except requests.exceptions.HTTPError as e:\n",
+    "            if response.status_code == 401:\n",
+    "                print(f\"❌ Authentication Error (401 Unauthorized) for {query_tag}. Check your API key!\")\n",
+    "                exit()  # Stop execution if authentication fails\n",
+    "            else:\n",
+    "                print(f\"❌ Error fetching {query_tag}: {e}\")\n",
+    "\n",
+    "    # If all attempts fail\n",
+    "    print(f\"❌ No data found for {tag} in any format\")\n",
+    "    missing_tags.append(tag)\n",
+    "    return None\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "def build_tag_hierarchy(tag_groups):\n",
+    "    \"\"\"\n",
+    "    Builds and structures the hierarchy properly, ensuring:\n",
+    "    - Categories with subcategories store their direct tags in `_general`\n",
+    "    - Parent categories exist before adding children\n",
+    "    - Ensures subject and other key groups are correctly retained\n",
+    "    \"\"\"\n",
+    "    processed_groups = set()\n",
+    "\n",
+    "    for hierarchy_path, tag_group in sorted(tag_groups.items(), key=lambda x: x[0]):\n",
+    "        is_list = tag_group in list_based_categories  # Check if it's a list-based category\n",
+    "\n",
+    "        levels = hierarchy_path.split(\".\")\n",
+    "        current_level = manual_hierarchy\n",
+    "\n",
+    "        for key in levels[:-1]:  # Ensure each parent level exists\n",
+    "            if key not in current_level or not isinstance(current_level[key], dict):\n",
+    "                current_level[key] = {}  # Create dictionary if missing\n",
+    "            current_level = current_level[key]\n",
+    "\n",
+    "        last_level = levels[-1]\n",
+    "        has_subcategories = any(k.startswith(hierarchy_path + \".\") for k in tag_groups.keys())\n",
+    "\n",
+    "        # ✅ Ensure the category itself exists\n",
+    "        if last_level not in current_level:\n",
+    "            current_level[last_level] = {} if has_subcategories else []\n",
+    "\n",
+    "        # ✅ Fetch tags from API\n",
+    "        wiki_text = fetch_wiki_page(tag_group, is_list)\n",
+    "        extracted_tags = extract_tags_from_wiki(wiki_text, is_list)\n",
+    "\n",
+    "        # ✅ Store tags under `<category>_general` if there are subcategories\n",
+    "        if has_subcategories:\n",
+    "            general_key = f\"{last_level}_general\"\n",
+    "\n",
+    "            if isinstance(current_level[last_level], list):\n",
+    "               # print(f\"⚠️ Warning: {last_level} was a list but has subcategories. Converting to dictionary.\")\n",
+    "                current_level[last_level] = {}\n",
+    "\n",
+    "            if general_key not in current_level[last_level]:\n",
+    "                current_level[last_level][general_key] = []  # Initialize `_general` list\n",
+    "\n",
+    "            current_level[last_level][general_key].extend(extracted_tags)\n",
+    "            #print(f\"✅ Added {len(extracted_tags)} tags under {hierarchy_path} → {general_key}\")\n",
+    "\n",
+    "        else:\n",
+    "            # ✅ If no subcategories exist, store tags directly\n",
+    "            if isinstance(current_level[last_level], list):\n",
+    "                current_level[last_level].extend(extracted_tags)\n",
+    "            else:\n",
+    "               # print(f\"⚠️ Warning: {last_level} was a dictionary but has no subcategories. Converting to list.\")\n",
+    "                current_level[last_level] = extracted_tags\n",
+    "\n",
+    "            print(f\"✅ Added {len(extracted_tags)} tags directly under {hierarchy_path}\")\n",
+    "\n",
+    "    print(f\"✅ Finished building hierarchy.\")\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "def extract_tags_from_wiki(wiki_text, is_list=False):\n",
+    "    \"\"\"\n",
+    "    Extracts valid tags from the wiki text.\n",
+    "    Uses different extraction logic for tag groups and list-based pages.\n",
+    "    \"\"\"\n",
+    "    if not wiki_text:\n",
+    "        return []\n",
+    "\n",
+    "    if is_list:\n",
+    "        # Extract `[[tag_name]]` from list pages\n",
+    "        tag_pattern = re.compile(r\"\\[\\[(.*?)\\]\\]\")\n",
+    "        tags = tag_pattern.findall(wiki_text)\n",
+    "    else:\n",
+    "        # Extract `[[tag_name]]` from tag groups (skip first item if \"Tag Groups\")\n",
+    "        tag_pattern = re.compile(r\"\\[\\[(.*?)\\]\\]\")\n",
+    "        tags = tag_pattern.findall(wiki_text)\n",
+    "        tags = tags[1:] if tags else []  # Skip first tag\n",
+    "\n",
+    "    # Clean tags by removing alternative names `[[Tag|Alternative]]`\n",
+    "    cleaned_tags = [tag.split(\"|\")[0].strip() for tag in tags]\n",
+    "\n",
+    "    return cleaned_tags\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    print(\"🚀 Starting Danbooru Tag Hierarchy API Fetcher...\")\n",
+    "\n",
+    "    # ✅ Ensure manual_hierarchy exists before cleaning\n",
+    "    manual_hierarchy = {}\n",
+    "\n",
+    "    # Build hierarchy dynamically\n",
+    "    build_tag_hierarchy(tag_groups)\n",
+    "\n",
+    "    # ✅ Clean hierarchy and count total tags\n",
+    "    manual_hierarchy, total_tags = clean_hierarchy_and_count_tags(manual_hierarchy)\n",
+    "\n",
+    "    # Save cleaned JSON\n",
+    "    output_file = current_dir.parent / \"misc/danbooru_donmai/danbooru_tags_step_01.json\"\n",
+    "    with open(output_file, \"w\", encoding=\"utf-8\") as f:\n",
+    "        json.dump(manual_hierarchy, f, indent=4, ensure_ascii=False)\n",
+    "\n",
+    "    print(f\"\\n✅ Hierarchy saved to `{output_file}`\")\n",
+    "    print(f\"📊 Total tags in the final hierarchy: {total_tags}\")\n",
+    "\n",
+    "    if missing_tags:\n",
+    "        print(f\"⚠️ The following tag groups were not found: {missing_tags}\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5bf6fa19-04dd-4799-8220-e37a4434a1f8",
+   "metadata": {},
+   "source": [
+    "### Add subject keys etc"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "9bf7c0ed-0cfd-41c8-89a4-c59f1783c823",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:49:06.895515Z",
+     "iopub.status.busy": "2025-03-17T12:49:06.895119Z",
+     "iopub.status.idle": "2025-03-17T12:49:07.135844Z",
+     "shell.execute_reply": "2025-03-17T12:49:07.135358Z",
+     "shell.execute_reply.started": "2025-03-17T12:49:06.895492Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Adding woman to subject.female.female_general\n",
+      "✅ Adding girl to subject.female.female_general\n",
+      "✅ Adding 1girl to subject.female.female_general\n",
+      "🔄 Moving 2girls from more.groups to subject.female.female_general\n",
+      "✅ Adding 2girls to subject.female.female_general\n",
+      "🔄 Moving 3girls from more.groups to subject.female.female_general\n",
+      "✅ Adding 3girls to subject.female.female_general\n",
+      "🔄 Moving 4girls from more.groups to subject.female.female_general\n",
+      "✅ Adding 4girls to subject.female.female_general\n",
+      "🔄 Moving 5girls from more.groups to subject.female.female_general\n",
+      "✅ Adding 5girls to subject.female.female_general\n",
+      "🔄 Moving 6+girls from more.groups to subject.female.female_general\n",
+      "✅ Adding 6+girls to subject.female.female_general\n",
+      "🔄 Moving multiple girls from more.groups to subject.female.female_general\n",
+      "✅ Adding multiple girls to subject.female.female_general\n",
+      "🔄 Moving guitar girl from objects.audio_tags to subject.female.female_general\n",
+      "✅ Adding guitar girl to subject.female.female_general\n",
+      "✅ Adding man to subject.male.male_general\n",
+      "✅ Adding boy to subject.male.male_general\n",
+      "✅ Adding 1boy to subject.male.male_general\n",
+      "🔄 Moving 2boys from more.groups to subject.male.male_general\n",
+      "✅ Adding 2boys to subject.male.male_general\n",
+      "🔄 Moving 3boys from more.groups to subject.male.male_general\n",
+      "✅ Adding 3boys to subject.male.male_general\n",
+      "🔄 Moving 4boys from more.groups to subject.male.male_general\n",
+      "✅ Adding 4boys to subject.male.male_general\n",
+      "🔄 Moving 5boys from more.groups to subject.male.male_general\n",
+      "✅ Adding 5boys to subject.male.male_general\n",
+      "🔄 Moving 6+boys from more.groups to subject.male.male_general\n",
+      "✅ Adding 6+boys to subject.male.male_general\n",
+      "🔄 Moving multiple boys from more.groups to subject.male.male_general\n",
+      "✅ Adding multiple boys to subject.male.male_general\n",
+      "✅ Adding guitar boy to subject.male.male_general\n",
+      "✅ Adding 1koma to subject.koma.koma_general\n",
+      "✅ Adding 2koma to subject.koma.koma_general\n",
+      "🔄 Moving cat girl from creatures.animals.cats to subject.anthro.anthro_general\n",
+      "✅ Adding cat girl to subject.anthro.anthro_general\n",
+      "✅ Adding fox girl to subject.anthro.anthro_general\n",
+      "✅ Adding dog girl to subject.anthro.anthro_general\n",
+      "🔄 Moving plant girl from plant.plant to subject.anthro.anthro_general\n",
+      "🔄 Moving plant girl from plants to subject.anthro.anthro_general\n",
+      "✅ Adding plant girl to subject.anthro.anthro_general\n",
+      "🔄 Moving plant boy from plant.plant to subject.anthro.anthro_general\n",
+      "🔄 Moving plant boy from plants to subject.anthro.anthro_general\n",
+      "✅ Adding plant boy to subject.anthro.anthro_general\n",
+      "🔄 Moving cat boy from creatures.animals.cats to subject.anthro.anthro_general\n",
+      "✅ Adding cat boy to subject.anthro.anthro_general\n",
+      "✅ Adding furry to subject.anthro.anthro_general\n",
+      "✅ Adding monster boy to subject.anthro.anthro_general\n",
+      "✅ Adding monster girl to subject.anthro.anthro_general\n",
+      "✅ Adding demon girl to subject.anthro.anthro_general\n",
+      "✅ Adding demon boy to subject.anthro.anthro_general\n",
+      "✅ Adding magical boy to subject.anthro.anthro_general\n",
+      "🔄 Moving magical girl from characters.sailor_moon to subject.anthro.anthro_general\n",
+      "✅ Adding magical girl to subject.anthro.anthro_general\n",
+      "✅ Updated JSON saved as `/shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/danbooru_tags_step_02.json`\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json\n",
+    "\n",
+    "# Load the existing JSON file\n",
+    "input_file =  current_dir.parent / \"misc/danbooru_donmai/danbooru_tags_step_01.json\"\n",
+    "output_file =  current_dir.parent / \"misc/danbooru_donmai/danbooru_tags_step_02.json\"\n",
+    "\n",
+    "with open(input_file, \"r\", encoding=\"utf-8\") as f:\n",
+    "    manual_hierarchy = json.load(f)\n",
+    "\n",
+    "# ✅ Define the correct subject structure\n",
+    "subject_structure = {\n",
+    "    \"subject\": {\n",
+    "        \"female\": {\n",
+    "            \"female_general\": [\"woman\", \"girl\", \"1girl\", \"2girls\", \"3girls\", \"4girls\", \"5girls\", \"6+girls\", \"multiple girls\", \"guitar girl\" ]\n",
+    "        },\n",
+    "        \"male\": {\n",
+    "            \"male_general\": [\"man\", \"boy\", \"1boy\", \"2boys\", \"3boys\", \"4boys\", \"5boys\", \"6+boys\", \"multiple boys\", \"guitar boy\"]\n",
+    "        },\n",
+    "        \"koma\": {\n",
+    "            \"koma_general\": [\"1koma\", \"2koma\"]\n",
+    "        },\n",
+    "        \"anthro\": {\n",
+    "            \"anthro_general\": [\"cat girl\", \"fox girl\", \"dog girl\", \"plant girl\", \"plant boy\", \"cat boy\", \"furry\", \"monster boy\", \"monster girl\", \"demon girl\" , \"demon boy\", \"magical boy\", \"magical girl\",  ]\n",
+    "        }\n",
+    "    }\n",
+    "}\n",
+    "\n",
+    "# ✅ Ensure \"subject\" exists in the hierarchy\n",
+    "if \"subject\" not in manual_hierarchy:\n",
+    "    manual_hierarchy[\"subject\"] = {}\n",
+    "\n",
+    "# ✅ Ensure all subcategories exist\n",
+    "for category, subcategories in subject_structure[\"subject\"].items():\n",
+    "    if category not in manual_hierarchy[\"subject\"]:\n",
+    "        manual_hierarchy[\"subject\"][category] = {}\n",
+    "\n",
+    "    for subcategory, tags in subcategories.items():\n",
+    "        if subcategory not in manual_hierarchy[\"subject\"][category]:\n",
+    "            manual_hierarchy[\"subject\"][category][subcategory] = []\n",
+    "\n",
+    "# ✅ Move misplaced tags and also ensure missing tags are added\n",
+    "for category, subcategories in subject_structure[\"subject\"].items():\n",
+    "    for subcategory, tags in subcategories.items():\n",
+    "        target_list = manual_hierarchy[\"subject\"][category][subcategory]\n",
+    "\n",
+    "        for tag in tags:\n",
+    "            found_in_wrong_place = False  # Track if tag was found elsewhere\n",
+    "\n",
+    "            # ✅ Search for the tag in other categories and remove if found\n",
+    "            for key, value in list(manual_hierarchy.items()):  # Use list() to avoid runtime changes\n",
+    "                if isinstance(value, list) and tag in value:\n",
+    "                    print(f\"🔄 Moving {tag} from {key} to subject.{category}.{subcategory}\")\n",
+    "                    value.remove(tag)\n",
+    "                    found_in_wrong_place = True\n",
+    "\n",
+    "                elif isinstance(value, dict):  # Search deeper\n",
+    "                    for subkey, subvalue in list(value.items()):\n",
+    "                        if isinstance(subvalue, list) and tag in subvalue:\n",
+    "                            print(f\"🔄 Moving {tag} from {key}.{subkey} to subject.{category}.{subcategory}\")\n",
+    "                            subvalue.remove(tag)\n",
+    "                            found_in_wrong_place = True\n",
+    "\n",
+    "                        elif isinstance(subvalue, dict):\n",
+    "                            for deepkey, deepvalue in list(subvalue.items()):\n",
+    "                                if isinstance(deepvalue, list) and tag in deepvalue:\n",
+    "                                    print(f\"🔄 Moving {tag} from {key}.{subkey}.{deepkey} to subject.{category}.{subcategory}\")\n",
+    "                                    deepvalue.remove(tag)\n",
+    "                                    found_in_wrong_place = True\n",
+    "\n",
+    "            # ✅ Add the tag to the correct subject category if missing\n",
+    "            if tag not in target_list:\n",
+    "                print(f\"✅ Adding {tag} to subject.{category}.{subcategory}\")\n",
+    "                target_list.append(tag)\n",
+    "\n",
+    "# ✅ Save the updated JSON\n",
+    "with open(output_file, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(manual_hierarchy, f, indent=4, ensure_ascii=False)\n",
+    "\n",
+    "print(f\"✅ Updated JSON saved as `{output_file}`\")\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "aff1a30d-17e7-4e55-b881-b5ce622a533a",
+   "metadata": {},
+   "source": [
+    "### Compare with wd-14 vocabulary"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "2a8f986c-0ccd-4235-ac50-be3d6e495f9f",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T12:49:10.749271Z",
+     "iopub.status.busy": "2025-03-17T12:49:10.748481Z",
+     "iopub.status.idle": "2025-03-17T12:49:11.628872Z",
+     "shell.execute_reply": "2025-03-17T12:49:11.628200Z",
+     "shell.execute_reply.started": "2025-03-17T12:49:10.749248Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "📌 Tags in CSV but NOT in JSON:\n",
+      "\n",
+      "✅ Missing tags saved to `missing_tags.txt`\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json\n",
+    "import pandas as pd\n",
+    "\n",
+    "# Load CSV file\n",
+    "csv_file = current_dir.parent /  \"misc/autotagging-vocabularies/danbooru.csv\" # Update with the actual CSV file name\n",
+    "df = pd.read_csv(csv_file)\n",
+    "\n",
+    "# Load JSON file\n",
+    "json_file = input_file =  current_dir.parent / \"misc/danbooru_donmai/danbooru_tags_step_03.json\"\n",
+    "with open(json_file, \"r\", encoding=\"utf-8\") as f:\n",
+    "    manual_hierarchy = json.load(f)\n",
+    "\n",
+    "# ✅ Extract all tags from the CSV\n",
+    "csv_tags = set(df[\"name\"].astype(str).str.replace(\"_\", \" \").str.lower())  # Normalize tags\n",
+    "\n",
+    "# ✅ Extract all tags from the JSON recursively\n",
+    "def extract_tags_from_json(data):\n",
+    "    tags = set()\n",
+    "    if isinstance(data, dict):\n",
+    "        for value in data.values():\n",
+    "            tags.update(extract_tags_from_json(value))\n",
+    "    elif isinstance(data, list):\n",
+    "        tags.update(str(tag).replace(\"_\", \" \").lower() for tag in data)\n",
+    "    return tags\n",
+    "\n",
+    "json_tags = extract_tags_from_json(manual_hierarchy)\n",
+    "\n",
+    "# ✅ Find tags in CSV but NOT in JSON\n",
+    "missing_tags = csv_tags - json_tags\n",
+    "\n",
+    "# ✅ Print the missing tags\n",
+    "print(\"\\n📌 Tags in CSV but NOT in JSON:\")\n",
+    "for tag in sorted(missing_tags):\n",
+    "    print(tag)\n",
+    "\n",
+    "# ✅ Save missing tags to a file for review (optional)\n",
+    "missing_tags_file = \"missing_tags.txt\"\n",
+    "with open(missing_tags_file, \"w\", encoding=\"utf-8\") as f:\n",
+    "    f.write(\"\\n\".join(sorted(missing_tags)))\n",
+    "\n",
+    "print(f\"\\n✅ Missing tags saved to `{missing_tags_file}`\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "706317c1-498d-40ab-a29a-ee658e1735e2",
+   "metadata": {
+    "execution": {
+     "iopub.execute_input": "2025-03-17T10:53:52.106543Z",
+     "iopub.status.busy": "2025-03-17T10:53:52.105944Z",
+     "iopub.status.idle": "2025-03-17T10:53:52.108812Z",
+     "shell.execute_reply": "2025-03-17T10:53:52.108451Z",
+     "shell.execute_reply.started": "2025-03-17T10:53:52.106526Z"
+    }
+   },
+   "source": [
+    "### Fetch wikidata for collected tags"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ec696e5b-be39-4407-a04f-6ce6b0b08855",
+   "metadata": {
+    "execution": {
+     "execution_failed": "2025-03-17T12:46:52.921Z",
+     "iopub.execute_input": "2025-03-17T12:45:32.542088Z",
+     "iopub.status.busy": "2025-03-17T12:45:32.541554Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "✅ Extracted 35559 unique tags.\n",
+      "🚀 Fetching data for amamiya_elena (1/35559)...\n",
+      "✅ Saved amamiya_elena to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/amamiya_elena.json\n",
+      "🚀 Fetching data for gilles_de_rais (2/35559)...\n",
+      "✅ Saved gilles_de_rais to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/gilles_de_rais.json\n",
+      "🚀 Fetching data for blue_scrunchie (3/35559)...\n",
+      "✅ Saved blue_scrunchie to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/blue_scrunchie.json\n",
+      "🚀 Fetching data for playing (4/35559)...\n",
+      "✅ Saved playing to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/playing.json\n",
+      "🚀 Fetching data for album_cover_redraw (5/35559)...\n",
+      "✅ Saved album_cover_redraw to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/album_cover_redraw.json\n",
+      "🚀 Fetching data for >3< (6/35559)...\n",
+      "✅ Saved >3< to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/>3<.json\n",
+      "🚀 Fetching data for damom (7/35559)...\n",
+      "🚀 Fetching data for shoulder_cannon (8/35559)...\n",
+      "✅ Saved shoulder_cannon to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/shoulder_cannon.json\n",
+      "🚀 Fetching data for cover_image (9/35559)...\n",
+      "✅ Saved cover_image to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/cover_image.json\n",
+      "🚀 Fetching data for ultraman_legend (10/35559)...\n",
+      "✅ Saved ultraman_legend to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/ultraman_legend.json\n",
+      "🚀 Fetching data for mastiff (11/35559)...\n",
+      "✅ Saved mastiff to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/mastiff.json\n",
+      "🚀 Fetching data for oingo (12/35559)...\n",
+      "✅ Saved oingo to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/oingo.json\n",
+      "🚀 Fetching data for mario_strikers_(series) (13/35559)...\n",
+      "✅ Saved mario_strikers_(series) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/mario_strikers_(series).json\n",
+      "🚀 Fetching data for phallic_symbol (14/35559)...\n",
+      "✅ Saved phallic_symbol to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/phallic_symbol.json\n",
+      "🚀 Fetching data for 4girls (15/35559)...\n",
+      "✅ Saved 4girls to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/4girls.json\n",
+      "🚀 Fetching data for nice_nature_(racehorse) (16/35559)...\n",
+      "✅ Saved nice_nature_(racehorse) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/nice_nature_(racehorse).json\n",
+      "🚀 Fetching data for hair_beads (17/35559)...\n",
+      "✅ Saved hair_beads to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/hair_beads.json\n",
+      "🚀 Fetching data for fu_po_(azur_lane) (18/35559)...\n",
+      "✅ Saved fu_po_(azur_lane) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/fu_po_(azur_lane).json\n",
+      "🚀 Fetching data for togepi (19/35559)...\n",
+      "✅ Saved togepi to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/togepi.json\n",
+      "🚀 Fetching data for yasopp (20/35559)...\n",
+      "✅ Saved yasopp to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/yasopp.json\n",
+      "🚀 Fetching data for oyafune_suama (21/35559)...\n",
+      "✅ Saved oyafune_suama to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/oyafune_suama.json\n",
+      "🚀 Fetching data for phantasy_star_iii (22/35559)...\n",
+      "✅ Saved phantasy_star_iii to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/phantasy_star_iii.json\n",
+      "🚀 Fetching data for qingdai_guanmao (23/35559)...\n",
+      "✅ Saved qingdai_guanmao to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/qingdai_guanmao.json\n",
+      "🚀 Fetching data for kufei (24/35559)...\n",
+      "✅ Saved kufei to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/kufei.json\n",
+      "🚀 Fetching data for stefan_(atelier) (25/35559)...\n",
+      "✅ Saved stefan_(atelier) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/stefan_(atelier).json\n",
+      "🚀 Fetching data for dille_blood (26/35559)...\n",
+      "✅ Saved dille_blood to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/dille_blood.json\n",
+      "🚀 Fetching data for vivillon_(modern) (27/35559)...\n",
+      "✅ Saved vivillon_(modern) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/vivillon_(modern).json\n",
+      "🚀 Fetching data for sweetie_(ragnarok_online) (28/35559)...\n",
+      "✅ Saved sweetie_(ragnarok_online) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/sweetie_(ragnarok_online).json\n",
+      "🚀 Fetching data for whisking (29/35559)...\n",
+      "✅ Saved whisking to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/whisking.json\n",
+      "🚀 Fetching data for h&k_hk33 (30/35559)...\n",
+      "✅ Saved h&k_hk33 to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/h&k_hk33.json\n",
+      "🚀 Fetching data for winx_club (31/35559)...\n",
+      "✅ Saved winx_club to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/winx_club.json\n",
+      "🚀 Fetching data for anti-tank_grenade (32/35559)...\n",
+      "✅ Saved anti-tank_grenade to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/anti-tank_grenade.json\n",
+      "🚀 Fetching data for devin_booker (33/35559)...\n",
+      "✅ Saved devin_booker to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/devin_booker.json\n",
+      "🚀 Fetching data for scylla_(azur_lane) (34/35559)...\n",
+      "✅ Saved scylla_(azur_lane) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/scylla_(azur_lane).json\n",
+      "🚀 Fetching data for penance_(arknights) (35/35559)...\n",
+      "✅ Saved penance_(arknights) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/penance_(arknights).json\n",
+      "🚀 Fetching data for toba_(oshiro_project) (36/35559)...\n",
+      "✅ Saved toba_(oshiro_project) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/toba_(oshiro_project).json\n",
+      "🚀 Fetching data for scott_adams_(style) (37/35559)...\n",
+      "✅ Saved scott_adams_(style) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/scott_adams_(style).json\n",
+      "🚀 Fetching data for 502nd_joint_fighter_wing (38/35559)...\n",
+      "✅ Saved 502nd_joint_fighter_wing to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/502nd_joint_fighter_wing.json\n",
+      "🚀 Fetching data for komica_wiki (39/35559)...\n",
+      "✅ Saved komica_wiki to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/komica_wiki.json\n",
+      "🚀 Fetching data for final_fantasy_vi (40/35559)...\n",
+      "✅ Saved final_fantasy_vi to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/final_fantasy_vi.json\n",
+      "🚀 Fetching data for h&k_hk45 (41/35559)...\n",
+      "✅ Saved h&k_hk45 to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/h&k_hk45.json\n",
+      "🚀 Fetching data for saint_seiya (42/35559)...\n",
+      "✅ Saved saint_seiya to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/saint_seiya.json\n",
+      "🚀 Fetching data for ike_(fire_emblem) (43/35559)...\n",
+      "✅ Saved ike_(fire_emblem) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/ike_(fire_emblem).json\n",
+      "🚀 Fetching data for cooperative_breast_smother (44/35559)...\n",
+      "✅ Saved cooperative_breast_smother to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/cooperative_breast_smother.json\n",
+      "🚀 Fetching data for pamiat_merkuria_(azur_lane) (45/35559)...\n",
+      "✅ Saved pamiat_merkuria_(azur_lane) to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/pamiat_merkuria_(azur_lane).json\n",
+      "🚀 Fetching data for ribbon-trimmed_skirt (46/35559)...\n",
+      "✅ Saved ribbon-trimmed_skirt to /shares/weddigen.ki.uzh/laura_wagner/Civitai_page_analysis/Civitai_visualizations/misc/danbooru_donmai/tags_wiki/ribbon-trimmed_skirt.json\n"
+     ]
+    }
+   ],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "import requests\n",
+    "import time\n",
+    "import urllib.parse\n",
+    "\n",
+    "# Load the cleaned hierarchy JSON\n",
+    "json_file_path = current_dir.parent / \"misc/danbooru_donmai/danbooru_tags_step_03.json\"\n",
+    "with open(json_file_path, \"r\", encoding=\"utf-8\") as file:\n",
+    "    tag_hierarchy = json.load(file)\n",
+    "\n",
+    "# Authentication (Replace with your credentials)\n",
+    "USERNAME = username\n",
+    "API_KEY = api_key\n",
+    "\n",
+    "# Base API URL\n",
+    "BASE_URL = \"https://danbooru.donmai.us\"\n",
+    "\n",
+    "# Output folder for JSON results\n",
+    "output_folder = current_dir.parent / \"misc/danbooru_donmai/tags_wiki\"\n",
+    "os.makedirs(output_folder, exist_ok=True)  # Ensure the directory exists\n",
+    "\n",
+    "# Function to extract all unique tags from nested dictionaries/lists\n",
+    "def extract_tags(data):\n",
+    "    tags = set()\n",
+    "    if isinstance(data, dict):\n",
+    "        for key, value in data.items():\n",
+    "            tags.update(extract_tags(value))  # Recursively extract tags\n",
+    "    elif isinstance(data, list):\n",
+    "        for item in data:\n",
+    "            if isinstance(item, str):\n",
+    "                formatted_tag = item.lower().replace(\" \", \"_\")  # ✅ Convert to lowercase and replace spaces\n",
+    "                tags.add(formatted_tag)\n",
+    "            else:\n",
+    "                tags.update(extract_tags(item))  # Handle nested lists\n",
+    "    return tags\n",
+    "\n",
+    "# Extract all unique tags\n",
+    "all_tags = extract_tags(tag_hierarchy)\n",
+    "print(f\"✅ Extracted {len(all_tags)} unique tags.\")\n",
+    "\n",
+    "# API query function\n",
+    "def fetch_tag_data(tag):\n",
+    "    \"\"\"Fetch tag details from Danbooru API.\"\"\"\n",
+    "    encoded_tag = urllib.parse.quote(tag, safe=\"\")\n",
+    "    api_url = f\"{BASE_URL}/tags.json?search[name]={encoded_tag}&only=id,name,category,post_count,is_deprecated,created_at,updated_at,wiki_page,artist,antecedent_alias,consequent_aliases,antecedent_implications,consequent_implications,dtext_links\"\n",
+    "\n",
+    "    try:\n",
+    "        response = requests.get(api_url, auth=(USERNAME, API_KEY))\n",
+    "        response.raise_for_status()\n",
+    "        return response.json()\n",
+    "    except requests.exceptions.RequestException as e:\n",
+    "        print(f\"⚠️ Error fetching data for '{tag}': {e}\")\n",
+    "        return None\n",
+    "\n",
+    "# Process each tag\n",
+    "for idx, tag in enumerate(all_tags):\n",
+    "    tag_filename = os.path.join(output_folder, f\"{tag}.json\")\n",
+    "\n",
+    "    # Skip if the file already exists to avoid redundant API calls\n",
+    "    if os.path.exists(tag_filename):\n",
+    "        print(f\"🔄 Skipping {tag}, already saved.\")\n",
+    "        continue\n",
+    "\n",
+    "    print(f\"🚀 Fetching data for {tag} ({idx+1}/{len(all_tags)})...\")\n",
+    "\n",
+    "    tag_data = fetch_tag_data(tag)\n",
+    "\n",
+    "    if tag_data:\n",
+    "        with open(tag_filename, \"w\", encoding=\"utf-8\") as f:\n",
+    "            json.dump(tag_data, f, indent=4, ensure_ascii=False)\n",
+    "        print(f\"✅ Saved {tag} to {tag_filename}\")\n",
+    "\n",
+    "    # Respect API rate limits\n",
+    "    time.sleep(1.5)  # Adjust delay if necessary\n",
+    "\n",
+    "print(\"\\n✅ All tags processed and saved in the 'tags/' folder.\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8e364060-36e9-4418-8bdd-6018c9edcc33",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "latm",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

jupyter_notebooks/SuppM_Figure_S14_co-occurence_training_data.ipynb

@@ -0,0 +1,152 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Prepare *.json for Figure 13"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import json\n",
+    "from itertools import combinations\n",
+    "from collections import Counter\n",
+    "\n",
+    "# Load the data with only needed columns\n",
+    "csv_file_path = \"all_models_with_tags.csv\"  # Update with your actual file path\n",
+    "\n",
+    "df = pd.read_csv(csv_file_path, low_memory=False)\n",
+    "\n",
+    "# Identify the starting point of tag columns (after 'civitai_id')\n",
+    "civitai_index = df.columns.get_loc(\"civitai_id\") + 1\n",
+    "\n",
+    "df_tags = df.iloc[:, civitai_index:]\n",
+    "\n",
+    "# Extract tag columns (tag01 to tag199) and corresponding counts\n",
+    "tag_columns = [col for col in df_tags.columns if col.startswith(\"tag\") and col[3:].isdigit() and int(col[3:]) <= 199]\n",
+    "tag_no_columns = [col for col in df_tags.columns if col.startswith(\"tag\") and col.endswith(\"_no\")]\n",
+    "\n",
+    "if not tag_columns:\n",
+    "    print(\"No tag columns found in the dataset.\")\n",
+    "    exit()\n",
+    "\n",
+    "df_tags = df[tag_columns]\n",
+    "df_tag_counts = df[tag_no_columns] if tag_no_columns else None\n",
+    "\n",
+    "# Load tag categories from JSON\n",
+    "json_file_path = \"danbooru_tags_step_03.json\"  # Update with your actual file path\n",
+    "with open(json_file_path, \"r\", encoding=\"utf-8\") as f:\n",
+    "    tag_categories = json.load(f)\n",
+    "\n",
+    "# Function to normalize tags (lowercase and replace underscores)\n",
+    "def normalize_tag(tag):\n",
+    "    return tag.lower().replace(\"_\", \" \") if isinstance(tag, str) else tag\n",
+    "\n",
+    "# Flatten JSON structure to map tags to top-level categories\n",
+    "def extract_tag_categories(json_data, current_category=None):\n",
+    "    tag_mapping = {}\n",
+    "    for key, value in json_data.items():\n",
+    "        if isinstance(value, dict):\n",
+    "            tag_mapping.update(extract_tag_categories(value, key))\n",
+    "        elif isinstance(value, list):\n",
+    "            for tag in value:\n",
+    "                normalized_tag = normalize_tag(tag)\n",
+    "                if normalized_tag:\n",
+    "                    tag_mapping[normalized_tag] = current_category\n",
+    "    return tag_mapping\n",
+    "\n",
+    "# Create mapping of tags to categories\n",
+    "tag_category_mapping = extract_tag_categories(tag_categories)\n",
+    "\n",
+    "# Flatten and count occurrences of individual tags\n",
+    "all_tags = []\n",
+    "tag_counts = Counter()\n",
+    "\n",
+    "for i, row in df_tags.iterrows():\n",
+    "    tags = [normalize_tag(tag) for tag in row if pd.notna(tag)]\n",
+    "    if df_tag_counts is not None:\n",
+    "        counts = df_tag_counts.iloc[i].fillna(1).tolist()\n",
+    "    else:\n",
+    "        counts = [1] * len(tags)\n",
+    "    for tag, count in zip(tags, counts):\n",
+    "        all_tags.append(tag)\n",
+    "        tag_counts[tag] += count\n",
+    "\n",
+    "if not all_tags:\n",
+    "    print(\"No valid tags found in the dataset.\")\n",
+    "    exit()\n",
+    "print(f\"Extracted {len(all_tags)} total tags.\")\n",
+    "\n",
+    "# Compute co-occurrence frequencies efficiently\n",
+    "co_occurrences = Counter()\n",
+    "for i, row in df_tags.iterrows():\n",
+    "    tags = [normalize_tag(tag) for tag in row if pd.notna(tag)]\n",
+    "    if len(tags) < 2:\n",
+    "        continue  # Skip rows with fewer than two tags\n",
+    "    for tag1, tag2 in combinations(tags, 2):\n",
+    "        co_occurrences[frozenset([tag1, tag2])] += 1\n",
+    "\n",
+    "if not co_occurrences:\n",
+    "    print(\"No co-occurrence data found.\")\n",
+    "    exit()\n",
+    "print(f\"Computed {len(co_occurrences)} co-occurrence pairs.\")\n",
+    "\n",
+    "# Convert co-occurrence counts to a weighted edge list\n",
+    "edges = [(tuple(pair)[0], tuple(pair)[1], weight) for pair, weight in co_occurrences.items() if len(pair) == 2]\n",
+    "\n",
+    "# Create a set of connected tags with a minimum connection threshold\n",
+    "min_connections = 5  # Increased threshold to reduce memory usage\n",
+    "connected_tags = Counter()\n",
+    "for tag1, tag2, _ in edges:\n",
+    "    connected_tags[tag1] += 1\n",
+    "    connected_tags[tag2] += 1\n",
+    "\n",
+    "# Filter nodes to keep only those that meet the minimum connection threshold\n",
+    "nodes = [\n",
+    "    {\"id\": tag, \"size\": count, \"category\": tag_category_mapping.get(tag, \"unknown\")}\n",
+    "    for tag, count in tag_counts.items()\n",
+    "    if connected_tags[tag] >= min_connections\n",
+    "]\n",
+    "\n",
+    "if not nodes:\n",
+    "    print(\"No nodes meet the connection threshold.\")\n",
+    "    exit()\n",
+    "print(f\"Final node count: {len(nodes)}\")\n",
+    "\n",
+    "links = [\n",
+    "    {\"source\": tag1, \"target\": tag2, \"value\": weight}\n",
+    "    for tag1, tag2, weight in edges\n",
+    "    if connected_tags[tag1] >= min_connections and connected_tags[tag2] >= min_connections\n",
+    "]\n",
+    "\n",
+    "if not links:\n",
+    "    print(\"No links meet the connection threshold.\")\n",
+    "    exit()\n",
+    "print(f\"Final link count: {len(links)}\")\n",
+    "\n",
+    "# Prepare JSON output\n",
+    "d3_data = {\"nodes\": nodes, \"links\": links}\n",
+    "\n",
+    "# Save as JSON for visualization\n",
+    "output_file = \"co_occurrence_network.json\"\n",
+    "with open(output_file, \"w\", encoding=\"utf-8\") as f:\n",
+    "    json.dump(d3_data, f, indent=4)\n",
+    "\n",
+    "print(f\"D3.js data saved to {output_file}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

md/DEEPFAKE_PIPELINE_GUIDE.md

@@ -0,0 +1,210 @@
+# Deepfake Adapter Dataset Processing - Quick Start Guide
+
+## Overview
+
+This pipeline processes the `real_person_adapters.csv` dataset to identify and annotate real people used in deepfake LoRA models using three LLM options: **Qwen**, **Llama**, and **Mistral**.
+
+## Quick Start
+
+### 1. Prerequisites
+
+```bash
+# Install required packages
+pip install pandas numpy emoji requests tqdm spacy
+
+# Download spaCy English model (for NER)
+python -m spacy download en_core_web_sm
+```
+
+**Note**: The spaCy model will be automatically downloaded when you run the notebook if not already installed.
+
+### 2. Set Up API Keys
+
+Choose at least ONE LLM provider and get an API key:
+
+| Provider | Model | Sign Up Link | Est. Cost (10k entries) |
+|----------|-------|--------------|-------------------------|
+| **Qwen** | Qwen-Max | https://dashscope.aliyun.com/ | Varies |
+| **Llama** | Llama-3.1-70B | https://www.together.ai/ | ~$5-10 |
+| **Mistral** | Mistral Large | https://mistral.ai/ | ~$40-80 |
+
+Create your API key file in `misc/credentials/`:
+
+```bash
+# For Qwen
+echo "your-api-key-here" > misc/credentials/qwen_api_key.txt
+
+# For Llama (via Together AI)
+echo "your-api-key-here" > misc/credentials/together_api_key.txt
+
+# For Mistral
+echo "your-api-key-here" > misc/credentials/mistral_api_key.txt
+```
+
+### 3. Run the Notebook
+
+Open `Section_2-3-4_Figure_8_deepfake_adapters.ipynb` and:
+
+1. **Run all cells sequentially** from top to bottom
+2. The default configuration uses Qwen in test mode (10 samples)
+3. Review the test results
+4. To process the full dataset, change in the LLM annotation cell:
+   ```python
+   TEST_MODE = False
+   ```
+
+## Pipeline Stages
+
+### Stage 1: NER & Name Cleaning
+- **Input**: `data/CSV/real_person_adapters.csv`
+- **Output**: `data/CSV/NER_POI_step01_pre_annotation.csv`
+- **Function**: Cleans adapter names to extract real person names
+  - Removes: emoji, "lora", "v1", special characters
+  - Example: "IU LoRA v2 🎤" → "IU"
+
+### Stage 2: Country/Nationality Mapping
+- **Input**: Step 1 output + `misc/lists/countries.csv`
+- **Output**: `data/CSV/NER_POI_step02_annotated.csv`
+- **Function**: Maps tags to standardized countries
+  - Example: "korean" → "South Korea"
+  - Excludes uninhabited territories
+
+### Stage 3: LLM Profession Annotation
+- **Input**: Step 2 output + `misc/lists/professions.csv`
+- **Output**: `data/CSV/{llm}_annotated_POI_test.csv` (test) or `{llm}_annotated_POI.csv` (full)
+- **Function**: Uses LLM to identify:
+  - Full name
+  - Gender
+  - Up to 3 professions (from profession list)
+  - Country
+- **Progress**: Automatically saves every 10 rows
+- **Resumable**: Can continue from last saved progress if interrupted
+
+## Configuration Options
+
+In the LLM annotation cell, you can configure:
+
+```python
+# Choose LLM provider
+SELECTED_LLM = 'qwen'  # Options: 'qwen', 'llama', 'mistral'
+
+# Test mode (recommended for first run)
+TEST_MODE = True       # True = test on small sample
+TEST_SIZE = 10         # Number of rows for testing
+
+# Processing limits
+MAX_ROWS = 20000      # Maximum rows to process (None = all)
+SAVE_INTERVAL = 10    # Save progress every N rows
+```
+
+## Expected Output Format
+
+The final dataset will include all original columns plus:
+
+| Column | Description | Example |
+|--------|-------------|---------|
+| `real_name` | Cleaned name | "IU" |
+| `full_name` | Full name from LLM | "Lee Ji-eun (IU)" |
+| `gender` | Gender from LLM | "Female" |
+| `profession_llm` | Up to 3 professions | "singer, actor, celebrity" |
+| `country` | Country from LLM | "South Korea" |
+| `likely_country` | Country from tags | "South Korea" |
+| `likely_nationality` | Nationality from tags | "South Korean" |
+| `tags` | Combined tags | "['korean', 'celebrity', 'singer']" |
+
+## Troubleshooting
+
+### API Key Errors
+```
+Warning: No API key for qwen
+```
+**Solution**: Ensure your API key file exists and contains only the key (no extra whitespace)
+
+### Rate Limiting
+```
+Qwen API error (attempt 1/3): 429 Too Many Requests
+```
+**Solution**: The code automatically retries with exponential backoff. You can also:
+- Increase `time.sleep(0.5)` to a higher value
+- Process in smaller batches
+
+### Progress Lost
+**Solution**: The pipeline saves progress automatically. Check:
+- `data/CSV/{llm}_annotated_POI_test.csv` - your partial results
+- `misc/{llm}_query_index.txt` - last processed index
+- Just re-run the cell and it will resume from the last saved progress
+
+### JSON Parse Errors from LLM
+```
+Qwen API error: JSONDecodeError
+```
+**Solution**: This is usually temporary. The code:
+- Returns "Unknown" for failed queries
+- Continues processing
+- You can manually review/reprocess failed entries later
+
+## Cost Management
+
+### Estimate Costs Before Processing
+
+For a dataset with N entries:
+- **Qwen**: Contact Alibaba Cloud for pricing
+- **Llama**: ~N × $0.0005 = ~$5 per 10k entries
+- **Mistral**: ~N × $0.004 = ~$40 per 10k entries
+
+### Best Practices
+
+1. **Always test first**: Run with `TEST_MODE = True` on 10 samples
+2. **Monitor API usage**: Check your API provider's dashboard
+3. **Use cheaper models first**: Try Llama before Mistral
+4. **Process in batches**: Set `MAX_ROWS` to process incrementally
+5. **Save intermediate results**: The automatic saving feature helps prevent data loss
+
+## Comparing Multiple LLMs
+
+To compare results from different LLMs:
+
+1. Run the pipeline with `SELECTED_LLM = 'qwen'`
+2. Change to `SELECTED_LLM = 'llama'` and run again
+3. Change to `SELECTED_LLM = 'mistral'` and run again
+4. Compare the three output files:
+   - `qwen_annotated_POI.csv`
+   - `llama_annotated_POI.csv`
+   - `mistral_annotated_POI.csv`
+
+## Files Created
+
+The pipeline creates these files:
+
+```
+data/CSV/
+├── NER_POI_step01_pre_annotation.csv     # After name cleaning
+├── NER_POI_step02_annotated.csv          # After country mapping
+├── qwen_annotated_POI_test.csv           # Test results (Qwen)
+├── qwen_annotated_POI.csv                # Full results (Qwen)
+├── llama_annotated_POI.csv               # Full results (Llama)
+└── mistral_annotated_POI.csv             # Full results (Mistral)
+
+misc/
+├── qwen_query_index.txt                  # Progress tracking
+├── llama_query_index.txt                 # Progress tracking
+└── mistral_query_index.txt               # Progress tracking
+```
+
+## Support
+
+For issues or questions:
+1. Check this guide for common problems
+2. Review `misc/credentials/README.md` for API setup
+3. Read the notebook documentation (first cell)
+4. Check API provider documentation for service-specific issues
+
+## Ethical Considerations
+
+This research documents ethical problems with AI deepfake models. The dataset and analysis help:
+- Understand the scope of unauthorized person likeness usage
+- Document professions/demographics most affected
+- Inform policy and technical solutions
+- Raise awareness about deepfake technology misuse
+
+Use this data responsibly and respect individual privacy and consent.

md/LLM_MODELS_COMPARISON.md

@@ -0,0 +1,326 @@
+# LLM Models for Deepfake Annotation
+
+## Overview
+
+The pipeline now includes **6 LLM options** in individual cells for easy comparison:
+
+1. **Deepseek** - Testing (use first!)
+2. **Qwen (API)** - Chinese (Alibaba Cloud)
+3. **Llama** - American (Meta)
+4. **Mixtral** - French (Mistral AI)
+5. **Gemma** - American Open Source (Google)
+6. **Qwen-2.5-32B Local** - FREE local inference (NEW!)
+
+## The 6 LLMs
+
+### 1. Deepseek (Testing)
+**Cell 10**
+
+- **Model**: deepseek-chat
+- **Provider**: DeepSeek
+- **API**: https://platform.deepseek.com/
+- **Cost**: ~$0.14-0.28 per 1M tokens (~$1-2 for 10k entries)
+- **Use case**: **Test this first!** Cheapest option to verify pipeline works
+- **API Key**: `misc/credentials/deepseek_api_key.txt`
+
+---
+
+### 2. Qwen API (Chinese)
+**Cells 11-12**
+
+- **Model**: qwen-max (automatically uses Qwen3-Max)
+- **Provider**: Alibaba Cloud DashScope
+- **API**: https://dashscope.aliyun.com/
+- **Cost**: Variable (check Alibaba pricing)
+- **Use case**: Chinese company, strong multilingual support
+- **API Key**: `misc/credentials/qwen_api_key.txt`
+- **Note**: Uses latest Qwen3-Max when you specify `qwen-max`
+
+---
+
+### 6. Qwen-2.5-32B Local (FREE!)
+**Cells 19-20** (NEW!)
+
+- **Model**: qwen2.5:32b-instruct
+- **Provider**: Ollama (local inference)
+- **Setup**: https://ollama.com/
+- **Cost**: **$0** (FREE - no API costs!)
+- **Requirements**:
+  - A100 80GB GPU (or similar)
+  - ~25GB VRAM during inference
+  - ~20GB storage for model download
+  - Ollama installed
+- **Speed**: 5-10 tokens/sec on A100 (~100-200 samples/hour)
+- **Use case**:
+  - ✅ Large datasets (>1000 samples) where cost matters
+  - ✅ Privacy-sensitive research data
+  - ✅ Offline processing
+  - ✅ Strong multilingual support
+- **Setup guide**: See `QWEN_LOCAL_SETUP.md`
+
+---
+
+### 3. Llama (American)
+**Cells 13-14**
+
+- **Model**: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
+- **Provider**: Together AI (hosting Meta's model)
+- **Developer**: Meta (American)
+- **API**: https://www.together.ai/
+- **Cost**: ~$0.90 per 1M tokens (~$5-10 for 10k entries)
+- **Use case**: Open-source American model, good quality
+- **API Key**: `misc/credentials/together_api_key.txt`
+
+---
+
+### 4. Mixtral (French)
+**Cells 15-16**
+
+- **Model**: open-mixtral-8x22b
+- **Provider**: Mistral AI
+- **Developer**: Mistral AI (French)
+- **API**: https://mistral.ai/
+- **Cost**: ~$2 per 1M tokens (~$10-20 for 10k entries)
+- **Use case**: European alternative, Mixture-of-Experts architecture
+- **API Key**: `misc/credentials/mistral_api_key.txt`
+- **Note**: Using open-mixtral-8x22b (cheaper than mistral-large)
+
+---
+
+### 5. Gemma (American Open Source)
+**Cells 17-18**
+
+- **Model**: google/gemma-2-27b-it
+- **Provider**: Together AI (hosting Google's model)
+- **Developer**: Google (American)
+- **API**: https://www.together.ai/ (same as Llama)
+- **Cost**: ~$0.80 per 1M tokens (~$4-8 for 10k entries)
+- **Use case**: American open-source alternative, competitive quality
+- **API Key**: `misc/credentials/together_api_key.txt` (same as Llama)
+- **Note**: Fully open-source, can be self-hosted
+
+---
+
+## Cost Comparison (10,000 entries)
+
+| Model | Provider | Cost | Time | Origin |
+|-------|----------|------|------|--------|
+| **Qwen-2.5-32B Local** | Ollama (local) | **$0** | ~50-100 hrs | 🇨🇳 Chinese |
+| **Deepseek** | DeepSeek | ~$1-2 | ~5-10 hrs | 🇨🇳 Chinese |
+| **Gemma 2** | Together AI | ~$4-8 | ~5-10 hrs | 🇺🇸 American (open) |
+| **Llama 3.1** | Together AI | ~$5-10 | ~5-10 hrs | 🇺🇸 American (open) |
+| **Mixtral** | Mistral AI | ~$10-20 | ~5-10 hrs | 🇫🇷 French (open) |
+| **Qwen API** | Alibaba | Variable | ~5-10 hrs | 🇨🇳 Chinese |
+
+**Note**: Local inference is FREE but slower. Good for large datasets where cost matters more than time.
+
+## Recommended Testing Order
+
+### 1. Start with Deepseek
+```python
+# Cell 10
+TEST_MODE = True
+TEST_SIZE = 10
+```
+- **Why**: Cheapest, verify pipeline works
+- **Cost**: Pennies for 10 samples
+
+### 2. Compare on Small Sample
+Pick 2-3 models and run on same 100 samples:
+```python
+# In each cell:
+TEST_MODE = True
+TEST_SIZE = 100
+```
+
+**Good combinations:**
+- Budget: Deepseek + Gemma
+- Quality: Llama + Mixtral
+- Geographic: Qwen + Llama + Mixtral
+
+### 3. Production Run
+Choose best model from testing and run full dataset:
+```python
+TEST_MODE = False
+MAX_ROWS = None  # or 20000
+```
+
+## API Key Setup
+
+### For Deepseek & Qwen (separate keys):
+```bash
+echo "your-deepseek-key" > misc/credentials/deepseek_api_key.txt
+echo "your-qwen-key" > misc/credentials/qwen_api_key.txt
+```
+
+### For Llama & Gemma (same Together AI key):
+```bash
+echo "your-together-key" > misc/credentials/together_api_key.txt
+```
+Both Llama and Gemma use the same Together AI key!
+
+### For Mixtral:
+```bash
+echo "your-mistral-key" > misc/credentials/mistral_api_key.txt
+```
+
+## Output Files
+
+Each LLM saves to a separate file:
+
+```
+data/CSV/
+├── deepseek_annotated_POI_test.csv       # Deepseek test
+├── deepseek_annotated_POI.csv            # Deepseek full
+├── qwen_annotated_POI_test.csv           # Qwen API test
+├── qwen_annotated_POI.csv                # Qwen API full
+├── qwen_local_annotated_POI_test.csv     # Qwen Local test (NEW!)
+├── qwen_local_annotated_POI.csv          # Qwen Local full (NEW!)
+├── llama_annotated_POI_test.csv          # Llama test
+├── llama_annotated_POI.csv               # Llama full
+├── mixtral_annotated_POI_test.csv        # Mixtral test
+├── mixtral_annotated_POI.csv             # Mixtral full
+├── gemma_annotated_POI_test.csv          # Gemma test
+└── gemma_annotated_POI.csv               # Gemma full
+```
+
+## Comparing Results
+
+After running multiple LLMs, compare results:
+
+```python
+import pandas as pd
+
+# Load results from different models
+deepseek_df = pd.read_csv('data/CSV/deepseek_annotated_POI_test.csv')
+qwen_df = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local_df = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')  # NEW!
+llama_df = pd.read_csv('data/CSV/llama_annotated_POI_test.csv')
+mixtral_df = pd.read_csv('data/CSV/mixtral_annotated_POI_test.csv')
+gemma_df = pd.read_csv('data/CSV/gemma_annotated_POI_test.csv')
+
+# Compare profession distributions
+print("Deepseek professions:", deepseek_df['profession_llm'].value_counts().head())
+print("Qwen API professions:", qwen_df['profession_llm'].value_counts().head())
+print("Qwen Local professions:", qwen_local_df['profession_llm'].value_counts().head())  # NEW!
+print("Llama professions:", llama_df['profession_llm'].value_counts().head())
+print("Mixtral professions:", mixtral_df['profession_llm'].value_counts().head())
+print("Gemma professions:", gemma_df['profession_llm'].value_counts().head())
+
+# Compare specific cases
+print("\nIrene identification:")
+print("Deepseek:", deepseek_df[deepseek_df['real_name'] == 'Irene']['full_name'].values)
+print("Qwen API:", qwen_df[qwen_df['real_name'] == 'Irene']['full_name'].values)
+print("Qwen Local:", qwen_local_df[qwen_local_df['real_name'] == 'Irene']['full_name'].values)
+print("Llama:", llama_df[llama_df['real_name'] == 'Irene']['full_name'].values)
+print("Mixtral:", mixtral_df[mixtral_df['real_name'] == 'Irene']['full_name'].values)
+print("Gemma:", gemma_df[gemma_df['real_name'] == 'Irene']['full_name'].values)
+```
+
+## Model Characteristics
+
+### Deepseek
+- ✅ Very cheap
+- ✅ Good for testing
+- ⚠️ Less documentation
+- 🇨🇳 Chinese company
+
+### Qwen (Qwen3-Max)
+- ✅ Latest version automatically used
+- ✅ Strong multilingual
+- ✅ Good Asian name recognition
+- 💰 Variable cost
+- 🇨🇳 Chinese company (Alibaba)
+
+### Llama 3.1 70B
+- ✅ Open-source
+- ✅ Strong overall performance
+- ✅ Well-documented
+- ✅ American (Meta)
+- 💰 Mid-range cost
+
+### Mixtral 8x22B
+- ✅ Open-source
+- ✅ MoE architecture (efficient)
+- ✅ European alternative
+- 💰 Mid-range cost
+- 🇫🇷 French company
+
+### Gemma 2 27B
+- ✅ Fully open-source
+- ✅ Can self-host
+- ✅ American (Google)
+- ✅ Cheap via API
+- ✅ Good quality for size
+
+### Qwen-2.5-32B Local (NEW!)
+- ✅ **FREE** - $0 cost (no API fees)
+- ✅ **FAST** - Local inference on A100 (5-10 tokens/sec)
+- ✅ **PRIVATE** - Data never leaves your machine
+- ✅ **OFFLINE** - Works without internet
+- ✅ **HIGH QUALITY** - 32B parameter model
+- ✅ Strong multilingual support
+- ⚠️ Requires: A100 80GB GPU, ~25GB VRAM, Ollama installed
+- 🇨🇳 Chinese company (Alibaba)
+- 📦 Model size: ~20GB download
+
+## Decision Matrix
+
+### If you prioritize...
+
+**FREE / Zero Cost**: Use **Qwen-2.5-32B Local** (no API fees!)
+
+**Cost** (with API): Use **Deepseek** or **Gemma**
+
+**Quality**: Use **Qwen-2.5-32B Local**, **Llama**, or **Mixtral**
+
+**Privacy**: Use **Qwen-2.5-32B Local** (data stays on your machine)
+
+**American/Open Source**: Use **Gemma** or **Llama**
+
+**Asian Names**: Use **Qwen** (API or Local - strong multilingual)
+
+**European Provider**: Use **Mixtral**
+
+**Testing**: Use **Deepseek** first, always!
+
+## Running Multiple Models
+
+You can run all 6 models in sequence:
+
+```python
+# 1. Run Cell 10 (Deepseek) - verify works (~$1-2 for 10k)
+# 2. Run Cell 12 (Qwen API) - Chinese perspective (~variable cost)
+# 3. Run Cell 14 (Llama) - American perspective (~$5-10 for 10k)
+# 4. Run Cell 16 (Mixtral) - European perspective (~$10-20 for 10k)
+# 5. Run Cell 18 (Gemma) - Open source perspective (~$4-8 for 10k)
+# 6. Run Cell 20 (Qwen-2.5-32B Local) - FREE local inference ($0!)
+```
+
+Each saves to its own file, so you can compare results!
+
+## Notes
+
+- **Llama and Gemma use the same API key** (Together AI)
+- All models use the **same 9 profession categories**
+- All models have **automatic retries** with exponential backoff
+- All models **save progress** every 10 rows
+- All models are **resumable** if interrupted
+
+## Summary
+
+You now have **6 LLM options** to choose from:
+
+1. 🧪 **Deepseek** - Test first (cheapest API)
+2. ����🇳 **Qwen3-Max API** - Chinese, strong multilingual
+3. 🇺🇸 **Llama 3.1 70B** - American, open-source
+4. 🇫🇷 **Mixtral 8x22B** - French, open-source MoE
+5. 🇺🇸 **Gemma 2 27B** - American open-source (Google)
+6. 💰 **Qwen-2.5-32B Local** - FREE local inference (NEW!)
+
+Each in its own cell, easy to run and compare! 🎉
+
+**Recommended workflow**:
+1. Test with Deepseek (Cell 10) - verify pipeline works
+2. For small datasets (<1000): Use API (Deepseek/Gemma/Llama)
+3. For large datasets (>1000): Use Qwen-2.5-32B Local (Cell 20) - FREE!

md/QUICK_START_LOCAL.md

@@ -0,0 +1,171 @@
+# Quick Start: Running Qwen-2.5-32B Locally
+
+This is a quick guide to get you started with FREE local LLM inference using your A100 GPU.
+
+## Why Local?
+
+✅ **$0 cost** - No API fees
+✅ **Privacy** - Data stays on your machine
+✅ **Quality** - 32B parameter model with strong performance
+
+## Setup (One-time)
+
+### 1. Pull the Model (~10-30 minutes)
+
+```bash
+# Pull Qwen-2.5-32B-Instruct
+ollama pull qwen2.5:32b-instruct
+
+# Wait for download to complete (~20GB)
+# Model will be cached at: ~/.ollama/models/
+```
+
+### 2. Verify Model is Ready
+
+```bash
+# List installed models
+ollama list
+
+# Should show: qwen2.5:32b-instruct
+
+# Test it
+ollama run qwen2.5:32b-instruct "Hello, who are you?"
+```
+
+If you see a response, you're ready! ✅
+
+## Running the Notebook
+
+### Open the Notebook
+
+```bash
+cd jupyter_notebooks
+jupyter notebook Section_2-3-4_Figure_8_deepfake_adapters.ipynb
+```
+
+### Run the Cells
+
+1. **Cell 5**: NER & Name Cleaning (processes names)
+2. **Cell 7**: Country/Nationality Mapping
+3. **Cell 20**: Qwen-2.5-32B Local Annotation 👈 **This is the new one!**
+
+### Configure Cell 20
+
+```python
+# Start with test mode
+TEST_MODE = True
+TEST_SIZE = 10
+
+# Then run full dataset
+TEST_MODE = False
+MAX_ROWS = 20000  # or None for all
+```
+
+### Run Cell 20
+
+Just click "Run" or press Shift+Enter. The cell will:
+1. Check if Ollama is installed ✅
+2. Check if model is available ✅
+3. Start annotating
+4. Save progress every 10 rows
+5. Show completion stats
+
+### Monitor Progress
+
+```
+Qwen Local: 100%|██████████| 10/10 [02:30<00:00, 15.0s/it]
+✅ Saved after 10 rows (~24.0 samples/hour)
+
+✅ Done! Results: data/CSV/qwen_local_annotated_POI_test.csv
+Total time: 2.5 minutes
+Average speed: 240.0 samples/hour
+```
+
+## Performance
+
+On your A100 80GB:
+- **Speed**: ~5-10 tokens/second
+- **Throughput**: ~100-200 samples/hour
+- **Memory**: ~22-25GB VRAM
+- **Cost**: $0
+
+### Time Estimates
+
+| Dataset Size | Time |
+|-------------|------|
+| 10 samples (test) | ~2-3 minutes |
+| 100 samples | ~20-30 minutes |
+| 1,000 samples | ~5-10 hours |
+| 10,000 samples | ~50-100 hours |
+
+**Tip**: Run overnight or over the weekend for large datasets!
+
+## Troubleshooting
+
+### "Model not found"
+
+```bash
+ollama pull qwen2.5:32b-instruct
+```
+
+### "Ollama not running"
+
+```bash
+ollama serve
+```
+
+### Out of Memory
+
+Your A100 has 80GB VRAM - this should NOT happen with the 32B model (~25GB VRAM).
+
+If it does, try the quantized version:
+```bash
+ollama pull qwen2.5:32b-instruct-q4_0  # Only ~12GB VRAM
+```
+
+## Output
+
+Results saved to:
+- Test: `data/CSV/qwen_local_annotated_POI_test.csv`
+- Full: `data/CSV/qwen_local_annotated_POI.csv`
+
+Same format as API results - easy to compare!
+
+## Custom Model Cache Location
+
+To store models in `data/models/`:
+
+```bash
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+ollama pull qwen2.5:32b-instruct
+```
+
+## Comparing API vs Local
+
+After running both:
+
+```python
+import pandas as pd
+
+qwen_api = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')
+
+# Check agreement
+agreement = (qwen_api['profession_llm'] == qwen_local['profession_llm']).mean()
+print(f"Agreement: {agreement*100:.1f}%")
+```
+
+## Full Documentation
+
+For more details, see:
+- `QWEN_LOCAL_SETUP.md` - Complete setup guide
+- `LLM_MODELS_COMPARISON.md` - All 6 LLM options compared
+
+## Summary
+
+✅ Ollama already installed
+✅ A100 80GB GPU - perfect for Qwen-2.5-32B
+✅ FREE inference - no API costs
+✅ Privacy - data stays local
+
+**Next step**: Run Cell 20 in the notebook! 🚀

md/QWEN_LOCAL_SETUP.md

@@ -0,0 +1,321 @@
+# Running Qwen-2.5-32B Locally with Ollama
+
+This guide explains how to run Qwen-2.5-32B-Instruct locally on your A100 GPU using Ollama.
+
+## Why Run Locally?
+
+✅ **FREE** - No API costs ($0 per query)
+✅ **FAST** - Local inference on A100 (5-10 tokens/sec)
+✅ **PRIVATE** - Data never leaves your machine
+✅ **OFFLINE** - Works without internet (after model download)
+✅ **HIGH QUALITY** - 32B parameter model with strong multilingual support
+
+## System Requirements
+
+### Minimum Specs
+- **GPU**: NVIDIA A100 80GB (or similar high-end GPU)
+- **VRAM**: 22-25GB during inference
+- **RAM**: 32GB system RAM (you have 265GB - more than enough!)
+- **Storage**: ~20GB for model download
+- **OS**: Linux (you're on Ubuntu)
+
+### Your Setup
+✅ NVIDIA A100 80GB - Perfect for Qwen-2.5-32B
+✅ 265GB RAM - Excellent
+✅ Linux (Ubuntu) - Supported
+✅ Ollama already installed at `/usr/local/bin/ollama`
+
+## Installation Steps
+
+### 1. Verify Ollama Installation
+
+```bash
+# Check if Ollama is installed
+which ollama
+# Should output: /usr/local/bin/ollama
+
+# Check Ollama version
+ollama --version
+```
+
+If not installed, install with:
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+```
+
+### 2. Pull Qwen-2.5-32B-Instruct Model
+
+```bash
+# This will download ~20GB
+ollama pull qwen2.5:32b-instruct
+
+# Alternative: Use the base model (not instruct-tuned)
+# ollama pull qwen2.5:32b
+```
+
+**Download time**: ~10-30 minutes depending on your internet speed.
+
+**Model cache location**: By default, models are cached at:
+- Linux: `~/.ollama/models/`
+
+To use custom cache location (e.g., `data/models/`):
+```bash
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+ollama pull qwen2.5:32b-instruct
+```
+
+### 3. Verify Model is Ready
+
+```bash
+# List all installed models
+ollama list
+
+# Test the model
+ollama run qwen2.5:32b-instruct "Hello, who are you?"
+```
+
+You should see a response from Qwen!
+
+### 4. Start Ollama Server (if needed)
+
+Ollama runs as a background service by default. If you need to start it manually:
+
+```bash
+# Start Ollama server
+ollama serve
+
+# Or run in background
+nohup ollama serve > /dev/null 2>&1 &
+```
+
+## Using Qwen-2.5-32B in the Notebook
+
+### Cell 20: Qwen-2.5-32B Local Annotation
+
+The notebook cell handles everything automatically:
+
+1. **Checks Ollama installation**
+2. **Verifies model availability**
+3. **Runs inference locally**
+4. **Saves progress every 10 rows**
+
+### Configuration
+
+```python
+# In Cell 20
+TEST_MODE = True        # Start with small test
+TEST_SIZE = 10          # Test on 10 samples first
+MAX_ROWS = 20000        # Full dataset size
+SAVE_INTERVAL = 10      # Save every 10 rows
+
+MODEL_NAME = "qwen2.5:32b-instruct"  # Model to use
+OLLAMA_HOST = "http://localhost:11434"  # Default Ollama port
+```
+
+### Running the Pipeline
+
+1. **Test run first** (recommended):
+   ```python
+   TEST_MODE = True
+   TEST_SIZE = 10
+   ```
+   Run Cell 20 to test on 10 samples (~1-2 minutes)
+
+2. **Check results**:
+   ```python
+   # Output saved to:
+   data/CSV/qwen_local_annotated_POI_test.csv
+   ```
+
+3. **Full run**:
+   ```python
+   TEST_MODE = False
+   MAX_ROWS = 20000  # or None for all rows
+   ```
+   Run Cell 20 for full dataset (~2-3 hours for 10k samples on A100)
+
+### Performance Expectations
+
+On NVIDIA A100 80GB:
+- **Speed**: 5-10 tokens/second
+- **Throughput**: 100-200 samples/hour (depends on prompt length)
+- **Memory**: ~22-25GB VRAM during inference
+- **Time for 10k samples**: ~50-100 hours (can run overnight/over weekend)
+
+### Monitoring
+
+The cell shows progress updates:
+```
+Qwen Local: 100%|██████████| 10/10 [02:30<00:00, 15.0s/it]
+✅ Saved after 10 rows (~24.0 samples/hour)
+
+✅ Done! Results: data/CSV/qwen_local_annotated_POI_test.csv
+Total time: 2.5 minutes
+Average speed: 240.0 samples/hour
+```
+
+## Troubleshooting
+
+### Model Not Found
+
+```bash
+# Check if model is installed
+ollama list
+
+# If not listed, pull it
+ollama pull qwen2.5:32b-instruct
+```
+
+### Ollama Server Not Running
+
+```bash
+# Check if Ollama is running
+ps aux | grep ollama
+
+# If not running, start it
+ollama serve
+```
+
+### GPU Not Detected
+
+```bash
+# Check NVIDIA GPU
+nvidia-smi
+
+# Check CUDA
+nvcc --version
+
+# Ollama should automatically detect GPU
+# If not, check Ollama logs
+journalctl -u ollama
+```
+
+### Out of Memory (OOM)
+
+If you get OOM errors:
+
+1. **Check VRAM usage**:
+   ```bash
+   watch -n 1 nvidia-smi
+   ```
+
+2. **Try smaller batch size** (not applicable here - we process 1 at a time)
+
+3. **Try quantized version** (smaller model):
+   ```bash
+   # 4-bit quantized version (~12GB VRAM)
+   ollama pull qwen2.5:32b-instruct-q4_0
+
+   # Update MODEL_NAME in notebook
+   MODEL_NAME = "qwen2.5:32b-instruct-q4_0"
+   ```
+
+### Slow Inference
+
+If inference is very slow (<1 token/sec):
+
+1. **Check GPU utilization**:
+   ```bash
+   nvidia-smi
+   ```
+   GPU should show ~90%+ utilization during inference
+
+2. **Check CPU vs GPU**:
+   Ollama might be using CPU instead of GPU
+   ```bash
+   # Force GPU usage
+   OLLAMA_GPU=1 ollama serve
+   ```
+
+## Model Variants
+
+Ollama provides several Qwen-2.5 variants:
+
+| Model | Size | VRAM | Speed | Quality |
+|-------|------|------|-------|---------|
+| `qwen2.5:32b-instruct` | 32B | ~25GB | Medium | Best |
+| `qwen2.5:32b-instruct-q4_0` | 32B (4-bit) | ~12GB | Fast | Good |
+| `qwen2.5:14b-instruct` | 14B | ~10GB | Fast | Good |
+| `qwen2.5:7b-instruct` | 7B | ~5GB | Very Fast | OK |
+
+For your A100 80GB, **`qwen2.5:32b-instruct`** is recommended (best quality, no VRAM issues).
+
+## Custom Model Cache Location
+
+To store models in `data/models/` directory:
+
+```bash
+# Set environment variable
+export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"
+
+# Add to ~/.bashrc for persistence
+echo 'export OLLAMA_MODELS="/home/lauhp/000_PHD/000_010_PUBLICATION/CODE/pm-paper/data/models"' >> ~/.bashrc
+
+# Pull model (will download to data/models/)
+ollama pull qwen2.5:32b-instruct
+
+# Verify
+ls -lh $OLLAMA_MODELS/
+```
+
+## Comparing Results
+
+After running both API and local versions, compare results:
+
+```python
+import pandas as pd
+
+# Load results
+qwen_api = pd.read_csv('data/CSV/qwen_annotated_POI_test.csv')
+qwen_local = pd.read_csv('data/CSV/qwen_local_annotated_POI_test.csv')
+
+# Compare professions
+print("API professions:", qwen_api['profession_llm'].value_counts().head())
+print("Local professions:", qwen_local['profession_llm'].value_counts().head())
+
+# Check agreement
+agreement = (qwen_api['profession_llm'] == qwen_local['profession_llm']).mean()
+print(f"Agreement rate: {agreement*100:.1f}%")
+```
+
+## Cost Comparison (10,000 samples)
+
+| Method | Cost | Time | Privacy |
+|--------|------|------|---------|
+| **Qwen Local (A100)** | **$0** | ~50-100 hours | ✅ Full |
+| Qwen API (Alibaba) | ~$10-20 | ~5-10 hours | ⚠️ Data sent to Alibaba |
+| Llama API (Together) | ~$5-10 | ~5-10 hours | ⚠️ Data sent to Together AI |
+| Deepseek API | ~$1-2 | ~5-10 hours | ⚠️ Data sent to Deepseek |
+
+**Recommendation**:
+- For **small tests** (<100 samples): Use API (faster)
+- For **large datasets** (>1000 samples): Use local (free, private)
+- For **research papers**: Use local to avoid data privacy concerns
+
+## Advanced: Parallel Processing
+
+For faster processing on multi-GPU setup:
+
+```python
+# Not implemented yet, but possible with:
+# - Multiple Ollama instances on different GPUs
+# - Ray or Dask for parallel processing
+# - ~4x speedup with 4 GPUs
+```
+
+## Summary
+
+✅ **Ollama** already installed
+✅ **A100 80GB** GPU - perfect for Qwen-2.5-32B
+✅ **Free inference** - no API costs
+✅ **Privacy** - data stays local
+
+**Next steps:**
+1. Pull model: `ollama pull qwen2.5:32b-instruct`
+2. Test with Cell 20: `TEST_MODE = True`, `TEST_SIZE = 10`
+3. Run full dataset: `TEST_MODE = False`
+
+**Estimated time for 10,000 samples**: ~50-100 hours
+**Cost**: $0
+
+Good luck! 🚀

md/SPACY_NER_EXPLANATION.md

@@ -0,0 +1,316 @@
+# spaCy NER Implementation
+
+## Why spaCy for NER?
+
+Using **spaCy's Named Entity Recognition (NER)** is significantly better than regex-based cleaning because:
+
+1. **Intelligent entity extraction**: Recognizes PERSON entities using machine learning
+2. **Context-aware**: Understands sentence structure and context
+3. **Robust**: Handles various name formats (first, last, full, stage names)
+4. **Language support**: Works with multiple languages and scripts
+5. **Industry standard**: Used in production NLP applications
+
+## How It Works
+
+### Pipeline Overview
+
+```
+Original Name
+    ↓
+1. Translate Leetspeak (4→a, 3→e, 1→i)
+    ↓
+2. Remove Noise (emoji, LoRA terms, versions)
+    ↓
+3. spaCy NER - Extract PERSON entities
+    ↓
+4. Fallback to capitalized words if needed
+    ↓
+Cleaned Name
+```
+
+### Detailed Steps
+
+#### Step 1: Leetspeak Translation
+```python
+"4kira LoRA v2" → "akira LoRA v2"
+"1rene Model" → "irene Model"
+"3mma Watson" → "emma Watson"
+```
+
+#### Step 2: Noise Removal
+```python
+"akira LoRA v2" → "akira"
+"irene Model" → "irene"
+"emma Watson" → "emma Watson"
+```
+
+#### Step 3: spaCy NER
+```python
+nlp("akira")
+# Entities: [("akira", PERSON)]
+# Result: "akira"
+
+nlp("emma Watson")
+# Entities: [("emma Watson", PERSON)]
+# Result: "emma Watson"
+```
+
+#### Step 4: Fallback
+If spaCy doesn't find a PERSON entity:
+- Extract capitalized words (likely names)
+- Or return cleaned text as-is
+
+## Examples
+
+### Case 1: Simple Name
+```
+Input:  "IU"
+Output: "IU"
+
+Process:
+  - Preprocess: "IU" (no noise)
+  - spaCy NER: Recognizes "IU" as PERSON
+  - Result: "IU"
+```
+
+### Case 2: Name with LoRA Terms
+```
+Input:  "Scarlett Johansson「LoRa」"
+Output: "Scarlett Johansson"
+
+Process:
+  - Preprocess: "Scarlett Johansson" (removed 「LoRa」)
+  - spaCy NER: Recognizes "Scarlett Johansson" as PERSON
+  - Result: "Scarlett Johansson"
+```
+
+### Case 3: Leetspeak Name
+```
+Input:  "4kira Anime Character v1"
+Output: "akira"
+
+Process:
+  - Leetspeak: "akira Anime Character v1"
+  - Preprocess: "akira Anime Character"
+  - spaCy NER: Recognizes "akira" as PERSON
+  - Result: "akira"
+```
+
+### Case 4: Complex Format
+```
+Input:  "Gakki | Aragaki Yui | 新垣結衣"
+Output: "Gakki"
+
+Process:
+  - Preprocess: "Gakki" (kept first part before |)
+  - spaCy NER: Recognizes "Gakki" as PERSON
+  - Result: "Gakki"
+```
+
+### Case 5: With Metadata
+```
+Input:  "Emma Watson (JG) v3.5"
+Output: "Emma Watson"
+
+Process:
+  - Preprocess: "Emma Watson" (removed (JG) and v3.5)
+  - spaCy NER: Recognizes "Emma Watson" as PERSON
+  - Result: "Emma Watson"
+```
+
+## Advantages Over Regex-Only
+
+### Old Approach (Regex Only)
+```python
+# Just remove noise and hope for the best
+name = remove_noise(name)
+name = name.strip()
+# Result: May include non-name words
+```
+
+Problems:
+- Can't distinguish names from other capitalized words
+- May include words like "Model", "Anime", "Character"
+- No context awareness
+- Language-dependent regex patterns needed
+
+### New Approach (spaCy NER)
+```python
+# Intelligent entity extraction
+preprocessed = remove_noise(name)
+doc = nlp(preprocessed)
+person_names = [ent.text for ent in doc.ents if ent.label_ == "PERSON"]
+# Result: Only actual person names
+```
+
+Benefits:
+- ✅ Identifies actual person entities
+- ✅ Ignores non-person words
+- ✅ Context-aware (understands "Emma Watson" is one entity)
+- ✅ Multi-language support
+- ✅ Handles various name formats
+
+## Comparison Examples
+
+| Input | Regex Only | spaCy NER |
+|-------|------------|-----------|
+| `"Emma Watson Model"` | `"Emma Watson Model"` ❌ | `"Emma Watson"` ✅ |
+| `"Anime Character Levi"` | `"Anime Character Levi"` ❌ | `"Levi"` ✅ |
+| `"Taylor Swift v2"` | `"Taylor Swift"` ✅ | `"Taylor Swift"` ✅ |
+| `"K4te Middleton"` | `"K4te Middleton"` ❌ | `"Kate Middleton"` ✅ |
+| `"Celebrity IU"` | `"Celebrity IU"` ❌ | `"IU"` ✅ |
+
+## spaCy Model Information
+
+### Model Used
+- **Name**: `en_core_web_sm`
+- **Language**: English (but works reasonably with romanized names)
+- **Size**: ~13 MB
+- **Entities**: Recognizes PERSON, ORG, GPE, etc.
+
+### Installation
+```bash
+# Install spaCy
+pip install spacy
+
+# Download model
+python -m spacy download en_core_web_sm
+```
+
+The notebook automatically downloads the model if not found.
+
+### Performance
+- **Speed**: ~1000-5000 docs/second
+- **Accuracy**: High for common names
+- **Memory**: Low (~100MB loaded)
+
+## Fallback Strategy
+
+If spaCy doesn't recognize a PERSON entity:
+
+1. **Extract capitalized words**:
+   ```python
+   "unknown name here" → ["unknown"]
+   ```
+
+2. **Return first few capitalized words**:
+   ```python
+   "Celebrity Model Actor" → "Celebrity Model Actor"
+   ```
+
+3. **Last resort**: Return cleaned text as-is
+
+This ensures we always get something, even for:
+- Uncommon/rare names
+- Nicknames
+- Non-English names
+- Stage names
+
+## Testing
+
+### How to Verify spaCy is Working
+
+Run Cell 5 and check the output:
+
+```
+✅ spaCy model loaded: en_core_web_sm
+
+📊 Name cleaning examples (with spaCy NER):
+===================================================================================================
+Original Name                                      | Cleaned Name
+===================================================================================================
+Scarlett Johansson「LoRa」                        | Scarlett Johansson
+Emma Watson (JG)                                  | Emma Watson
+IU                                                | IU
+Belle Delphine                                    | Belle Delphine
+...
+```
+
+### Key Indicators
+
+✅ **Good signs**:
+- Person names cleanly extracted
+- No extra words like "Model", "LoRA", "Celebrity"
+- Multi-word names kept together (e.g., "Emma Watson" not just "Emma")
+
+❌ **Issues to watch**:
+- Empty results (increase fallback logic)
+- Partial names (e.g., only first name)
+- Non-names included (tune preprocessing)
+
+## Customization
+
+### Add More Languages
+
+For better support of non-English names:
+
+```python
+# Download multilingual model
+python -m spacy download xx_ent_wiki_sm
+
+# Use in code
+nlp = spacy.load("xx_ent_wiki_sm")
+```
+
+### Adjust Entity Extraction
+
+To extract other entities:
+
+```python
+# Extract organizations too
+entities = [ent.text for ent in doc.ents
+            if ent.label_ in ["PERSON", "ORG"]]
+```
+
+### Custom Entity Rules
+
+Add custom patterns for names spaCy might miss:
+
+```python
+from spacy.matcher import Matcher
+
+matcher = Matcher(nlp.vocab)
+# Add patterns for specific name formats
+```
+
+## Benefits for This Project
+
+### Better Person Identification
+
+With cleaner names:
+- LLMs receive recognizable names
+- "Emma Watson" instead of "Emma Watson Model LoRA v3"
+- Better identification accuracy
+
+### Reduced Ambiguity
+
+spaCy helps distinguish:
+- Person names vs. descriptive words
+- "Celebrity IU" → "IU" (person)
+- "Model Bella" → "Bella" (person)
+
+### Improved Context for LLMs
+
+Cleaner input = better prompts:
+```
+Before: "Given 'Celebrity Model Emma Watson LoRA v2' (actress)..."
+After:  "Given 'Emma Watson' (actress)..."
+```
+
+The LLM can now focus on identifying the person, not parsing the noise.
+
+## Summary
+
+✅ **spaCy NER** provides intelligent, context-aware name extraction
+✅ **Better than regex** for handling complex name formats
+✅ **Fallback strategy** ensures we always get a result
+✅ **Industry standard** tool used in production NLP
+✅ **Easy to use** with minimal code
+
+The combination of:
+1. Leetspeak translation
+2. Noise removal
+3. spaCy NER
+4. Smart fallbacks
+
+...results in clean, accurate person names ready for LLM annotation!

md/TESTING_INSTRUCTIONS.md

@@ -0,0 +1,148 @@
+# Quick Testing Instructions
+
+## Start Here! 🚀
+
+You mentioned you have Deepseek credits, so **start by testing with Deepseek first** before trying the other LLMs.
+
+## Step-by-Step Testing
+
+### 1. Make sure your Deepseek API key is in place
+
+Check if this file exists:
+```bash
+cat misc/credentials/deepseek_api_key.txt
+```
+
+If not, create it:
+```bash
+echo "your-deepseek-api-key" > misc/credentials/deepseek_api_key.txt
+```
+
+### 2. Open the notebook
+
+```bash
+jupyter notebook jupyter_notebooks/Section_2-3-4_Figure_8_deepfake_adapters.ipynb
+```
+
+### 3. Run the cells in order
+
+1. **Cell 0-4**: Introduction and setup (just markdown, no execution needed)
+2. **Cell 5**: NER & Name Cleaning (processes `real_person_adapters.csv`)
+3. **Cell 7**: Country/Nationality Mapping
+4. **Cell 10**: 🌟 **DEEPSEEK ANNOTATION** (TEST THIS FIRST!)
+   - Default: `TEST_MODE = True` (10 samples)
+   - Will create: `data/CSV/deepseek_annotated_POI_test.csv`
+5. **Cell 12**: Qwen/Llama/Mistral (run later after Deepseek works)
+
+### 4. Review Deepseek Results
+
+After Cell 10 completes, check:
+- Console output shows summary statistics
+- Output file: `data/CSV/deepseek_annotated_POI_test.csv`
+
+Example output should look like:
+```
+✅ Progress saved after 10 rows
+✅ Done! Final results saved to data/CSV/deepseek_annotated_POI_test.csv
+
+=== Summary Statistics ===
+Total processed: 10
+
+Gender distribution:
+Female    8
+Male      2
+...
+```
+
+### 5. If Deepseek Works Well
+
+Once you're satisfied with the Deepseek results:
+
+**Option A: Process full dataset with Deepseek**
+```python
+# In Cell 10, change:
+TEST_MODE = False
+```
+
+**Option B: Try other LLMs for comparison**
+1. Set up API keys for Qwen/Llama/Mistral (see `misc/credentials/README.md`)
+2. Run Cell 12 with your chosen LLM:
+   ```python
+   SELECTED_LLM = 'qwen'  # or 'llama' or 'mistral'
+   TEST_MODE = True       # Test first!
+   ```
+
+## Expected Cost (Deepseek)
+
+- **10 samples** (test): ~$0.01 or less
+- **1,000 entries**: ~$0.10-0.20
+- **10,000 entries**: ~$1-2
+
+Much cheaper than the other options, making it perfect for testing!
+
+## Troubleshooting
+
+### "deepseek_api_key.txt not found"
+```bash
+# Create the file with your key
+echo "your-api-key" > misc/credentials/deepseek_api_key.txt
+```
+
+### "File does not exist: real_person_adapters.csv"
+Make sure the input file exists:
+```bash
+ls -lh data/CSV/real_person_adapters.csv
+```
+
+### API Rate Limiting
+The code includes automatic rate limiting (`time.sleep(1)` between requests). If you still get rate limited:
+- Increase the sleep time in Cell 10: change `time.sleep(1)` to `time.sleep(2)`
+
+### Pipeline Interrupted
+No problem! The code saves progress every 10 rows. Just re-run the cell and it will resume from where it left off.
+
+## What's Next?
+
+After testing with Deepseek:
+
+1. **If results look good**: Scale up to full dataset with Deepseek
+2. **Compare LLMs**: Test Qwen/Llama/Mistral on the same sample to see which gives best results
+3. **Production run**: Choose your preferred LLM and process the full dataset
+
+## File Outputs
+
+The pipeline creates these files:
+
+```
+data/CSV/
+├── NER_POI_step01_pre_annotation.csv       # After Cell 5 (name cleaning)
+├── NER_POI_step02_annotated.csv            # After Cell 7 (country mapping)
+├── deepseek_annotated_POI_test.csv         # After Cell 10 (test mode)
+├── deepseek_annotated_POI.csv              # After Cell 10 (full mode)
+├── qwen_annotated_POI_test.csv             # After Cell 12 (if using Qwen)
+└── ...
+
+misc/
+├── deepseek_query_index.txt                # Progress tracking
+└── ...
+```
+
+## Quick Commands
+
+```bash
+# View first few results
+head -20 data/CSV/deepseek_annotated_POI_test.csv
+
+# Count processed rows
+wc -l data/CSV/deepseek_annotated_POI_test.csv
+
+# Check progress
+cat misc/deepseek_query_index.txt
+
+# Reset progress (start from scratch)
+rm misc/deepseek_query_index.txt
+```
+
+---
+
+**Ready to start?** Open the notebook and run Cell 5 → Cell 7 → Cell 10! 🎉

md/UPDATES_AND_FIXES.md

@@ -0,0 +1,235 @@
+# Recent Updates and Fixes
+
+## Overview
+
+Two important fixes have been implemented based on testing feedback:
+
+1. **Leetspeak Translation** (before NER)
+2. **Improved Country Mapping** (check ALL tags)
+
+---
+
+## Fix 1: Leetspeak Translation
+
+### Problem
+Names with leetspeak (numbers replacing letters) weren't being properly cleaned:
+- `4kira` should be `Akira`
+- `1rene` should be `Irene`
+- `3mma` should be `Emma`
+
+### Solution
+Added leetspeak translation **before** other NER processing in Cell 5.
+
+### Mapping Table
+| Leetspeak | Letter |
+|-----------|--------|
+| 4 | A |
+| 3 | E |
+| 1 | I |
+| 0 | O |
+| 7 | T |
+| 5 | S |
+| 8 | B |
+| 9 | G |
+| @ | A |
+| $ | S |
+| ! | I |
+
+### Examples
+```
+4kira     -> akira
+3mma      -> emma
+1rene     -> irene
+L3vi      -> Levi
+S4sha     -> Sasha
+K4te      -> Kate
+J3ssica   -> Jessica
+```
+
+### Implementation
+The `translate_leetspeak()` function runs FIRST in `clean_name()`, before emoji removal and other cleaning steps. This ensures leetspeak is converted to proper letters before any other processing.
+
+---
+
+## Fix 2: Improved Country Mapping
+
+### Problem
+The country mapping was stopping at the first match, which meant:
+- **Irene** with tags `['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']`
+- The `'korean'` tag wasn't being properly mapped to `'South Korea'`
+- This resulted in incomplete hints being sent to the LLM
+- **Expected**: Deepseek should identify **Bae Joo-hyun (Irene)** from Red Velvet
+
+### Solution
+Updated Cell 7 to:
+1. **Check ALL tags** (not just stop at first match)
+2. **Use a priority system** to select the best match:
+   - Priority 3: Exact country name match (highest)
+   - Priority 2: Nationality match (medium)
+   - Priority 1: Word parts (lowest)
+
+### How It Works
+
+#### Before (Broken)
+```python
+def infer_country_and_nationality(tags):
+    for tag in tags:
+        if tag in mapping:
+            return mapping[tag]  # ❌ Stops at first match!
+    return ("", "")
+```
+
+#### After (Fixed)
+```python
+def infer_country_and_nationality(tags):
+    best_match = None
+    best_priority = 0
+
+    for tag in tags:  # ✅ Check ALL tags
+        if tag in mapping:
+            country, nationality, priority = mapping[tag]
+            if priority > best_priority:
+                best_match = (country, nationality)
+                best_priority = priority
+
+    return best_match or ("", "")
+```
+
+### Example: Irene Case
+
+**Input Tags**: `['girl', 'photorealistic', 'asian', 'woman', 'beautiful', 'celebrity', 'korean']`
+
+**Processing**:
+1. Check `'girl'` → no match
+2. Check `'photorealistic'` → no match
+3. Check `'asian'` → no match (too generic)
+4. Check `'woman'` → no match
+5. Check `'beautiful'` → no match
+6. Check `'celebrity'` → no match
+7. Check `'korean'` → ✅ **MATCH!**
+   - Maps to nationality: `'South Korean'`
+   - Which maps to country: `'South Korea'`
+   - Priority: 2 (nationality match)
+
+**Output**:
+- `likely_country`: `'South Korea'`
+- `likely_nationality`: `'South Korean'`
+
+**Sent to Deepseek**:
+```
+Given 'Irene' (celebrity, South Korea), provide:
+1. Full legal name
+2. Aliases
+3. Gender
+4. Top 3 professions
+5. Country
+```
+
+**Expected Result**: Deepseek can now identify this as **Bae Joo-hyun (Irene)**, a South Korean singer/actress from the K-pop group Red Velvet.
+
+---
+
+## Impact on Results
+
+### Better Name Recognition
+- Leetspeak names are now properly translated
+- LLMs receive cleaner, more recognizable names
+
+### Better Country Context
+- All tags are now considered for country mapping
+- More accurate country/nationality hints sent to LLMs
+- Better identification of international celebrities
+
+### Example Improvements
+
+| Name | Tags | Before | After |
+|------|------|--------|-------|
+| `4kira LoRA` | `['japanese', 'actress']` | `'4kira'` + no country | `'Akira'` + `'Japan'` |
+| `Irene` | `['korean', 'celebrity']` | `'Irene'` + no country | `'Irene'` + `'South Korea'` |
+| `1U` | `['korean', 'singer']` | `'1U'` + no country | `'IU'` + `'South Korea'` |
+| `3lsa` | `['model']` | `'3lsa'` + no country | `'Elsa'` + country if tagged |
+
+---
+
+## Testing Recommendations
+
+### Before Running Full Pipeline
+
+1. **Test Leetspeak Translation** (Cell 5):
+   ```python
+   # Look for names with numbers in the output
+   # Verify they're properly translated
+   ```
+
+2. **Test Country Mapping** (Cell 7):
+   ```python
+   # Check the debug output at the end:
+   # "🔍 Checking 'Irene' entries:"
+   # Verify country is properly mapped
+   ```
+
+3. **Test Deepseek Results** (Cell 10):
+   ```python
+   # Look for Irene in the results
+   # Should now identify as Bae Joo-hyun
+   ```
+
+### Validation Checklist
+
+- [ ] Leetspeak names are translated (check console output in Cell 5)
+- [ ] Country mapping shows high success rate (check stats in Cell 7)
+- [ ] Irene is correctly identified as Bae Joo-hyun (check results in Cell 10)
+- [ ] Other K-pop/Korean celebrities are properly identified
+- [ ] Japanese/Chinese celebrities also benefit from better country mapping
+
+---
+
+## Notes
+
+### Why Check ALL Tags?
+
+Some entries have many tags, and the most informative tag might not be first:
+```
+tags = ['girl', 'sexy', 'beautiful', 'asian', 'korean', 'celebrity', 'kpop']
+                                                  ^^^^ Most informative!
+```
+
+The old code might stop at `'girl'` or `'asian'` (no country info), missing the `'korean'` tag.
+
+### Why Use Priority?
+
+Some tags might match multiple countries. Priority ensures we get the best match:
+- `'american'` → exact nationality match (priority 2) → USA
+- `'america'` → could be North/South/Central America (priority 1)
+
+The system picks the higher priority match.
+
+### Word Length Filter
+
+Word parts only match if >4 characters to avoid false positives:
+- ✅ `'china'` → matches China (5 chars)
+- ❌ `'us'` → too short, might be part of other words
+
+---
+
+## Future Improvements
+
+Potential enhancements:
+1. **More leetspeak patterns**: `|\/|` for M, `(_)` for U, etc.
+2. **Fuzzy country matching**: Handle typos like `'corean'` → `'korean'`
+3. **Multi-country support**: Some celebrities work in multiple countries
+4. **Language detection**: Use name structure to infer origin
+
+---
+
+## Summary
+
+✅ **Leetspeak translation** ensures names are readable before NER
+✅ **ALL tags checked** ensures no country hints are missed
+✅ **Priority system** ensures best match is selected
+✅ **Better LLM results** from improved name quality and country context
+
+These fixes should significantly improve the accuracy of person identification, especially for:
+- International celebrities (K-pop, J-pop, C-pop)
+- Names with leetspeak
+- Entries where country info appears later in tag list

misc/assets/fonts/DejaVuSans.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7da195a74c55bef988d0d48f9508bd5d849425c1770dba5d7bfc6ce9ed848954
+size 757076

misc/assets/fonts/Noto_Sans.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7801615f2c5a7a107313cd0a88a15c3b1f15a2da9d4a3648cf49711d8be44da
+size 47636998

misc/assets/fonts/Noto_Sans/NotoSans-Italic-VariableFont_wdth,wght.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe5a1fafb96618733aa4ea4c14a2e76ee65fee0d042b040e374f17575467e433
+size 2637272

misc/assets/fonts/Noto_Sans/NotoSans-VariableFont_wdth,wght.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74df1f61ab9d4bfaa961c65f8dc991deaae2885b0a6a6d6a60ed23980b3c8554
+size 2490816

misc/assets/fonts/Noto_Sans/OFL.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be2f3f8727ac2e18b714ad1c4336d4ddb3f3adbeb9a7f70bfab74d21f4d2b3fb
+size 4489