Upload 15 files

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+codabench_portrait_train_balanced_v2.json filter=lfs diff=lfs merge=lfs -text
+codabench_portrait_train_multitask_v2.json filter=lfs diff=lfs merge=lfs -text
+codabench_portrait_train_score_v2.json filter=lfs diff=lfs merge=lfs -text
+codabench_portrait_train.json filter=lfs diff=lfs merge=lfs -text
+codabench_portrait_val_multitask_v2.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,191 @@
+# Portrait Composition Aesthetics Evaluation
+
+This project provides an inference and fusion pipeline for portrait composition aesthetics evaluation. It converts the official test set into multi-prompt prediction inputs, runs prediction with Stage2 LoRA adapters, and fuses multiple prediction files into the final `track_1_test.json`.
+
+## 1. Project Structure
+
+```text
+project_root/
+  prepare_test_prompt_v2.py          # Build multi-prompt test inputs
+  convert_to_answer.py               # Fuse multi-path prediction results
+  train_track1_stage1_qwen3vl8b_v1.yaml
+  train_track1_stage2_qwen3vl8b_v1.yaml
+  qwen3vl8b_r1stage2_seed_ensemble_fast_20260511_112825/
+    seed20260511_stage2/             # Stage2 adapter 1
+    seed20260512_stage2/             # Stage2 adapter 2
+  datasets/
+    images/                          # Test images
+    original_annotations/
+      track_1_test.json              # Official test template
+  outputs/
+    predictions/                     # Model prediction outputs
+    submissions/answers/             # Final submission file
+```
+
+For path conventions, see `REPRODUCIBLE_PATHS.md`.
+
+## 2. Environment
+
+Use an environment that supports Qwen3-VL and LoRA inference. LLaMA-Factory is recommended for prediction.
+
+Basic dependencies:
+
+```bash
+pip install torch transformers peft accelerate
+```
+
+If you use LLaMA-Factory, install it first and make sure `llamafactory-cli` is available.
+
+Base model:
+
+```text
+Qwen/Qwen3-VL-8B-Instruct
+```
+
+## 3. Prepare Test Data
+
+Place the official test file at:
+
+```text
+datasets/original_annotations/track_1_test.json
+```
+
+Place test images under:
+
+```text
+datasets/images/
+```
+
+Generate multi-prompt test inputs:
+
+```bash
+python prepare_test_prompt_v2.py \
+  --input_json datasets/original_annotations/track_1_test.json \
+  --image_dir datasets/images \
+  --output_json outputs/predictions/track1_test_convert.json \
+  --prompt_variants v1,v2,v3,v4,v6,v7,v8
+```
+
+This command generates:
+
+```text
+outputs/predictions/track1_test_convert_v1.json
+outputs/predictions/track1_test_convert_v2.json
+outputs/predictions/track1_test_convert_v3.json
+outputs/predictions/track1_test_convert_v4.json
+outputs/predictions/track1_test_convert_v6.json
+outputs/predictions/track1_test_convert_v7.json
+outputs/predictions/track1_test_convert_v8.json
+```
+
+## 4. Run Multi-Path Prediction
+
+Use the Stage2 adapters to run prediction on the generated prompt variants:
+
+```text
+qwen3vl8b_r1stage2_seed_ensemble_fast_20260511_112825/seed20260511_stage2
+qwen3vl8b_r1stage2_seed_ensemble_fast_20260511_112825/seed20260512_stage2
+```
+
+If you use LLaMA-Factory, register each test JSON in `dataset_info.json`. Example:
+
+```json
+{
+  "track1_test_v1": {
+    "file_name": "outputs/predictions/track1_test_convert_v1.json",
+    "formatting": "sharegpt",
+    "columns": {
+      "messages": "messages",
+      "images": "images"
+    }
+  }
+}
+```
+
+Example prediction config:
+
+```yaml
+model_name_or_path: Qwen/Qwen3-VL-8B-Instruct
+adapter_name_or_path: qwen3vl8b_r1stage2_seed_ensemble_fast_20260511_112825/seed20260511_stage2
+template: qwen3_vl_nothink
+stage: sft
+finetuning_type: lora
+
+dataset_dir: .
+eval_dataset: track1_test_v1
+do_predict: true
+predict_with_generate: true
+cutoff_len: 4096
+image_max_pixels: 786432
+per_device_eval_batch_size: 8
+max_new_tokens: 384
+do_sample: false
+top_p: 1.0
+repetition_penalty: 1.02
+bf16: true
+flash_attn: auto
+
+output_dir: outputs/predictions/seed20260511_v1
+```
+
+Run prediction:
+
+```bash
+llamafactory-cli train predict_seed20260511_v1.yaml
+```
+
+Each prediction run should produce:
+
+```text
+outputs/predictions/<run_name>/generated_predictions.jsonl
+```
+
+## 5. Fuse Prediction Results
+
+Copy or symlink the official test template to:
+
+```text
+outputs/predictions/track_1_test.json
+```
+
+Fuse prediction results:
+
+```bash
+python convert_to_answer.py \
+  --template_json outputs/predictions/track_1_test.json \
+  --predictions_jsonl \
+    outputs/predictions/seed20260511_v1/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v2/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v3/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v4/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v6/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v7/generated_predictions.jsonl \
+    outputs/predictions/seed20260511_v8/generated_predictions.jsonl \
+  --weights 1 1 1 1 1 1 1 \
+  --best_index 0 \
+  --total_score_fusion mean \
+  --output_json outputs/submissions/answers/track_1_test.json
+```
+
+To fuse results from both Stage2 adapters, add all corresponding `generated_predictions.jsonl` files to `--predictions_jsonl` and provide the same number of values in `--weights`.
+
+## 6. Output
+
+The final result is:
+
+```text
+outputs/submissions/answers/track_1_test.json
+```
+
+The file preserves the official template structure and fills in the fused:
+
+- `criteria.level`
+- `total_score`
+- `answer`
+
+## 7. Notes
+
+- All paths are relative to the project root.
+- Image paths use the format `datasets/images/<image_name>.jpg`.
+- `convert_to_answer.py` supports multi-path prediction fusion, weighted answer voting, majority voting for levels, and mean fusion for `total_score`.
+- If the model output is not strict JSON, the fusion script attempts to extract the JSON snippet and answer field from text.

REPRODUCIBLE_PATHS.md

@@ -0,0 +1,65 @@
+# Reproducible Path Convention
+
+This project no longer depends on machine-specific absolute paths. All code and data files use public model IDs or paths relative to the project root.
+
+## Expected Directory Layout
+
+```text
+project_root/
+  codabench_portrait_train.json
+  codabench_portrait_val.json
+  prepare_dataset.py
+  prepare_test_prompt_v2.py
+  convert_to_answer.py
+  train_track1_stage1_qwen3vl8b_v1.yaml
+  train_track1_stage2_qwen3vl8b_v1.yaml
+  datasets/
+    images/
+      *.jpg
+    original_annotations/
+      track_1_test.json
+  outputs/
+    predictions/
+    submissions/
+  saves/
+```
+
+## Path Rules
+
+- Base model: `Qwen/Qwen3-VL-8B-Instruct`
+- Training and validation image paths: `datasets/images/<image_name>.jpg`
+- Official test template: `outputs/predictions/track_1_test.json` or `datasets/original_annotations/track_1_test.json`
+- Prediction outputs: `outputs/predictions/`
+- Submission outputs: `outputs/submissions/answers/`
+- LoRA adapter outputs: `saves/`
+- Data generation logs: `logs/`
+
+## Typical Usage
+
+Generate training data:
+
+```bash
+python prepare_dataset.py \
+  --train_json codabench_portrait_train.json \
+  --val_json codabench_portrait_val.json \
+  --output_dir .
+```
+
+Generate multi-prompt test data:
+
+```bash
+python prepare_test_prompt_v2.py \
+  --input_json datasets/original_annotations/track_1_test.json \
+  --image_dir datasets/images \
+  --output_json outputs/predictions/track1_test_convert.json \
+  --prompt_variants v1,v2,v3,v4,v6,v7,v8
+```
+
+Fuse prediction results:
+
+```bash
+python convert_to_answer.py \
+  --template_json outputs/predictions/track_1_test.json \
+  --predictions_jsonl outputs/predictions/run_v1/generated_predictions.jsonl outputs/predictions/run_v2/generated_predictions.jsonl \
+  --output_json outputs/submissions/answers/track_1_test.json
+```

codabench_portrait_train.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ead68273d71a50331d47af49f9b1194cfe2d4755abf1fed66d7b1999745878e2
+size 82860499

codabench_portrait_train_balanced_v2.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1c3bfe5b58b18736b429e8c452690e5930c636c50a39eebf23bf92d6f35c0e7
+size 237557794

codabench_portrait_train_multitask_v2.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02d06def9036381eda440c83e290bb9d73643d8e4d8beb9f975402f8f18fd478
+size 96688216

codabench_portrait_train_score_v2.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f96f0445c5c5acd5a84fd2c6d029ca2317a52fc44185b4463dd0c845219f8d24
+size 79931263

codabench_portrait_val_multitask_v2.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b2556588585076c9be25afad59c0214c82cd3b2c77a51afe4768a09645fffd7
+size 10742247

convert_to_answer.py

@@ -0,0 +1,566 @@
+import argparse
+import copy
+import json
+import os
+import re
+from collections import Counter
+from glob import glob
+from typing import Any, Dict, List, Optional, Tuple
+
+
+DEFAULT_TEMPLATE_PATH = "outputs/predictions/track_1_test.json"
+DEFAULT_PRED_PATH = "outputs/predictions/predict_ckpt2660/generated_predictions.jsonl"
+DEFAULT_OUTPUT_PATH = "outputs/submissions/answers/track_1_test.json"
+
+
+def load_json(path: str) -> Any:
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def load_jsonl(path: str) -> List[Dict[str, Any]]:
+    rows: List[Dict[str, Any]] = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line_no, line in enumerate(f, start=1):
+            raw = line.strip()
+            if not raw:
+                continue
+            try:
+                rows.append(json.loads(raw))
+            except Exception:
+                print(f"[WARN] skip invalid jsonl line: {line_no}")
+    return rows
+
+
+def normalize_level(level: Any) -> Optional[str]:
+    if level is None:
+        return None
+    s = str(level).strip()
+    if not s:
+        return None
+
+    m = {
+        "poor": "Poor",
+        "medium": "Medium",
+        "good": "Good",
+        "a": "A",
+        "b": "B",
+        "c": "C",
+    }
+    return m.get(s.lower(), s)
+
+
+def score_to_submission_level(score: float, low_threshold: float = 5.0, high_threshold: float = 7.0) -> str:
+    # 比赛映射：A=Poor(0~5), B=Medium(5~7), C=Good(7~10)
+    if score < low_threshold:
+        return "A"
+    if score < high_threshold:
+        return "B"
+    return "C"
+
+
+def to_submission_level(level: Any) -> Optional[str]:
+    """
+    Submission mapping required by Track-1 template:
+    A -> Poor, B -> Medium, C -> Good
+    """
+    norm = normalize_level(level)
+    if norm is None:
+        return None
+    m = {
+        "Poor": "A",
+        "Medium": "B",
+        "Good": "C",
+        "A": "A",
+        "B": "B",
+        "C": "C",
+    }
+    return m.get(norm)
+
+
+def normalize_prediction_text(pred_row: Dict[str, Any]) -> str:
+    # 兼容不同推理后端字段名。
+    for k in ("predict", "response", "output", "text", "generation"):
+        if k in pred_row:
+            return str(pred_row.get(k, ""))
+    return str(pred_row)
+
+
+def extract_level_from_text(pred_text: str, crit_key: str) -> Optional[str]:
+    pattern = rf'"{re.escape(crit_key)}"\s*:\s*\{{.*?"level"\s*:\s*"([^"]+)"'
+    m = re.search(pattern, pred_text, flags=re.IGNORECASE | re.DOTALL)
+    if not m:
+        return None
+    return to_submission_level(m.group(1))
+
+
+def try_parse_predict_json(text: str) -> Optional[Dict[str, Any]]:
+    text = str(text or "").strip()
+    if not text:
+        return None
+
+    # 1) 直接解析
+    try:
+        obj = json.loads(text)
+        if isinstance(obj, dict):
+            return obj
+    except Exception:
+        pass
+
+    # 2) 尝试提取从首个 { 到最后一个 } 的片段
+    left = text.find("{")
+    right = text.rfind("}")
+    if left != -1 and right != -1 and left < right:
+        snippet = text[left:right + 1]
+        try:
+            obj = json.loads(snippet)
+            if isinstance(obj, dict):
+                return obj
+        except Exception:
+            pass
+
+    return None
+
+
+def extract_answer(pred_obj: Optional[Dict[str, Any]], pred_text: str) -> Optional[str]:
+    if pred_obj is not None:
+        answer = str(pred_obj.get("answer", "")).strip().upper()
+        if answer in {"A", "B", "C", "D"}:
+            return answer
+
+    m = re.search(r'"answer"\s*:\s*"([A-D])"', pred_text, re.IGNORECASE)
+    if m:
+        return m.group(1).upper()
+
+    # 兼容 QA-only 推理：模型可能只输出单个字母（如 "C"）。
+    raw = str(pred_text or "").strip().upper()
+    if raw:
+        # 情况1：整行仅有一个候选字母（允许尾随标点）。
+        m = re.match(r"^\s*([A-D])(?:[\.\)\]:：]|\s)*$", raw)
+        if m:
+            return m.group(1)
+        # 情况2：短文本中只出现唯一一个 A/B/C/D，且不包含常见 JSON/选项结构。
+        if len(raw) <= 12 and ("{" not in raw) and ("\"" not in raw):
+            hits = re.findall(r"[A-D]", raw)
+            if len(hits) == 1:
+                return hits[0]
+    return None
+
+
+def extract_total_score(pred_obj: Optional[Dict[str, Any]], pred_text: str) -> Optional[float]:
+    val: Optional[float] = None
+    if pred_obj is not None and "total_score" in pred_obj:
+        try:
+            val = float(pred_obj["total_score"])
+        except Exception:
+            val = None
+
+    if val is None:
+        m = re.search(r'"total_score"\s*:\s*([0-9]+(?:\.[0-9]+)?)', pred_text)
+        if m:
+            val = float(m.group(1))
+
+    if val is None:
+        return None
+
+    return max(0.0, min(100.0, float(val)))
+
+
+def merge_total_scores(total_votes: List[float], mode: str) -> Optional[int]:
+    if not total_votes:
+        return None
+
+    vals = [float(v) for v in total_votes]
+    if mode == "trim_mean" and len(vals) >= 3:
+        vals = sorted(vals)[1:-1]
+    merged = sum(vals) / len(vals)
+    return max(0, min(100, int(round(merged))))
+
+
+def parse_score_from_text(pred_text: str, crit_key: str) -> Optional[float]:
+    pattern = rf'"{re.escape(crit_key)}"\s*:\s*\{{.*?"score"\s*:\s*([0-9]+(?:\.[0-9]+)?)'
+    m = re.search(pattern, pred_text, flags=re.IGNORECASE | re.DOTALL)
+    if not m:
+        return None
+    try:
+        return float(m.group(1))
+    except Exception:
+        return None
+
+
+def extract_one_criteria_level_and_score(
+    crit_key: str,
+    pred_obj: Optional[Dict[str, Any]],
+    pred_text: str,
+) -> Tuple[Optional[str], Optional[float]]:
+    score: Optional[float] = None
+    level: Optional[str] = None
+
+    if isinstance(pred_obj, dict):
+        src = pred_obj.get("criteria", {})
+        if isinstance(src, dict) and crit_key in src:
+            value = src.get(crit_key)
+            if isinstance(value, dict):
+                if "score" in value:
+                    try:
+                        score = float(value["score"])
+                    except Exception:
+                        score = None
+                level = to_submission_level(value.get("level"))
+            else:
+                level = to_submission_level(value)
+
+    if score is None:
+        score = parse_score_from_text(pred_text, crit_key)
+
+    if level is None:
+        level = extract_level_from_text(pred_text, crit_key)
+
+    if level not in {"A", "B", "C"}:
+        level = None
+
+    return level, score
+
+
+def choose_majority_level(
+    levels: List[str],
+    scores: List[float],
+    fallback_level: Optional[str],
+    low_threshold: float,
+    high_threshold: float,
+) -> str:
+    if levels:
+        cnt = Counter(levels)
+        top_n = max(cnt.values())
+        top_levels = sorted([k for k, v in cnt.items() if v == top_n])
+        if len(top_levels) == 1:
+            return top_levels[0]
+
+    # 平票时，用多次预测的均值 score 判定等级。
+    if scores:
+        mean_score = sum(scores) / len(scores)
+        return score_to_submission_level(mean_score, low_threshold=low_threshold, high_threshold=high_threshold)
+
+    if fallback_level in {"A", "B", "C"}:
+        return fallback_level
+    return "B"
+
+
+def parse_weights(raw_weights: Optional[List[float]], n_models: int) -> List[float]:
+    # 默认等权；若传入权重则要求和预测文件数一致。
+    if raw_weights is None:
+        return [1.0] * n_models
+    if len(raw_weights) != n_models:
+        raise ValueError(
+            f"--weights length ({len(raw_weights)}) must equal number of prediction files ({n_models})."
+        )
+    for w in raw_weights:
+        if w < 0:
+            raise ValueError("weights must be non-negative.")
+    # 全零没有意义，回退等权。
+    if sum(raw_weights) == 0:
+        return [1.0] * n_models
+    return raw_weights
+
+
+def load_thresholds(
+    thresholds_json: str,
+) -> Tuple[Dict[str, Dict[str, float]], Dict[str, float]]:
+    """
+    Accepts JSON in either format:
+    1) {"criteria": {"Color Harmony": {"low": 4.9, "high": 7.1}}, "default": {"low":5,"high":7}}
+    2) {"Color Harmony": {"low": 4.9, "high": 7.1}, ...}
+    """
+    default = {"low": 5.0, "high": 7.0}
+    per_criteria: Dict[str, Dict[str, float]] = {}
+
+    if not thresholds_json:
+        return per_criteria, default
+    if not os.path.exists(thresholds_json):
+        print(f"[WARN] thresholds file not found: {thresholds_json}, fallback to default 5/7")
+        return per_criteria, default
+
+    obj = load_json(thresholds_json)
+    if not isinstance(obj, dict):
+        print(f"[WARN] invalid thresholds json format: {thresholds_json}, fallback to default 5/7")
+        return per_criteria, default
+
+    if "default" in obj and isinstance(obj.get("default"), dict):
+        d = obj["default"]
+        low = d.get("low", 5.0)
+        high = d.get("high", 7.0)
+        try:
+            low_f = float(low)
+            high_f = float(high)
+            if low_f < high_f:
+                default = {"low": low_f, "high": high_f}
+        except Exception:
+            pass
+
+    src = obj.get("criteria") if isinstance(obj.get("criteria"), dict) else obj
+    if isinstance(src, dict):
+        for k, v in src.items():
+            if not isinstance(v, dict):
+                continue
+            if "low" not in v or "high" not in v:
+                continue
+            try:
+                low = float(v["low"])
+                high = float(v["high"])
+            except Exception:
+                continue
+            if low < high:
+                per_criteria[str(k)] = {"low": low, "high": high}
+
+    return per_criteria, default
+
+
+def resolve_best_index(best_index: int, weights: List[float]) -> int:
+    # best_index=-1 表示自动选择权重最高的模型作为平票时的优先模型。
+    if best_index >= 0:
+        if best_index >= len(weights):
+            raise ValueError(f"--best_index out of range: {best_index}, num_models={len(weights)}")
+        return best_index
+    return max(range(len(weights)), key=lambda i: weights[i])
+
+
+def extract_criteria_voting(
+    template_item: Dict[str, Any],
+    pred_objs: List[Optional[Dict[str, Any]]],
+    pred_texts: List[str],
+    per_criteria_thresholds: Dict[str, Dict[str, float]],
+    default_thresholds: Dict[str, float],
+) -> Dict[str, Dict[str, str]]:
+    out: Dict[str, Dict[str, str]] = {}
+
+    for crit_key, crit_val in template_item.get("criteria", {}).items():
+        thresholds = per_criteria_thresholds.get(crit_key, default_thresholds)
+        low = float(thresholds.get("low", 5.0))
+        high = float(thresholds.get("high", 7.0))
+        if not (low < high):
+            low, high = 5.0, 7.0
+
+        level_votes: List[str] = []
+        score_votes: List[float] = []
+        for obj, text in zip(pred_objs, pred_texts):
+            level, score = extract_one_criteria_level_and_score(crit_key, obj, text)
+            if level is not None:
+                level_votes.append(level)
+            if score is not None:
+                score_votes.append(score)
+
+        prev = str(crit_val.get("level", "")).strip() if isinstance(crit_val, dict) else ""
+        final_level = choose_majority_level(
+            level_votes,
+            score_votes,
+            prev,
+            low_threshold=low,
+            high_threshold=high,
+        )
+        out[crit_key] = {"level": final_level}
+
+    return out
+
+
+def pick_default_predictions_path() -> str:
+    candidates = [
+        "outputs/predictions/predict_ckpt2660/generated_predictions.jsonl",
+        "outputs/predictions/generated_predictions.jsonl",
+    ]
+    # 自动兜底：在预测目录里找最近一次 generated_predictions.jsonl。
+    dynamic = sorted(
+        glob("outputs/predictions/**/generated_predictions.jsonl", recursive=True),
+        key=lambda x: os.path.getmtime(x),
+        reverse=True,
+    )
+    candidates = dynamic + candidates
+
+    for p in candidates:
+        if os.path.exists(p):
+            return p
+    return DEFAULT_PRED_PATH
+
+
+def choose_weighted_answer(
+    votes_by_model: List[Optional[str]],
+    weights: List[float],
+    best_index: int,
+    tie_break_answer: Optional[str],
+) -> str:
+    """
+    更合理的答案融合策略：
+    1) 加权投票（按各变体可靠性权重）
+    2) 若平票且提供 tie-break 结果，则优先用 tie-break
+    3) 若仍平票，采用最佳变体(best_index)在平票选项中的答案
+    4) 最后才做稳定兜底（字母序）
+    """
+    label_scores = {"A": 0.0, "B": 0.0, "C": 0.0, "D": 0.0}
+    for i, ans in enumerate(votes_by_model):
+        if ans in label_scores:
+            label_scores[ans] += weights[i]
+
+    max_score = max(label_scores.values())
+    if max_score <= 0:
+        return "A"
+
+    tied = sorted([k for k, v in label_scores.items() if v == max_score])
+    if len(tied) == 1:
+        return tied[0]
+
+    if tie_break_answer in tied:
+        return tie_break_answer  # 用专门 tie-break 结果判平票
+
+    best_vote = votes_by_model[best_index]
+    if best_vote in tied:
+        return str(best_vote)
+
+    return tied[0]
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--template_json", type=str, default=DEFAULT_TEMPLATE_PATH)
+    parser.add_argument(
+        "--predictions_jsonl",
+        type=str,
+        nargs="+",
+        default=None,
+        help="One or more generated_predictions.jsonl paths for voting.",
+    )
+    parser.add_argument(
+        "--weights",
+        type=float,
+        nargs="+",
+        default=None,
+        help="Optional weights for prediction files (same length as --predictions_jsonl).",
+    )
+    parser.add_argument(
+        "--best_index",
+        type=int,
+        default=-1,
+        help="Best model index for tie fallback. -1 means auto argmax(weights).",
+    )
+    parser.add_argument(
+        "--tie_break_jsonl",
+        type=str,
+        default="",
+        help="Optional tie-break predictions file used only when weighted vote ties.",
+    )
+    parser.add_argument(
+        "--thresholds_json",
+        type=str,
+        default="",
+        help="Optional per-criterion thresholds json for score->A/B/C mapping.",
+    )
+    parser.add_argument(
+        "--total_score_fusion",
+        type=str,
+        default="mean",
+        choices=["mean", "trim_mean"],
+        help="Fusion mode for total_score across multi-prompt predictions.",
+    )
+    parser.add_argument("--output_json", type=str, default=DEFAULT_OUTPUT_PATH)
+    args = parser.parse_args()
+
+    pred_paths = args.predictions_jsonl if args.predictions_jsonl else [pick_default_predictions_path()]
+    pred_paths = [p for p in pred_paths if str(p).strip()]
+    if not pred_paths:
+        raise ValueError("No predictions_jsonl provided or discovered.")
+
+    template_data = load_json(args.template_json)
+    pred_sets = [load_jsonl(p) for p in pred_paths]
+    weights = parse_weights(args.weights, len(pred_sets))
+    best_index = resolve_best_index(args.best_index, weights)
+    per_criteria_thresholds, default_thresholds = load_thresholds(args.thresholds_json)
+    tie_break_rows: List[Dict[str, Any]] = []
+    if args.tie_break_jsonl:
+        tie_break_rows = load_jsonl(args.tie_break_jsonl)
+
+    if not isinstance(template_data, list):
+        raise ValueError("template_json must be a list.")
+
+    print(f"[INFO] template items: {len(template_data)}")
+    for p, rows in zip(pred_paths, pred_sets):
+        print(f"[INFO] prediction rows: {len(rows)} ({p})")
+    print(f"[INFO] answer weights: {weights}")
+    print(f"[INFO] answer best_index: {best_index}")
+    print(f"[INFO] criteria thresholds default: low={default_thresholds['low']}, high={default_thresholds['high']}")
+    if per_criteria_thresholds:
+        print(f"[INFO] criteria thresholds loaded: {len(per_criteria_thresholds)}")
+    if args.tie_break_jsonl:
+        print(f"[INFO] tie_break rows: {len(tie_break_rows)} ({args.tie_break_jsonl})")
+
+    final_submission: List[Dict[str, Any]] = []
+    parsed_ok = 0
+    filled = 0
+
+    for i, item in enumerate(template_data):
+        new_item = copy.deepcopy(item)
+        # 收集每次预测在第 i 条样本上的结果。
+        row_texts: List[str] = []
+        row_objs: List[Optional[Dict[str, Any]]] = []
+        answer_votes_by_model: List[Optional[str]] = []
+        total_votes: List[int] = []
+
+        for rows in pred_sets:
+            if i >= len(rows):
+                answer_votes_by_model.append(None)
+                continue
+            pred_text = normalize_prediction_text(rows[i])
+            pred_obj = try_parse_predict_json(pred_text)
+            if pred_obj is not None:
+                parsed_ok += 1
+            row_texts.append(pred_text)
+            row_objs.append(pred_obj)
+
+            ans = extract_answer(pred_obj, pred_text)
+            answer_votes_by_model.append(ans)
+
+            ts = extract_total_score(pred_obj, pred_text)
+            if ts is not None:
+                total_votes.append(ts)
+
+        if not row_texts:
+            final_submission.append(new_item)
+            continue
+
+        new_item["criteria"] = extract_criteria_voting(
+            new_item,
+            row_objs,
+            row_texts,
+            per_criteria_thresholds=per_criteria_thresholds,
+            default_thresholds=default_thresholds,
+        )
+
+        # total_score 用多次预测均值，减少单次抖动。
+        merged_total = merge_total_scores(total_votes, args.total_score_fusion)
+        if merged_total is not None:
+            new_item["total_score"] = max(0, min(100, merged_total))
+
+        tie_break_answer: Optional[str] = None
+        if i < len(tie_break_rows):
+            tb_text = normalize_prediction_text(tie_break_rows[i])
+            tb_obj = try_parse_predict_json(tb_text)
+            tie_break_answer = extract_answer(tb_obj, tb_text)
+
+        new_item["answer"] = choose_weighted_answer(
+            votes_by_model=answer_votes_by_model,
+            weights=weights,
+            best_index=best_index,
+            tie_break_answer=tie_break_answer,
+        )
+
+        final_submission.append(new_item)
+        filled += 1
+
+    os.makedirs(os.path.dirname(args.output_json), exist_ok=True)
+    with open(args.output_json, "w", encoding="utf-8") as f:
+        json.dump(final_submission, f, ensure_ascii=False, indent=2)
+
+    parsed_total = filled * len(pred_paths)
+    print(f"[INFO] parsed predict json ok: {parsed_ok}/{parsed_total}")
+    print(f"[INFO] saved submission: {args.output_json}")
+
+
+if __name__ == "__main__":
+    main()

prepare_dataset.py

@@ -0,0 +1,361 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import argparse
+import json
+import math
+import os
+import random
+from collections import Counter, defaultdict
+from dataclasses import dataclass
+from typing import Any, Dict, List, Tuple
+
+
+CRITERIA = [
+    "Color Harmony",
+    "Visual Style Consistency",
+    "Sharpness",
+    "Light and Shadow Modeling",
+    "Creativity and Originality",
+    "Exposure Control",
+    "Application of Classical Composition Principles",
+    "Depth of Field and Layering",
+    "Visual Center Stability",
+    "Visual Flow Guidance",
+    "Structural Support Stability",
+    "Appropriateness of Negative Space",
+    "Subject Integrity",
+]
+
+
+PROMPT_SCORE = (
+    "You are an aesthetics expert. Evaluate the image on the following 13 criteria:\n"
+    "Color Harmony, Visual Style Consistency, Sharpness, Light and Shadow Modeling, "
+    "Creativity and Originality, Exposure Control, Application of Classical Composition Principles, "
+    "Depth of Field and Layering, Visual Center Stability, Visual Flow Guidance, "
+    "Structural Support Stability, Appropriateness of Negative Space, Subject Integrity.\n\n"
+    "For each criterion, output a numeric score in [0.0, 10.0].\n"
+    "Then output total_score as an integer in [0, 100].\n"
+    "Return STRICT JSON only.\n"
+    "JSON format:\n"
+    "{\n"
+    "  \"criteria\": {\n"
+    "    \"Color Harmony\": {\"score\": 0.0}\n"
+    "  },\n"
+    "  \"total_score\": 0\n"
+    "}\n\n"
+    "<image>"
+)
+
+
+PROMPT_MULTITASK = (
+    "You are an aesthetics expert. Evaluate the image on the following 13 criteria:\n"
+    "Color Harmony, Visual Style Consistency, Sharpness, Light and Shadow Modeling, "
+    "Creativity and Originality, Exposure Control, Application of Classical Composition Principles, "
+    "Depth of Field and Layering, Visual Center Stability, Visual Flow Guidance, "
+    "Structural Support Stability, Appropriateness of Negative Space, Subject Integrity.\n\n"
+    "For each criterion, output:\n"
+    "1) score in [0.0, 10.0]\n"
+    "2) level in {A,B,C} where A:<5, B:[5,7), C:>=7\n"
+    "Then output total_score as an integer in [0, 100].\n"
+    "Return STRICT JSON only.\n"
+    "JSON format:\n"
+    "{\n"
+    "  \"criteria\": {\n"
+    "    \"Color Harmony\": {\"score\": 0.0, \"level\": \"A|B|C\"}\n"
+    "  },\n"
+    "  \"total_score\": 0\n"
+    "}\n\n"
+    "<image>"
+)
+
+
+@dataclass
+class Sample:
+    image_path: str
+    scores: Dict[str, float]
+    levels: Dict[str, str]
+    total_score: int
+    boundary_count: int
+
+
+def clamp_score(x: float) -> float:
+    return max(0.0, min(10.0, x))
+
+
+def clamp_int(x: int, lo: int, hi: int) -> int:
+    return max(lo, min(hi, x))
+
+
+def score_to_level(score: float) -> str:
+    if score < 5.0:
+        return "A"
+    if score < 7.0:
+        return "B"
+    return "C"
+
+
+def safe_float(x: Any, default: float = 0.0) -> float:
+    try:
+        return float(x)
+    except Exception:
+        return default
+
+
+def safe_int(x: Any, default: int = 0) -> int:
+    try:
+        return int(round(float(x)))
+    except Exception:
+        return default
+
+
+def parse_assistant_payload(payload: str) -> Dict[str, Any]:
+    obj = json.loads(payload)
+    if not isinstance(obj, dict):
+        raise ValueError("assistant payload is not dict")
+    return obj
+
+
+def normalize_record(rec: Dict[str, Any], boundary_margin: float) -> Sample:
+    assistant = parse_assistant_payload(rec["messages"][1]["content"])
+    criteria = assistant.get("criteria", {})
+
+    scores: Dict[str, float] = {}
+    levels: Dict[str, str] = {}
+    boundary_count = 0
+
+    for c in CRITERIA:
+        v = (criteria.get(c, {}) or {}).get("score", 0.0)
+        s = round(clamp_score(safe_float(v, 0.0)), 1)
+        lv = score_to_level(s)
+        scores[c] = s
+        levels[c] = lv
+        if abs(s - 5.0) <= boundary_margin or abs(s - 7.0) <= boundary_margin:
+            boundary_count += 1
+
+    t = assistant.get("total_score", 0)
+    total_score = clamp_int(safe_int(t, 0), 0, 100)
+
+    img = rec.get("images", [""])
+    image_path = str(img[0]) if isinstance(img, list) and img else ""
+
+    return Sample(
+        image_path=image_path,
+        scores=scores,
+        levels=levels,
+        total_score=total_score,
+        boundary_count=boundary_count,
+    )
+
+
+def build_score_record(s: Sample) -> Dict[str, Any]:
+    assistant = {
+        "criteria": {k: {"score": round(v, 1)} for k, v in s.scores.items()},
+        "total_score": int(s.total_score),
+    }
+    return {
+        "messages": [
+            {"role": "user", "content": PROMPT_SCORE},
+            {"role": "assistant", "content": json.dumps(assistant, ensure_ascii=False, indent=2)},
+        ],
+        "images": [s.image_path],
+    }
+
+
+def build_multitask_record(s: Sample) -> Dict[str, Any]:
+    assistant = {
+        "criteria": {
+            k: {"score": round(s.scores[k], 1), "level": s.levels[k]} for k in CRITERIA
+        },
+        "total_score": int(s.total_score),
+    }
+    return {
+        "messages": [
+            {"role": "user", "content": PROMPT_MULTITASK},
+            {"role": "assistant", "content": json.dumps(assistant, ensure_ascii=False, indent=2)},
+        ],
+        "images": [s.image_path],
+    }
+
+
+def compute_level_counts(samples: List[Sample]) -> Dict[str, Counter]:
+    counts: Dict[str, Counter] = {c: Counter() for c in CRITERIA}
+    for s in samples:
+        for c in CRITERIA:
+            counts[c][s.levels[c]] += 1
+    return counts
+
+
+def compute_sample_dup(
+    s: Sample,
+    level_counts: Dict[str, Counter],
+    max_class_weight: float,
+    boundary_bonus: float,
+    max_dup: int,
+) -> int:
+    weights: List[float] = []
+    for c in CRITERIA:
+        cnt = level_counts[c]
+        max_freq = max(cnt.values()) if cnt else 1
+        cur = cnt.get(s.levels[c], 1)
+        w = math.sqrt(float(max_freq) / float(max(1, cur)))
+        w = min(max_class_weight, max(1.0, w))
+        weights.append(w)
+
+    avg_w = sum(weights) / len(weights)
+    raw_dup = avg_w + boundary_bonus * float(s.boundary_count)
+    dup = int(round(raw_dup))
+    return clamp_int(dup, 1, max_dup)
+
+
+def save_json(path: str, obj: Any) -> None:
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(obj, f, ensure_ascii=False, indent=2)
+
+
+def build_stats(
+    train_samples: List[Sample],
+    val_samples: List[Sample],
+    level_counts: Dict[str, Counter],
+    dup_counts: Counter,
+    balanced_size: int,
+) -> Dict[str, Any]:
+    agg = Counter()
+    for c in CRITERIA:
+        agg.update(level_counts[c])
+
+    total_lv = sum(agg.values())
+    agg_ratio = {k: round(v / total_lv, 6) for k, v in agg.items()} if total_lv else {}
+
+    per_criterion = {}
+    for c in CRITERIA:
+        cnt = level_counts[c]
+        n = sum(cnt.values())
+        per_criterion[c] = {
+            "counts": dict(cnt),
+            "ratio": {k: round(cnt[k] / n, 6) if n else 0.0 for k in ["A", "B", "C"]},
+        }
+
+    train_total = [s.total_score for s in train_samples]
+    val_total = [s.total_score for s in val_samples]
+
+    return {
+        "train_count": len(train_samples),
+        "val_count": len(val_samples),
+        "balanced_train_count": balanced_size,
+        "level_distribution_aggregate": dict(agg),
+        "level_distribution_aggregate_ratio": agg_ratio,
+        "level_distribution_per_criterion": per_criterion,
+        "duplication_histogram": {str(k): v for k, v in sorted(dup_counts.items())},
+        "boundary_stats": {
+            "train_mean_boundary_count": round(sum(s.boundary_count for s in train_samples) / max(1, len(train_samples)), 4),
+            "train_max_boundary_count": max((s.boundary_count for s in train_samples), default=0),
+        },
+        "total_score": {
+            "train_mean": round(sum(train_total) / max(1, len(train_total)), 4),
+            "val_mean": round(sum(val_total) / max(1, len(val_total)), 4),
+            "train_min": min(train_total) if train_total else None,
+            "train_max": max(train_total) if train_total else None,
+            "val_min": min(val_total) if val_total else None,
+            "val_max": max(val_total) if val_total else None,
+        },
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--train_json",
+        type=str,
+        default="./codabench_portrait_train.json",
+    )
+    parser.add_argument(
+        "--val_json",
+        type=str,
+        default="./codabench_portrait_val.json",
+    )
+    parser.add_argument("--output_dir", type=str, default=".")
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--boundary_margin", type=float, default=0.4)
+    parser.add_argument("--boundary_bonus", type=float, default=0.3)
+    parser.add_argument("--max_class_weight", type=float, default=4.0)
+    parser.add_argument("--max_dup", type=int, default=5)
+    parser.add_argument(
+        "--stats_json",
+        type=str,
+        default="logs/prepare_dataset_v2_stats.json",
+    )
+    args = parser.parse_args()
+
+    random.seed(args.seed)
+
+    with open(args.train_json, "r", encoding="utf-8") as f:
+        train_raw = json.load(f)
+    with open(args.val_json, "r", encoding="utf-8") as f:
+        val_raw = json.load(f)
+
+    train_samples = [normalize_record(x, args.boundary_margin) for x in train_raw]
+    val_samples = [normalize_record(x, args.boundary_margin) for x in val_raw]
+
+    level_counts = compute_level_counts(train_samples)
+
+    train_score = [build_score_record(s) for s in train_samples]
+    train_multitask = [build_multitask_record(s) for s in train_samples]
+    val_score = [build_score_record(s) for s in val_samples]
+    val_multitask = [build_multitask_record(s) for s in val_samples]
+
+    train_balanced: List[Dict[str, Any]] = []
+    dup_hist = Counter()
+    for s in train_samples:
+        dup = compute_sample_dup(
+            s=s,
+            level_counts=level_counts,
+            max_class_weight=args.max_class_weight,
+            boundary_bonus=args.boundary_bonus,
+            max_dup=args.max_dup,
+        )
+        dup_hist[dup] += 1
+        rec = build_multitask_record(s)
+        for _ in range(dup):
+            train_balanced.append(rec)
+
+    random.shuffle(train_balanced)
+
+    os.makedirs(args.output_dir, exist_ok=True)
+
+    f_train_score = os.path.join(args.output_dir, "codabench_portrait_train_score_v2.json")
+    f_train_multitask = os.path.join(args.output_dir, "codabench_portrait_train_multitask_v2.json")
+    f_train_balanced = os.path.join(args.output_dir, "codabench_portrait_train_balanced_v2.json")
+    f_val_score = os.path.join(args.output_dir, "codabench_portrait_val_score_v2.json")
+    f_val_multitask = os.path.join(args.output_dir, "codabench_portrait_val_multitask_v2.json")
+
+    save_json(f_train_score, train_score)
+    save_json(f_train_multitask, train_multitask)
+    save_json(f_train_balanced, train_balanced)
+    save_json(f_val_score, val_score)
+    save_json(f_val_multitask, val_multitask)
+
+    stats = build_stats(
+        train_samples=train_samples,
+        val_samples=val_samples,
+        level_counts=level_counts,
+        dup_counts=dup_hist,
+        balanced_size=len(train_balanced),
+    )
+    save_json(args.stats_json, stats)
+
+    print("=" * 60)
+    print("saved:", f_train_score)
+    print("saved:", f_train_multitask)
+    print("saved:", f_train_balanced)
+    print("saved:", f_val_score)
+    print("saved:", f_val_multitask)
+    print("stats:", args.stats_json)
+    print("train_count:", len(train_samples), "balanced_count:", len(train_balanced), "val_count:", len(val_samples))
+    print("dup_hist:", dict(sorted(dup_hist.items())))
+    print("=" * 60)
+
+
+if __name__ == "__main__":
+    main()

prepare_test_prompt_v2.py

@@ -0,0 +1,639 @@
+import argparse
+import json
+import os
+from typing import Dict, List
+
+CRITERIA = [
+    "Color Harmony",
+    "Visual Style Consistency",
+    "Sharpness",
+    "Light and Shadow Modeling",
+    "Creativity and Originality",
+    "Exposure Control",
+    "Application of Classical Composition Principles",
+    "Depth of Field and Layering",
+    "Visual Center Stability",
+    "Visual Flow Guidance",
+    "Structural Support Stability",
+    "Appropriateness of Negative Space",
+    "Subject Integrity",
+]
+
+# 三组不同措辞的 prompt，用于后续多次推理投票。
+# 输出 schema 完全一致，避免后处理复杂化。
+PROMPT_VARIANTS = {
+    "v1": """You are a portrait-aesthetics judge focused on robust cross-image ranking.
+
+Task:
+- Score all 13 criteria with floats in [0.0, 10.0].
+- Output total_score as an integer in [0, 100].
+- Choose answer from A/B/C/D.
+
+Scoring policy for total_score:
+- Use holistic visual quality, not a plain arithmetic average of 13 scores.
+- Keep strong separation when quality difference is obvious.
+- Penalize major defects decisively (blur, broken anatomy, unstable structure, severe exposure failure).
+- Reward clearly excellent lighting/composition/subject quality.
+
+Typical anchors (not strict formula):
+- severe failure: 0-30
+- poor: 31-45
+- fair: 46-60
+- good: 61-75
+- very good: 76-88
+- excellent: 89-100
+
+Hard output constraints:
+- STRICT JSON only.
+- No level/reason/x.
+- No extra keys.
+- One decimal place for criterion scores when possible.
+
+Return exactly:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v2": """Act as a strict technical-quality portrait judge.
+
+Priority:
+1) Technical correctness first (sharpness, exposure, structure, subject integrity).
+2) Then composition and visual flow.
+3) Then style/creativity refinement.
+
+Rules:
+- If critical technical defects are obvious, lower affected criteria strongly and reduce total_score accordingly.
+- Do not overuse middle scores.
+- total_score must reflect defect severity and overall impression, not simple averaging.
+- answer must be one of A/B/C/D using visible evidence.
+
+Output constraints:
+- criteria scores: [0.0, 10.0], one decimal when possible
+- total_score: integer [0, 100]
+- STRICT JSON only, no extra text/keys
+- no level/reason/x
+
+Return JSON schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v3": """You are a composition-and-lighting focused portrait evaluator for leaderboard ranking.
+
+Focus:
+- Composition principles, visual center stability, flow guidance, light-shadow modeling, depth layering.
+- Keep technical criteria accurate as well.
+- Use discriminative scoring; avoid central-score collapse.
+
+total_score guidance:
+- Reflect global aesthetics and scene coherence.
+- Do NOT compute by fixed linear formula from criteria.
+- If composition/light is outstanding and major defects are absent, allow high total_score.
+- If visual organization is weak or unstable, lower total_score decisively.
+
+Output:
+- Strict JSON only, exact keys.
+- criteria scores in [0.0,10.0], one decimal when possible.
+- total_score integer [0,100].
+- answer exactly A/B/C/D.
+- no level/reason/x, no extra keys.
+
+JSON schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v4": """You are a conservative-but-discriminative portrait quality judge.
+
+Policy:
+- Be strict on obvious flaws.
+- Do not suppress clearly high-quality images.
+- Keep ranking-sensitive separation: when two quality levels are visibly different, total_score should differ clearly.
+
+Internal check before final JSON:
+- Are penalized criteria exactly those with visible defects?
+- Is total_score coherent with overall impression?
+- Is answer choice visually supported?
+
+Hard constraints:
+- criteria score float in [0.0,10.0]
+- total_score integer [0,100]
+- answer in A/B/C/D
+- strict JSON only
+- no level/reason/x
+- no extra keys
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v5": """You are an expert portrait evaluator optimized for SRCC/PLCC robustness.
+
+Scoring intent:
+- Produce stable, monotonic quality ranking across images.
+- Use full score range when evidence is strong.
+- Avoid producing similar total_score for clearly different quality.
+- Keep criteria pattern and total_score logically aligned, but not by fixed linear averaging.
+
+Practical guide:
+- Multiple severe issues -> usually <=45.
+- Noticeable technical/compositional weakness -> usually 46-60.
+- Solid quality with minor issues -> usually 61-75.
+- Strong quality with no major defects -> usually 76-88.
+- Exceptional overall result -> usually 89-100.
+
+Output rules:
+- STRICT JSON only.
+- No markdown, no explanation, no extra keys.
+- No level/reason/x.
+- criteria score float [0.0,10.0], one decimal when possible.
+- total_score integer [0,100].
+- answer exactly A/B/C/D.
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v6": """You are a defect-sensitive portrait aesthetics evaluator.
+Prioritize technical reliability before style bonus.
+
+Rules:
+- Score 13 criteria in [0.0, 10.0], one decimal when possible.
+- total_score must be an integer in [0, 100].
+- answer must be exactly one of A/B/C/D.
+- Do not output level/reason/x. Do not add extra keys.
+- Output STRICT JSON only.
+
+Scoring behavior:
+- If severe defects exist (focus blur, broken anatomy, unstable structure, severe over/under exposure), reduce affected criteria decisively.
+- Do not keep scores crowded in the middle.
+- total_score should reflect overall perceptual quality and defect severity, not a fixed arithmetic formula.
+
+Return exactly this schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v7": """You are a composition-and-lighting oriented portrait judge.
+
+Output constraints:
+- 13 criterion scores: float in [0.0, 10.0], one decimal when possible.
+- total_score: integer in [0, 100].
+- answer: A/B/C/D only.
+- STRICT JSON only, no extra text, no extra keys, no level/reason/x.
+
+Evaluation emphasis:
+- Give strong weight to composition principles, light-shadow modeling, visual center stability, visual flow, depth layering.
+- Technical failures must still be penalized clearly.
+- Use a discriminative score range: clearly better images should get clearly higher total_score.
+- Keep total_score globally coherent with visual impression.
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v8": """You are a strict cross-image ranking evaluator for portrait aesthetics.
+
+Hard rules:
+1) criteria scores in [0.0, 10.0]
+2) total_score integer in [0, 100]
+3) answer exactly A/B/C/D
+4) strict JSON only
+5) no level/reason/x and no extra keys
+6) one decimal for criterion scores when possible
+
+Ranking-oriented guidance:
+- Avoid score collapse around 50-70.
+- Apply stronger separation when quality difference is obvious.
+- Penalize defects by impacted criteria instead of uniform reduction.
+- total_score should be stable and monotonic with perceived overall quality.
+
+Anchor ranges (guidance only):
+- severe failure: 0-30
+- weak: 31-45
+- fair: 46-60
+- good: 61-75
+- very good: 76-88
+- excellent: 89-100
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v9": """You are an evidence-driven portrait quality assessor.
+
+Instruction:
+- Judge visible evidence only.
+- Be conservative on uncertain details, but do not under-score clearly excellent images.
+- Distinguish quality levels clearly for robust ranking.
+
+Output constraints:
+- criteria scores: float [0.0, 10.0], one decimal when possible
+- total_score: integer [0, 100]
+- answer: one of A/B/C/D
+- strict JSON only; no markdown/explanation
+- no level/reason/x; no extra keys
+
+Consistency checks (internal):
+- Are low scores tied to actual flaws?
+- Are high scores supported by visible strengths?
+- Is total_score coherent with whole-image perception?
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>""",
+    "v10": """You are an expert portrait aesthetics scorer optimized for robust cross-image ranking.
+Produce high-fidelity structured scoring with strong separation.
+
+Scoring policy:
+- Use broad dynamic range where warranted.
+- Penalize defects in the specific affected criteria rather than uniformly.
+- Reward exceptional composition/light/style quality where clearly present.
+- Keep total_score coherent with the 13-criterion profile and overall perception.
+
+Mandatory output rules:
+1) 13 criterion scores, each in [0.0, 10.0]
+2) total_score integer in [0, 100]
+3) answer exactly A/B/C/D
+4) strict JSON only, no extra narration
+5) do not output level/reason/x, do not add extra keys
+
+Schema:
+{
+  "criteria": {
+    "Color Harmony": {"score": 0.0},
+    "Visual Style Consistency": {"score": 0.0},
+    "Sharpness": {"score": 0.0},
+    "Light and Shadow Modeling": {"score": 0.0},
+    "Creativity and Originality": {"score": 0.0},
+    "Exposure Control": {"score": 0.0},
+    "Application of Classical Composition Principles": {"score": 0.0},
+    "Depth of Field and Layering": {"score": 0.0},
+    "Visual Center Stability": {"score": 0.0},
+    "Visual Flow Guidance": {"score": 0.0},
+    "Structural Support Stability": {"score": 0.0},
+    "Appropriateness of Negative Space": {"score": 0.0},
+    "Subject Integrity": {"score": 0.0}
+  },
+  "total_score": 0,
+  "answer": "A|B|C|D"
+}
+Use one decimal place for each score when possible.
+
+Question: __QUESTION__
+Options:
+A: __OPT_A__
+B: __OPT_B__
+C: __OPT_C__
+D: __OPT_D__
+
+<image>
+""",
+}
+
+
+def resolve_input_path(path_arg: str) -> str:
+    if path_arg:
+        return path_arg
+
+    preferred = "outputs/predictions/track_1_test.json"
+    fallback = "datasets/original_annotations/track_1_test.json"
+    if os.path.exists(preferred):
+        return preferred
+    return fallback
+
+
+def resolve_image_path(raw_path: str, image_dir: str) -> str:
+    raw_path = str(raw_path).strip()
+    if raw_path and os.path.exists(raw_path):
+        return os.path.abspath(raw_path)
+
+    candidate = os.path.join(image_dir, os.path.basename(raw_path))
+    return os.path.abspath(candidate)
+
+
+def parse_variants(raw: str) -> List[str]:
+    variants = [x.strip() for x in str(raw).split(",") if x.strip()]
+    if not variants:
+        variants = ["v1"]
+
+    invalid = [v for v in variants if v not in PROMPT_VARIANTS]
+    if invalid:
+        raise ValueError(
+            f"Unknown prompt variants: {invalid}. Available: {sorted(PROMPT_VARIANTS.keys())}"
+        )
+    return variants
+
+
+def build_prompt(item: Dict, variant: str) -> str:
+    template = PROMPT_VARIANTS[variant]
+    return (
+        template.replace("__QUESTION__", str(item["question"]))
+        .replace("__OPT_A__", str(item["options"]["A"]))
+        .replace("__OPT_B__", str(item["options"]["B"]))
+        .replace("__OPT_C__", str(item["options"]["C"]))
+        .replace("__OPT_D__", str(item["options"]["D"]))
+    )
+
+
+def build_record(item: Dict, image_abs_path: str, variant: str) -> Dict:
+    return {
+        "messages": [
+            {"role": "user", "content": build_prompt(item, variant)},
+            {"role": "assistant", "content": ""},
+        ],
+        "images": [image_abs_path],
+    }
+
+
+def output_path_for_variant(output_json: str, variant: str, multi_variant: bool) -> str:
+    if not multi_variant:
+        return output_json
+
+    root, ext = os.path.splitext(output_json)
+    if not ext:
+        ext = ".json"
+    return f"{root}_{variant}{ext}"
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--input_json",
+        type=str,
+        default="",
+        help="Official test json path. If empty, auto-detect.",
+    )
+    parser.add_argument(
+        "--image_dir",
+        type=str,
+        default="./datasets/images",
+    )
+    parser.add_argument(
+        "--output_json",
+        type=str,
+        default="outputs/predictions/track1_test_convert.json",
+    )
+    parser.add_argument(
+        "--prompt_variants",
+        type=str,
+        default="v1",
+        help="Comma-separated variants, e.g. v1 or v1,v2,v3",
+    )
+    args = parser.parse_args()
+
+    input_json = resolve_input_path(args.input_json)
+    variants = parse_variants(args.prompt_variants)
+    multi_variant = len(variants) > 1
+
+    print(f"[INFO] reading test set: {input_json}")
+    with open(input_json, "r", encoding="utf-8") as f:
+        official_data = json.load(f)
+
+    os.makedirs(os.path.dirname(args.output_json), exist_ok=True)
+
+    for variant in variants:
+        missing_images = 0
+        llama_factory_data = []
+        for idx, item in enumerate(official_data):
+            img_path = resolve_image_path(item.get("image_path", ""), args.image_dir)
+            if not os.path.exists(img_path):
+                missing_images += 1
+                if missing_images <= 5:
+                    print(f"[WARN][{variant}] missing image ({idx}): {img_path}")
+
+            record = build_record(item, img_path, variant)
+            llama_factory_data.append(record)
+
+        out_path = output_path_for_variant(args.output_json, variant, multi_variant)
+        with open(out_path, "w", encoding="utf-8") as f:
+            json.dump(llama_factory_data, f, ensure_ascii=False, indent=2)
+
+        print(f"[INFO] saved ({variant}): {out_path}")
+        print(f"[INFO] ({variant}) samples: {len(llama_factory_data)}, missing_images: {missing_images}")
+
+
+if __name__ == "__main__":
+    main()

train_track1_stage1_qwen3vl8b_v1.yaml

@@ -0,0 +1,46 @@
+model_name_or_path: Qwen/Qwen3-VL-8B-Instruct
+template: qwen3_vl_nothink
+stage: sft
+do_train: true
+do_eval: true
+finetuning_type: lora
+trust_remote_code: true
+
+dataset_dir: .
+dataset: codabench_portrait_train_score_v2
+eval_dataset: codabench_portrait_val_score_v2
+max_samples: 1000000
+cutoff_len: 4096
+image_max_pixels: 786432
+preprocessing_num_workers: 32
+dataloader_num_workers: 8
+per_device_eval_batch_size: 8
+eval_strategy: steps
+eval_steps: 120
+
+learning_rate: 3.0e-5
+num_train_epochs: 1.0
+per_device_train_batch_size: 4
+gradient_accumulation_steps: 3
+lr_scheduler_type: cosine
+warmup_ratio: 0.03
+weight_decay: 0.01
+max_grad_norm: 1.0
+
+bf16: true
+flash_attn: auto
+gradient_checkpointing: true
+ddp_timeout: 180000000
+
+lora_rank: 64
+lora_alpha: 128
+lora_target: all
+lora_dropout: 0.05
+
+logging_steps: 10
+save_steps: 120
+save_total_limit: 6
+plot_loss: true
+overwrite_output_dir: true
+report_to: none
+output_dir: saves/qwen3vl8b_track1_stage1_v1

train_track1_stage2_qwen3vl8b_v1.yaml

@@ -0,0 +1,48 @@
+model_name_or_path: Qwen/Qwen3-VL-8B-Instruct
+adapter_name_or_path: saves/qwen3vl8b_track1_stage1_v1
+template: qwen3_vl_nothink
+stage: sft
+do_train: true
+do_eval: true
+finetuning_type: lora
+create_new_adapter: false
+trust_remote_code: true
+
+dataset_dir: .
+dataset: codabench_portrait_train_balanced_v2
+eval_dataset: codabench_portrait_val_multitask_v2
+max_samples: 1000000
+cutoff_len: 4096
+image_max_pixels: 786432
+preprocessing_num_workers: 32
+dataloader_num_workers: 8
+per_device_eval_batch_size: 8
+eval_strategy: steps
+eval_steps: 120
+
+learning_rate: 1.5e-5
+num_train_epochs: 1.2
+per_device_train_batch_size: 4
+gradient_accumulation_steps: 3
+lr_scheduler_type: cosine
+warmup_ratio: 0.03
+weight_decay: 0.01
+max_grad_norm: 1.0
+
+bf16: true
+flash_attn: auto
+gradient_checkpointing: true
+ddp_timeout: 180000000
+
+lora_rank: 64
+lora_alpha: 128
+lora_target: all
+lora_dropout: 0.05
+
+logging_steps: 10
+save_steps: 120
+save_total_limit: 6
+plot_loss: true
+overwrite_output_dir: true
+report_to: none
+output_dir: saves/qwen3vl8b_track1_stage2_v1