dev-strender's picture
feat: switch demo to FT + Opus article review (pipeline_exp_261021_ft_hybrid)
3dfc93b verified
Raw History Blame Contribute Delete
17.9 kB
"""`solar-eval runs eval` โ€” ์ƒˆ ์ถ”๋ก  ์—†์ด ๋””์Šคํฌ์˜ ๊ฒฐ๊ณผ๋งŒ ๋‹ค์‹œ ์ฑ„์ ํ•œ๋‹ค (๊ณ„ํš 2026-09-24-phase1a D11).
`runs.py` ์—์„œ ๋–ผ์–ด ๋ƒˆ๋‹ค(E1). ์˜ฎ๊ธฐ๋ฉด์„œ ๋ฐ”๋€ ๊ฒƒ ๋‘ ๊ฐ€์ง€:
- evaluation.json `eval_model` ์€ ์˜ˆ์ „์˜ `"local"` ์ด ์•„๋‹ˆ๋ผ ํ‰๊ฐ€๊ธฐ๊ฐ€ ๋ณด๊ณ ํ•˜๋Š” judge ๋ชจ๋ธ์ด๋‹ค
(judge ๊ฐ€ ์—†๋Š” lcs_diffยทrule_based ๋Š” null). ๋Ÿฌ๋„ˆ์™€ ๊ฐ™์€ ๊ทœ์น™์ด๋‹ค.
- run.json ์— `snapshot` ์ด ์žˆ์œผ๋ฉด `snapshot.config.evaluator_config` ๋ฅผ ์ƒˆ ํ‰๊ฐ€๊ธฐ๋กœ ๋‹ค์‹œ ์“ฐ๊ณ 
`config_fingerprint` ๋ฅผ ๋‹ค์‹œ ๊ณ„์‚ฐํ•˜๋ฉฐ `provenance.re_evaluated_at` ์„ ๋‚จ๊ธด๋‹ค โ€” ๋ฒˆ๋“ค๋กœ ๊ฐ์Œ€ ๋•Œ
ํ‰๊ฐ€ ํŒŒ์ผ๊ณผ ์Šค๋ƒ…์ƒท์ด ์–ด๊ธ‹๋‚˜์ง€ ์•Š๊ฒŒ(SPEC ยง4 "ํ‰๊ฐ€ ์ •์ฒด์„ฑ ์—ฐ๊ธฐ"์˜ ์ตœ์†Œ ๋Œ€์‘, Q14).
**๋‘ ๊ฒฝ๋กœ(๊ณ„ํš 2026-09-27 L6 ยง2.12 B4 ยท ๋งˆ์Šคํ„ฐ D-M2-6).** hub ๋Œ€์ƒ lcs_diff ๊ณ„์—ด์˜ ์™„๋ฃŒ๋œ run ์€ **run ๋””๋ ‰ํ† ๋ฆฌ๋ฅผ ๊ฑด๋“œ๋ฆฌ์ง€
์•Š๋Š”๋‹ค** โ€” ์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค์˜ ๊ฒฐ๊ณผ๋กœ ์ฑ„์ ํ•ด ํ‰๊ฐ€ ๋””๋ ‰ํ† ๋ฆฌ์— ๋‚จ๊ธฐ๊ณ  `purpose=rescore_after_fix` ์ฑ„์  ๋ฒˆ๋“ค๋กœ ๋ฐœํ–‰ํ•œ ๋’ค
(opt-in) hub ์— ์‹ฃ๋Š”๋‹ค(`solar_eval.scoring_publish`). ๊ทธ ๋ฐ–(hub ๋Œ€์ƒ ์•„๋‹˜ยท๋น„-lcsยท๋ฏธ์™„๋ฃŒ)์€ ์œ„์˜ ์˜ค๋Š˜ ๋™์ž‘(in-place) +
์•ˆ๋‚ด ํ•œ ์ค„์ด๋‹ค. hub ๋Œ€์ƒ run ์„ ์˜› ๋™์ž‘์œผ๋กœ ๋Œ๋ฆฌ๋˜ ํƒˆ์ถœ๊ตฌ ํ”Œ๋ž˜๊ทธ๋Š” L10 ์ปท์˜ค๋ฒ„์—์„œ ์ง€์› ๋‹ค(๊ณ„ํš 2026-09-28 L10 ยง2.6 ยท D-L10-7 โ€” ์ฃผ๋ฉด click ์‚ฌ์šฉ
์˜ค๋ฅ˜, ์ข…๋ฃŒ 2). ์ข…๋ฃŒ ์ฝ”๋“œ: ์ฑ„์ ์„ ๋งˆ์ณค์œผ๋ฉด 0(๋ฐœํ–‰ยทhub ์‹คํŒจ์™€ ๋ฌด๊ด€), `--strict-ingest`/
`SOLAR_EVAL_STRICT_INGEST=1` ์ด๊ณ  ์‹คํŒจ ๋‹ค๋ฆฌ๊ฐ€ ์žˆ์œผ๋ฉด 3, ์ƒˆ ๊ฒฝ๋กœ์—์„œ ๋ฒˆ๋“ค์ด ์—†๊ฑฐ๋‚˜ ์ฑ„์ ์„ ๋ชป ํ–ˆ์œผ๋ฉด 1.
**judge ๊ฒฝ๋กœ(๊ณ„ํš 2026-09-28 L12 ยง2.5 ยท D-L12-6ยท9ยท13).** ํƒœ์Šคํฌ ํ‰๊ฐ€๊ธฐ๊ฐ€ `rubric_judge` ์ธ hub ๋Œ€์ƒ ์™„๋ฃŒ run ์€ ๊ธฐ์ค€๋ณ„ judge ์ฑ„์ 
๋ฒˆ๋“ค์„ ๋ฐœํ–‰ํ•œ๋‹ค(`scoring_publish.judge`). `--purpose {primary,variance_check}` ์™€ `--retry-of <evaluation_run_id>` ๋Š” ์ด ๊ฒฝ๋กœ์—์„œ๋งŒ
๋ฐ›๋Š”๋‹ค(ํ•จ๊ป˜ ์“ฐ๊ฑฐ๋‚˜ ๋‹ค๋ฅธ ๊ฒฝ๋กœ์—์„œ ์“ฐ๋ฉด ์‚ฌ์šฉ ์˜ค๋ฅ˜, ์ข…๋ฃŒ 2). judge ์„ค์ •์€ ๊ฒฝ๋กœ ํŒ์ • ์ „์— ํ™•์ธํ•œ๋‹ค.
judge ๊ฐ€์ง€์˜ ์ข…๋ฃŒ ์ฝ”๋“œ: ๋ฐœํ–‰ ์‚ฌ์ „ ๊ฑฐ๋ถ€ 1 ยท retry ๋Œ€์ƒ ์—†์Œ 0 ยท ์ฑ„์  ๋’ค ์‹คํŒจ๋Š” ์œ„ ๊ทœ์น™ ๊ทธ๋Œ€๋กœ.
"""
from __future__ import annotations
import asyncio
from collections.abc import Mapping
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
import click
from solar_eval.cli.commands import run_meta as run_meta_module
from solar_eval.cli.commands.run_meta import PREP_ERRORS, find_task, load_run
from solar_eval.cli.formatters import format_scores
from solar_eval.core.run_records import (
build_eval_detail_docs,
build_eval_sample_from_result,
build_evaluation_doc,
)
from solar_eval.core.snapshot import self_check, with_evaluator
from solar_eval.hub_hook import exit_code_for
from solar_eval.hub_hook.optin import strict_requested
from solar_eval.hub_hook.types import LEG_PUBLISH, PUBLISH_NO_BUNDLE_PREFIX, STATUS_SKIPPED
from solar_eval.scoring_publish import FAMILY_JUDGE, EvalJob, Route, decide_route, publish_run_eval
from solar_eval.scoring_publish import judge as judge_publish
from solar_eval.scoring_publish.eval_dir import REPORT_SAMPLES
from solar_eval.scoring_publish.publish import PublishOutcome, Scored
#: `--no-ingest`ยท`--strict-ingest` ๋„์›€๋ง(๊ณ„ํš L6 ยง2.2 `runs eval` ํ–‰).
NO_INGEST_HELP = "์ƒˆ ๊ฒฝ๋กœ์—์„œ hub ๋‹ค๋ฆฌ(์ ์žฌ)๋งŒ ๊ฑด๋„ˆ๋›ด๋‹ค โ€” ํ‰๊ฐ€ ๋””๋ ‰ํ† ๋ฆฌยท์ฑ„์  ๋ฒˆ๋“ค ๋ฐœํ–‰์€ ํ•œ๋‹ค"
STRICT_INGEST_HELP = (
"์ƒˆ ๊ฒฝ๋กœ์˜ ๋ฒˆ๋“คยท๋ฐœํ–‰ยทhub ๋‹ค๋ฆฌ๊ฐ€ ํ•˜๋‚˜๋ผ๋„ ์‹คํŒจํ•˜๋ฉด ์ข…๋ฃŒ ์ฝ”๋“œ 3 (๊ธฐ๋ณธ 0). "
"SOLAR_EVAL_STRICT_INGEST=1 ๊ณผ ๊ฐ™๋‹ค"
)
#: judge ๊ฒฝ๋กœ ์˜ต์…˜ ๋„์›€๋ง(๊ณ„ํš 2026-09-28 L12 D-L12-9).
PURPOSE_HELP = (
"rubric judge ์ฑ„์  ๋ฒˆ๋“ค์˜ purpose. ๊ธฐ๋ณธ primary โ€” ๊ฐ™์€ ์„ค์ •์˜ primary ๊ฐ€ ์ด๋ฏธ ์žˆ์œผ๋ฉด ๊ฑฐ๋ถ€ํ•œ๋‹ค. "
"๊ฐ™์€ ์„ค์ •์„ ๋‹ค์‹œ ์ฑ„์ ํ•ด ํŒ์ • ๋ถ„์‚ฐ์„ ์žฌ๋ ค๋ฉด variance_check"
)
RETRY_OF_HELP = "์ด ํ‰๊ฐ€ ์‹คํ–‰(evaluation_run_id, ULID)์˜ error ํŒ์ •๋งŒ ๋‹ค์‹œ ๋ถ€๋ฅธ๋‹ค โ€” purpose=retry"
#: judge ๊ฒฝ๋กœ์—์„œ๋งŒ ๋ฐ›๋Š” ์˜ต์…˜ โ€” ํ•จ๊ป˜ ์“ฐ๊ฑฐ๋‚˜ ๋‹ค๋ฅธ ๊ฒฝ๋กœ์—์„œ ์“ฐ๋ฉด ์‚ฌ์šฉ ์˜ค๋ฅ˜(์ข…๋ฃŒ 2).
USAGE_PURPOSE_WITH_RETRY = "--purpose ์™€ --retry-of ๋Š” ํ•จ๊ป˜ ์“ธ ์ˆ˜ ์—†๋‹ค"
USAGE_JUDGE_ONLY = "--purpose/--retry-of ๋Š” rubric judge ์ฑ„์  ๋ฒˆ๋“ค ๊ฒฝ๋กœ์—์„œ๋งŒ ์“ด๋‹ค"
#: ์ด run ์ด judge ๊ฒฝ๋กœ๊ฐ€ ์•„๋‹Œ ์ด์œ (๊ฒฝ๋กœ ์•ˆ๋‚ด)๊ฐ€ ์žˆ์œผ๋ฉด ์‚ฌ์šฉ ์˜ค๋ฅ˜ ๋’ค์— ๋ถ™์ธ๋‹ค.
USAGE_NOT_JUDGE_REASON = " โ€” ์ด run ์€ judge ๊ฒฝ๋กœ๊ฐ€ ์•„๋‹ˆ๋‹ค: {notice}"
#: judge ๊ฐ€์ง€์—์„œ ์ฑ„์ ํ•˜์ง€ ๋ชปํ–ˆ์„ ๋•Œ โ€” ์ข…๋ฃŒ 1. ํ”Œ๋ž˜๊ทธ ๊ถŒ์œ ๋Š” ๊ทธ ํ”Œ๋ž˜๊ทธ๊ฐ€ ํ‘ธ๋Š” ์‹คํŒจ์—๋งŒ ๋ถ™์ธ๋‹ค(์•„๋ž˜ ๋‘˜).
JUDGE_NOT_SCORED = "์ฑ„์ ํ•˜์ง€ ์•Š์•˜๋‹ค โ€” ์œ„ ์š”์•ฝ์˜ ์‹คํŒจ ๋‹ค๋ฆฌ์™€ ๊ทธ ์•ˆ๋‚ด๋ฅผ ๋ณธ๋‹ค"
#: ๊ฐ™์€ ์„ค์ •์˜ primary ๊ฐ€ ์ด๋ฏธ ์žˆ๋‹ค(`publish_judge:primary_exists`) โ€” purposeยทretry ํ”Œ๋ž˜๊ทธ๊ฐ€ ํ‘ธ๋Š” ์œ ์ผํ•œ ์‹คํŒจ.
JUDGE_NOT_SCORED_PRIMARY_EXISTS = (
f"{JUDGE_NOT_SCORED}. ๊ฐ™์€ ์„ค์ •์„ ๋‹ค์‹œ ์ฑ„์ ํ•˜๋ ค๋ฉด --purpose variance_check, "
"error ๋งŒ ๋‹ค์‹œ ๋ถ€๋ฅด๋ ค๋ฉด --retry-of <evaluation_run_id>"
)
#: ์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค์ด ์—†๋‹ค(`publish_no_bundle:*`) โ€” ํ”Œ๋ž˜๊ทธ๋กœ๋Š” ํ’€๋ฆฌ์ง€ ์•Š๋Š”๋‹ค.
JUDGE_NOT_SCORED_NO_BUNDLE = f"{JUDGE_NOT_SCORED}. ์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค์ด ์—†๋‹ค โ€” ์š”์•ฝ์˜ ๋ฐฑํ•„ ๋ช…๋ น์œผ๋กœ ๋ถ€๋ฅ˜๋ฅผ ํ™•์ธํ•œ ๋’ค ๋‹ค์‹œ ๋Œ๋ฆฐ๋‹ค"
#: retry ๋Œ€์ƒ์ด ์—†์„ ๋•Œ(์ข…๋ฃŒ 0).
JUDGE_RETRY_NOTHING = "๋‹ค์‹œ ๋ถ€๋ฅผ error ํŒ์ •์ด ์—†๋‹ค โ€” ์› ํ‰๊ฐ€ ์‹คํ–‰์˜ error ๊ฐ€ ์ด๋ฏธ ๋ชจ๋‘ ํ’€๋ ธ๋‹ค"
def _ulid_option(ctx: click.Context, param: click.Parameter, value: str | None) -> str | None:
"""`--retry-of` ๋Š” ULID ๋ชจ์–‘๋งŒ ๋ฐ›๋Š”๋‹ค โ€” ํ‹€๋ฆฌ๋ฉด ์‚ฌ์šฉ ์˜ค๋ฅ˜(์ข…๋ฃŒ 2)."""
from poc_bundle.mirror import ULID_RE
if value is None or ULID_RE.fullmatch(value) is not None:
return value
raise click.BadParameter("evaluation_run_id ๋Š” ULID(26์ž Crockford base32)์—ฌ์•ผ ํ•œ๋‹ค")
async def _score_all(evaluator, judge_provider, results) -> tuple[list[dict], list[dict]]:
"""์ƒ˜ํ”Œ๋ณ„ ์ฑ„์ . ์ฑ„์  ์‹คํŒจ๋Š” ์ ์ˆ˜๊ฐ€ ์•„๋‹ˆ๋ผ ์‹คํŒจ๋กœ ๋‚จ๊ธด๋‹ค โ€” aggregate() ์—๋Š” ์„ฑ๊ณต๋ถ„๋งŒ ๋„˜๊ธด๋‹ค."""
eval_results: list[dict[str, Any]] = []
eval_failures: list[dict[str, Any]] = []
for result in results:
# ๋ฆฌ์ŠคํŠธ ์œ„์น˜๊ฐ€ ์•„๋‹ˆ๋ผ ์›๋ณธ sample_idx ๊ฐ€ ์ •๋ณธ์ด๋‹ค(F6).
sample_idx = result["sample_idx"]
try:
eval_sample = build_eval_sample_from_result(result, reference=result.get("golden"))
evaluator.validate_required_fields(eval_sample)
eval_result = await evaluator.evaluate(sample=eval_sample, provider=judge_provider)
eval_results.append({**eval_result, "sample_idx": sample_idx})
except Exception as e:
click.secho(f" Sample {sample_idx} eval failed: {e}", fg="yellow")
eval_failures.append({"sample_idx": sample_idx, "error": str(e)})
return eval_results, eval_failures
def _updated_snapshot(
run_meta: dict[str, Any], evaluator_config: dict[str, Any] | None, judge_provider: Any
) -> dict[str, Any] | None:
"""์Šค๋ƒ…์ƒท์ด ์žˆ์œผ๋ฉด ์ƒˆ ํ‰๊ฐ€๊ธฐ๋กœ ๋‹ค์‹œ ์“ด ์Šค๋ƒ…์ƒท, ์—†์œผ๋ฉด None(์˜› run ์— ์Šค๋ƒ…์ƒท์„ ์ƒˆ๋กœ ์“ฐ์ง€ ์•Š๋Š”๋‹ค).
์ฑ„์  **์ „์—** ๋ถ€๋ฅธ๋‹ค โ€” ๊ธฐ๋ก๋œ ์Šค๋ƒ…์ƒท์ด ๊นจ์กŒ๊ฑฐ๋‚˜ ์ƒˆ ํ‰๊ฐ€๊ธฐ๋กœ ์ง€๋ฌธ์„ ๋‹ค์‹œ ๋œฐ ์ˆ˜ ์—†์œผ๋ฉด ์œ ๋ฃŒ judge
ํ˜ธ์ถœ์„ ํ•˜๊ธฐ ์ „์— ๋ฉˆ์ถ˜๋‹ค. `judge_provider` ๋Š” ์ด๋ฒˆ ์žฌ์ฑ„์ ์ด ์‹ค์ œ๋กœ ์“ฐ๋Š” ๊ฒƒ์ด๋‹ค โ€” judge ๊ฐ€ ์—†๋˜ run ์„
llm_judge ๋กœ ๋‹ค์‹œ ์ฑ„์ ํ•˜๋ฉด ๊ทธ ํ”„๋กœ๋ฐ”์ด๋” ๋ชจ๋“ˆ์ด `runtime_impl_hash` ์— ๋“ค์–ด๊ฐ€์•ผ ํ•œ๋‹ค(D7).
`re_evaluated_at` ์€ ์ฑ„์ ์ด ๋๋‚œ ๋’ค `_stamp` ๊ฐ€ ๋ฎ์–ด์“ด๋‹ค.
"""
snapshot = run_meta.get("snapshot")
if snapshot is None:
return None
started = datetime.now(timezone.utc).isoformat()
try:
# ๊ธฐ๋ก๋œ ์Šค๋ƒ…์ƒท๋ถ€ํ„ฐ ๊ฒ€์‚ฌํ•œ๋‹ค โ€” `config` ๊ฐ€ ์—†๋Š” ์Šค๋ƒ…์ƒท์ด ๋งจ KeyError ๋กœ ์ฃฝ์ง€ ์•Š๊ฒŒ.
self_check(snapshot)
updated = with_evaluator(
snapshot, evaluator_config, re_evaluated_at=started, judge_provider=judge_provider
)
self_check(updated)
except PREP_ERRORS as e:
raise click.ClickException(f"cannot update the run snapshot's evaluator: {e}") from None
return updated
def _stamp(snapshot: dict[str, Any], at: str) -> dict[str, Any]:
"""`provenance.re_evaluated_at` ์„ ์ฑ„์ ์ด ๋๋‚œ ์‹œ๊ฐ์œผ๋กœ โ€” provenance ๋Š” ์ง€๋ฌธ ๋ฐ–์ด๋‹ค."""
return {**snapshot, "provenance": {**snapshot["provenance"], "re_evaluated_at": at}}
def _make_evaluator(evaluator_config: dict[str, Any] | None):
"""๋Ÿฌ๋„ˆยท์Šค๋ƒ…์ƒท๊ณผ ๊ฐ™์€ ๊ทœ์น™ โ€” ํ‚ค๊ฐ€ ์—†์„ ๋•Œ๋งŒ ๊ธฐ๋ณธ ํ‰๊ฐ€๊ธฐ. ์ž˜๋ชป๋œ ์„ค์ •์€ ์ฑ„์  ์ „์— ๋ฉˆ์ถ˜๋‹ค."""
from solar_eval.evaluators.registry import DEFAULT_EVALUATOR_CONFIG, create_evaluator
raw = DEFAULT_EVALUATOR_CONFIG if evaluator_config is None else evaluator_config
try:
return create_evaluator(raw)
except ValueError as e:
raise click.ClickException(f"invalid evaluator config: {e}") from None
async def _evaluate_run(store, run_id, run_meta, task_config, results) -> dict[str, Any]:
"""์ฑ„์  โ†’ evaluation.jsonยทeval_detailsยทrun.json(์ƒํƒœยท์Šค๋ƒ…์ƒท) ์ €์žฅ. ์ง‘๊ณ„๋ฅผ ๋Œ๋ ค์ค€๋‹ค."""
evaluator_config = task_config.get("evaluator")
evaluator = _make_evaluator(evaluator_config)
# judge provider ๋ฅผ ์•ˆ ๋„˜๊ธฐ๋ฉด llm_judge ๊ณ„์—ด์€ ์ „ ์ƒ˜ํ”Œ์—์„œ ์‹คํŒจํ•œ๋‹ค -- ๊ทธ๋ฆฌ๊ณ 
# ๊ทธ ์‹คํŒจ๊ฐ€ ์˜ˆ์ „์—” 0์  placeholder ๋กœ ๋ฎ์—ฌ ์กฐ์šฉํžˆ completed ๋กœ ๋๋‚ฌ๋‹ค(F8).
_, judge_provider = run_meta_module.make_providers(evaluator_config)
snapshot = _updated_snapshot(run_meta, evaluator_config, judge_provider)
eval_results, eval_failures = await _score_all(evaluator, judge_provider, results)
now = datetime.now(timezone.utc)
if snapshot is not None:
snapshot = _stamp(snapshot, now.isoformat())
aggregated = evaluator.aggregate(eval_results)
# eval_model = ํ‰๊ฐ€๊ธฐ๊ฐ€ ๋ณด๊ณ ํ•˜๋Š” judge ๋ชจ๋ธ(์—†์œผ๋ฉด None) โ€” ์˜ˆ์ „์˜ "local" ์€ ๊ฑฐ์ง“์ด์—ˆ๋‹ค(D11).
doc = build_evaluation_doc(
run_id, aggregated, eval_results, eval_failures, evaluator.judge_model
)
eval_id = await store.create_evaluation(doc)
await store.insert_eval_details(build_eval_detail_docs(eval_id, eval_results, eval_failures))
update: dict[str, Any] = {"status": "completed", "completed_at": now}
if snapshot is not None:
update["snapshot"] = snapshot
await store.update_run(run_id, update)
return {**aggregated, "overall_score": doc["overall_score"], "failed": len(eval_failures)}
def _report(aggregated: Mapping[str, Any], run_dir: Path) -> None:
click.echo()
click.secho("Evaluation completed!", fg="green", bold=True)
overall = aggregated.get("overall_score")
if overall is None:
# ์ฑ„์ ์ด ํ•œ ๊ฑด๋„ ์„ฑ๊ณตํ•˜์ง€ ๋ชปํ–ˆ๋‹ค -- ์ˆซ์ž๋ฅผ ์ฐ์œผ๋ฉด "0์ "์œผ๋กœ ์ฝํžŒ๋‹ค.
click.secho(" Overall Score: ์ธก์ • ์—†์Œ (์ „ ์ƒ˜ํ”Œ ์ฑ„์  ์‹คํŒจ)", fg="red", bold=True)
else:
click.echo(f" Overall Score: {overall:.4f}")
samples = aggregated.get(REPORT_SAMPLES)
if samples is not None:
click.echo(f" Samples: {samples} (์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค ๊ธฐ์ค€)")
failed = aggregated.get("failed", 0)
if failed:
click.secho(f" ์ฑ„์  ์‹คํŒจ: {failed}๊ฑด (score: null ๋กœ ๊ธฐ๋ก๋จ)", fg="yellow")
scores = aggregated.get("scores", {})
if scores:
click.echo(f" Scores: {format_scores(scores)}")
click.echo(f"\nResults saved to: {run_dir}/")
@click.command("eval")
@click.argument("project")
@click.option("--run-id", "-r", required=True, help="Run ID to evaluate")
@click.option("--no-ingest", is_flag=True, help=NO_INGEST_HELP)
@click.option("--strict-ingest", is_flag=True, help=STRICT_INGEST_HELP)
@click.option(
"--purpose", type=click.Choice(judge_publish.CLI_PURPOSES), default=None, help=PURPOSE_HELP
)
@click.option(
"--retry-of",
"retry_of",
default=None,
metavar="EVALUATION_RUN_ID",
callback=_ulid_option,
help=RETRY_OF_HELP,
)
@click.pass_context
def eval_run(
ctx: click.Context,
project: str,
run_id: str,
no_ingest: bool,
strict_ingest: bool,
purpose: str | None,
retry_of: str | None,
) -> None:
"""Run evaluation only on existing inference results (no new inference)."""
cfg = ctx.obj["config"]
if cfg.is_remote:
raise click.ClickException("Eval is local-mode only")
if purpose is not None and retry_of is not None:
raise click.UsageError(USAGE_PURPOSE_WITH_RETRY)
run_dir, store, run_meta = load_run(cfg, project, run_id)
results = store.load_existing_results()
if not results:
raise click.ClickException(f"No inference results found in {run_dir}/results.jsonl")
task = run_meta["task"]
click.secho(f"Evaluating run: {run_id}", fg="blue", bold=True)
_, task_config = find_task(cfg, project, task)
run_meta_module.preflight_judge_config(task_config)
route = decide_route(cfg, project, run_dir, run_meta, task_config)
if route.family != FAMILY_JUDGE and (purpose is not None or retry_of is not None):
raise click.UsageError(_judge_only_message(route))
if route.target is None:
click.secho(f" Task: {task}, Samples: {len(results)}", fg="blue")
_echo_route(route)
# ์ด์ „ ํ‰๊ฐ€๋ฅผ ๋ฏธ๋ฆฌ ์ง€์šฐ์ง€ ์•Š๋Š”๋‹ค(F3) -- ์žฌํ‰๊ฐ€๊ฐ€ ์‹คํŒจํ•ด๋„ ์˜› ์ ์ˆ˜๋Š” ๋‚จ๋Š”๋‹ค.
aggregated = asyncio.run(_evaluate_run(store, run_id, run_meta, task_config, results))
_report(aggregated, run_dir)
return
# ์ƒˆ ๊ฒฝ๋กœ์˜ ์ฑ„์  ์ž…๋ ฅ์€ ์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค ์‚ฌ๋ณธ์ด๋‹ค โ€” ์›๋ณธ ํ–‰ ์ˆ˜(์ค‘๋ณต ํฌํ•จ)๋Š” ์ฐ์ง€ ์•Š๊ณ  ์ฑ„์  ๋’ค ๋ณด๊ณ ์— ์‹ฃ๋Š”๋‹ค(๋ฆฌ๋ทฐ ์ง€์  4).
click.secho(f" Task: {task}, Samples: ์ฑ„ํƒ๋œ run ๋ฒˆ๋“ค ๊ธฐ์ค€(์•„๋ž˜ ๋ณด๊ณ )", fg="blue")
strict = strict_requested(strict_ingest)
judge_options = (purpose, retry_of)
scored, code = _publish(route, project, run_dir, task_config, no_ingest, strict, judge_options)
if scored is not None:
_report(scored.aggregated, scored.path)
if code:
ctx.exit(code)
def _judge_only_message(route: Route) -> str:
"""judge ์ „์šฉ ํ”Œ๋ž˜๊ทธ๋ฅผ ๋‹ค๋ฅธ ๊ฒฝ๋กœ์—์„œ ์“ด ์‚ฌ์šฉ ์˜ค๋ฅ˜ โ€” ๊ฒฝ๋กœ ์•ˆ๋‚ด(๋ฏธ์™„๋ฃŒ statusยท๋Œ€์ƒ ์•„๋‹˜ ๋“ฑ)๊ฐ€ ์žˆ์œผ๋ฉด ์ด์œ ๋กœ ๋ถ™์ธ๋‹ค."""
if route.notice is None:
return USAGE_JUDGE_ONLY
return USAGE_JUDGE_ONLY + USAGE_NOT_JUDGE_REASON.format(notice=route.notice)
def _echo_route(route: Route) -> None:
"""์˜› ๊ฒฝ๋กœ ์•ˆ๋‚ด ํ•œ ์ค„(standalone ์€ ์—†๋‹ค)."""
if route.notice is not None:
click.secho(f" โš  {route.notice}", fg="blue")
def _publish(
route: Route,
project: str,
run_dir: Path,
task_config: dict[str, Any],
no_ingest: bool,
strict: bool,
judge_options: tuple[str | None, str | None] = (None, None),
) -> tuple[Scored | None, int]:
"""์ƒˆ ๊ฒฝ๋กœ(`scoring_publish.publish_run_eval`) โ†’ (์ฑ„์ , ์ข…๋ฃŒ ์ฝ”๋“œ). ์ฑ„์ ์„ ๋ชป ํ–ˆ์œผ๋ฉด ClickException(์ข…๋ฃŒ 1).
`judge_options` = (`--purpose`, `--retry-of`) โ€” judge ๊ฒฝ๋กœ๋งŒ ์“ด๋‹ค. judge ๊ฐ€์ง€๋Š” `_judge_outcome` ์ด ์ข…๋ฃŒ ์ฝ”๋“œ๋ฅผ ์ •ํ•œ๋‹ค
(retry ๋Œ€์ƒ ์—†์Œ์ด๋ฉด ์ฑ„์  ์—†์ด ์ข…๋ฃŒ 0 โ€” ๊ทธ๋•Œ ์ฑ„์ ์€ None).
"""
evaluator_config = task_config.get("evaluator")
evaluator = _make_evaluator(evaluator_config) # ์ž˜๋ชป๋œ ์„ค์ •์€ ์ž ๊ธˆยท๋ฐฑํ•„ ์ „์— ๋ฉˆ์ถ˜๋‹ค
_, judge_provider = run_meta_module.make_providers(evaluator_config)
purpose, retry_of = judge_options
job = EvalJob(
project=project,
run_dir_name=run_dir.name,
evaluator=evaluator,
evaluator_config=evaluator_config,
judge_provider=judge_provider,
score_all=_score_all,
no_ingest=no_ingest,
strict=strict,
family=route.family,
purpose=purpose,
retry_of=retry_of,
)
outcome = publish_run_eval(route.target, job)
if route.family == FAMILY_JUDGE:
return _judge_outcome(outcome, strict)
if outcome.scored is None:
raise click.ClickException(
"์ฑ„์ ํ•˜์ง€ ์•Š์•˜๋‹ค โ€” ์œ„ ์š”์•ฝ์˜ ์‹คํŒจ ๋‹ค๋ฆฌ์™€ ๋˜์‚ด๋ฆฌ๊ธฐ๋ฅผ ๋ณธ๋‹ค(run ๋””๋ ‰ํ† ๋ฆฌ๋Š” ๊ทธ๋Œ€๋กœ๋‹ค)"
)
return outcome.scored, exit_code_for(outcome.result, strict)
def _judge_outcome(outcome: PublishOutcome, strict: bool) -> tuple[Scored | None, int]:
"""judge ๊ฐ€์ง€์˜ ์ข…๋ฃŒ ์ฝ”๋“œ(ยง2.5 ยท ๊ฒ€ํ†  F8). ์ฑ„์ ํ–ˆ์œผ๋ฉด L6b ๊ทœ์น™(0, strict ์ด๊ณ  ์‹คํŒจ ๋‹ค๋ฆฌ๋ฉด 3).
์ฑ„์ ํ•˜์ง€ ๋ชปํ–ˆ์œผ๋ฉด: retry ๋Œ€์ƒ ์—†์Œ โ†’ ์•ˆ๋‚ด ํ•œ ์ค„ ๋’ค (None, 0), ๊ทธ ๋ฐ–(๋ฐœํ–‰ ์‚ฌ์ „ ๊ฑฐ๋ถ€ยท๋ฒˆ๋“ค ์—†์Œยท์ž ๊ธˆ ๋“ฑ) โ†’ judge ์ „์šฉ ๋ฌธ๊ตฌ๋กœ
์ข…๋ฃŒ 1. purposeยทretry ํ”Œ๋ž˜๊ทธ๋Š” ๊ทธ๊ฒƒ์ด ํ‘ธ๋Š” ์‹คํŒจ(`primary_exists`)์—๋งŒ ๊ถŒํ•œ๋‹ค โ€” ๋ฒˆ๋“ค ์—†์Œยท์ž ๊ธˆ ์‹คํŒจ์—๋Š” ์š”์•ฝ์˜ ์•ˆ๋‚ด๋ฅผ ๊ฐ€๋ฆฌํ‚จ๋‹ค.
"""
if outcome.scored is not None:
return outcome.scored, exit_code_for(outcome.result, strict)
leg = outcome.result.legs.get(LEG_PUBLISH)
code = leg.code if leg is not None else None
if leg is not None and (leg.status, code) == (
STATUS_SKIPPED,
judge_publish.RETRY_NOTHING_TO_DO,
):
click.secho(f" {JUDGE_RETRY_NOTHING}", fg="blue")
return None, 0
raise click.ClickException(_judge_not_scored_message(code))
def _judge_not_scored_message(code: str | None) -> str:
"""๋ฐœํ–‰ ๋‹ค๋ฆฌ ์ฝ”๋“œ โ†’ ์ข…๋ฃŒ 1 ๋ฌธ๊ตฌ. ์ฝ”๋“œ ๋ฌธ์ž์—ด์€ `scoring_publish.judge`ยท`hub_hook.types` ์˜ ์ƒ์ˆ˜๋งŒ ์“ด๋‹ค."""
if code == f"{judge_publish.PUBLISH_JUDGE_PREFIX}{judge_publish.JUDGE_PRIMARY_EXISTS}":
return JUDGE_NOT_SCORED_PRIMARY_EXISTS
if isinstance(code, str) and code.startswith(PUBLISH_NO_BUNDLE_PREFIX):
return JUDGE_NOT_SCORED_NO_BUNDLE
return JUDGE_NOT_SCORED