Spaces:
Running
Running
Download solar_eval/cli/commands/run_eval.py from dev-strender/proofread-demo: direct link, hf CLI and curl.
- Browser
- Download file 17.9 kB
-
https://huggingface.co/spaces/dev-strender/proofread-demo/resolve/main/solar_eval/cli/commands/run_eval.py
- Command line
-
hf download hf://spaces/dev-strender/proofread-demo/solar_eval/cli/commands/run_eval.py
-
curl -L -o run_eval.py https://huggingface.co/spaces/dev-strender/proofread-demo/resolve/main/solar_eval/cli/commands/run_eval.py
17.9 kB
| """`solar-eval runs eval` โ ์ ์ถ๋ก ์์ด ๋์คํฌ์ ๊ฒฐ๊ณผ๋ง ๋ค์ ์ฑ์ ํ๋ค (๊ณํ 2026-09-24-phase1a D11). | |
| `runs.py` ์์ ๋ผ์ด ๋๋ค(E1). ์ฎ๊ธฐ๋ฉด์ ๋ฐ๋ ๊ฒ ๋ ๊ฐ์ง: | |
| - evaluation.json `eval_model` ์ ์์ ์ `"local"` ์ด ์๋๋ผ ํ๊ฐ๊ธฐ๊ฐ ๋ณด๊ณ ํ๋ judge ๋ชจ๋ธ์ด๋ค | |
| (judge ๊ฐ ์๋ lcs_diffยทrule_based ๋ null). ๋ฌ๋์ ๊ฐ์ ๊ท์น์ด๋ค. | |
| - run.json ์ `snapshot` ์ด ์์ผ๋ฉด `snapshot.config.evaluator_config` ๋ฅผ ์ ํ๊ฐ๊ธฐ๋ก ๋ค์ ์ฐ๊ณ | |
| `config_fingerprint` ๋ฅผ ๋ค์ ๊ณ์ฐํ๋ฉฐ `provenance.re_evaluated_at` ์ ๋จ๊ธด๋ค โ ๋ฒ๋ค๋ก ๊ฐ์ ๋ | |
| ํ๊ฐ ํ์ผ๊ณผ ์ค๋ ์ท์ด ์ด๊ธ๋์ง ์๊ฒ(SPEC ยง4 "ํ๊ฐ ์ ์ฒด์ฑ ์ฐ๊ธฐ"์ ์ต์ ๋์, Q14). | |
| **๋ ๊ฒฝ๋ก(๊ณํ 2026-09-27 L6 ยง2.12 B4 ยท ๋ง์คํฐ D-M2-6).** hub ๋์ lcs_diff ๊ณ์ด์ ์๋ฃ๋ run ์ **run ๋๋ ํ ๋ฆฌ๋ฅผ ๊ฑด๋๋ฆฌ์ง | |
| ์๋๋ค** โ ์ฑํ๋ run ๋ฒ๋ค์ ๊ฒฐ๊ณผ๋ก ์ฑ์ ํด ํ๊ฐ ๋๋ ํ ๋ฆฌ์ ๋จ๊ธฐ๊ณ `purpose=rescore_after_fix` ์ฑ์ ๋ฒ๋ค๋ก ๋ฐํํ ๋ค | |
| (opt-in) hub ์ ์ฃ๋๋ค(`solar_eval.scoring_publish`). ๊ทธ ๋ฐ(hub ๋์ ์๋ยท๋น-lcsยท๋ฏธ์๋ฃ)์ ์์ ์ค๋ ๋์(in-place) + | |
| ์๋ด ํ ์ค์ด๋ค. hub ๋์ run ์ ์ ๋์์ผ๋ก ๋๋ฆฌ๋ ํ์ถ๊ตฌ ํ๋๊ทธ๋ L10 ์ปท์ค๋ฒ์์ ์ง์ ๋ค(๊ณํ 2026-09-28 L10 ยง2.6 ยท D-L10-7 โ ์ฃผ๋ฉด click ์ฌ์ฉ | |
| ์ค๋ฅ, ์ข ๋ฃ 2). ์ข ๋ฃ ์ฝ๋: ์ฑ์ ์ ๋ง์ณค์ผ๋ฉด 0(๋ฐํยทhub ์คํจ์ ๋ฌด๊ด), `--strict-ingest`/ | |
| `SOLAR_EVAL_STRICT_INGEST=1` ์ด๊ณ ์คํจ ๋ค๋ฆฌ๊ฐ ์์ผ๋ฉด 3, ์ ๊ฒฝ๋ก์์ ๋ฒ๋ค์ด ์๊ฑฐ๋ ์ฑ์ ์ ๋ชป ํ์ผ๋ฉด 1. | |
| **judge ๊ฒฝ๋ก(๊ณํ 2026-09-28 L12 ยง2.5 ยท D-L12-6ยท9ยท13).** ํ์คํฌ ํ๊ฐ๊ธฐ๊ฐ `rubric_judge` ์ธ hub ๋์ ์๋ฃ run ์ ๊ธฐ์ค๋ณ judge ์ฑ์ | |
| ๋ฒ๋ค์ ๋ฐํํ๋ค(`scoring_publish.judge`). `--purpose {primary,variance_check}` ์ `--retry-of <evaluation_run_id>` ๋ ์ด ๊ฒฝ๋ก์์๋ง | |
| ๋ฐ๋๋ค(ํจ๊ป ์ฐ๊ฑฐ๋ ๋ค๋ฅธ ๊ฒฝ๋ก์์ ์ฐ๋ฉด ์ฌ์ฉ ์ค๋ฅ, ์ข ๋ฃ 2). judge ์ค์ ์ ๊ฒฝ๋ก ํ์ ์ ์ ํ์ธํ๋ค. | |
| judge ๊ฐ์ง์ ์ข ๋ฃ ์ฝ๋: ๋ฐํ ์ฌ์ ๊ฑฐ๋ถ 1 ยท retry ๋์ ์์ 0 ยท ์ฑ์ ๋ค ์คํจ๋ ์ ๊ท์น ๊ทธ๋๋ก. | |
| """ | |
| from __future__ import annotations | |
| import asyncio | |
| from collections.abc import Mapping | |
| from datetime import datetime, timezone | |
| from pathlib import Path | |
| from typing import Any | |
| import click | |
| from solar_eval.cli.commands import run_meta as run_meta_module | |
| from solar_eval.cli.commands.run_meta import PREP_ERRORS, find_task, load_run | |
| from solar_eval.cli.formatters import format_scores | |
| from solar_eval.core.run_records import ( | |
| build_eval_detail_docs, | |
| build_eval_sample_from_result, | |
| build_evaluation_doc, | |
| ) | |
| from solar_eval.core.snapshot import self_check, with_evaluator | |
| from solar_eval.hub_hook import exit_code_for | |
| from solar_eval.hub_hook.optin import strict_requested | |
| from solar_eval.hub_hook.types import LEG_PUBLISH, PUBLISH_NO_BUNDLE_PREFIX, STATUS_SKIPPED | |
| from solar_eval.scoring_publish import FAMILY_JUDGE, EvalJob, Route, decide_route, publish_run_eval | |
| from solar_eval.scoring_publish import judge as judge_publish | |
| from solar_eval.scoring_publish.eval_dir import REPORT_SAMPLES | |
| from solar_eval.scoring_publish.publish import PublishOutcome, Scored | |
| #: `--no-ingest`ยท`--strict-ingest` ๋์๋ง(๊ณํ L6 ยง2.2 `runs eval` ํ). | |
| NO_INGEST_HELP = "์ ๊ฒฝ๋ก์์ hub ๋ค๋ฆฌ(์ ์ฌ)๋ง ๊ฑด๋๋ด๋ค โ ํ๊ฐ ๋๋ ํ ๋ฆฌยท์ฑ์ ๋ฒ๋ค ๋ฐํ์ ํ๋ค" | |
| STRICT_INGEST_HELP = ( | |
| "์ ๊ฒฝ๋ก์ ๋ฒ๋คยท๋ฐํยทhub ๋ค๋ฆฌ๊ฐ ํ๋๋ผ๋ ์คํจํ๋ฉด ์ข ๋ฃ ์ฝ๋ 3 (๊ธฐ๋ณธ 0). " | |
| "SOLAR_EVAL_STRICT_INGEST=1 ๊ณผ ๊ฐ๋ค" | |
| ) | |
| #: judge ๊ฒฝ๋ก ์ต์ ๋์๋ง(๊ณํ 2026-09-28 L12 D-L12-9). | |
| PURPOSE_HELP = ( | |
| "rubric judge ์ฑ์ ๋ฒ๋ค์ purpose. ๊ธฐ๋ณธ primary โ ๊ฐ์ ์ค์ ์ primary ๊ฐ ์ด๋ฏธ ์์ผ๋ฉด ๊ฑฐ๋ถํ๋ค. " | |
| "๊ฐ์ ์ค์ ์ ๋ค์ ์ฑ์ ํด ํ์ ๋ถ์ฐ์ ์ฌ๋ ค๋ฉด variance_check" | |
| ) | |
| RETRY_OF_HELP = "์ด ํ๊ฐ ์คํ(evaluation_run_id, ULID)์ error ํ์ ๋ง ๋ค์ ๋ถ๋ฅธ๋ค โ purpose=retry" | |
| #: judge ๊ฒฝ๋ก์์๋ง ๋ฐ๋ ์ต์ โ ํจ๊ป ์ฐ๊ฑฐ๋ ๋ค๋ฅธ ๊ฒฝ๋ก์์ ์ฐ๋ฉด ์ฌ์ฉ ์ค๋ฅ(์ข ๋ฃ 2). | |
| USAGE_PURPOSE_WITH_RETRY = "--purpose ์ --retry-of ๋ ํจ๊ป ์ธ ์ ์๋ค" | |
| USAGE_JUDGE_ONLY = "--purpose/--retry-of ๋ rubric judge ์ฑ์ ๋ฒ๋ค ๊ฒฝ๋ก์์๋ง ์ด๋ค" | |
| #: ์ด run ์ด judge ๊ฒฝ๋ก๊ฐ ์๋ ์ด์ (๊ฒฝ๋ก ์๋ด)๊ฐ ์์ผ๋ฉด ์ฌ์ฉ ์ค๋ฅ ๋ค์ ๋ถ์ธ๋ค. | |
| USAGE_NOT_JUDGE_REASON = " โ ์ด run ์ judge ๊ฒฝ๋ก๊ฐ ์๋๋ค: {notice}" | |
| #: judge ๊ฐ์ง์์ ์ฑ์ ํ์ง ๋ชปํ์ ๋ โ ์ข ๋ฃ 1. ํ๋๊ทธ ๊ถ์ ๋ ๊ทธ ํ๋๊ทธ๊ฐ ํธ๋ ์คํจ์๋ง ๋ถ์ธ๋ค(์๋ ๋). | |
| JUDGE_NOT_SCORED = "์ฑ์ ํ์ง ์์๋ค โ ์ ์์ฝ์ ์คํจ ๋ค๋ฆฌ์ ๊ทธ ์๋ด๋ฅผ ๋ณธ๋ค" | |
| #: ๊ฐ์ ์ค์ ์ primary ๊ฐ ์ด๋ฏธ ์๋ค(`publish_judge:primary_exists`) โ purposeยทretry ํ๋๊ทธ๊ฐ ํธ๋ ์ ์ผํ ์คํจ. | |
| JUDGE_NOT_SCORED_PRIMARY_EXISTS = ( | |
| f"{JUDGE_NOT_SCORED}. ๊ฐ์ ์ค์ ์ ๋ค์ ์ฑ์ ํ๋ ค๋ฉด --purpose variance_check, " | |
| "error ๋ง ๋ค์ ๋ถ๋ฅด๋ ค๋ฉด --retry-of <evaluation_run_id>" | |
| ) | |
| #: ์ฑํ๋ run ๋ฒ๋ค์ด ์๋ค(`publish_no_bundle:*`) โ ํ๋๊ทธ๋ก๋ ํ๋ฆฌ์ง ์๋๋ค. | |
| JUDGE_NOT_SCORED_NO_BUNDLE = f"{JUDGE_NOT_SCORED}. ์ฑํ๋ run ๋ฒ๋ค์ด ์๋ค โ ์์ฝ์ ๋ฐฑํ ๋ช ๋ น์ผ๋ก ๋ถ๋ฅ๋ฅผ ํ์ธํ ๋ค ๋ค์ ๋๋ฆฐ๋ค" | |
| #: retry ๋์์ด ์์ ๋(์ข ๋ฃ 0). | |
| JUDGE_RETRY_NOTHING = "๋ค์ ๋ถ๋ฅผ error ํ์ ์ด ์๋ค โ ์ ํ๊ฐ ์คํ์ error ๊ฐ ์ด๋ฏธ ๋ชจ๋ ํ๋ ธ๋ค" | |
| def _ulid_option(ctx: click.Context, param: click.Parameter, value: str | None) -> str | None: | |
| """`--retry-of` ๋ ULID ๋ชจ์๋ง ๋ฐ๋๋ค โ ํ๋ฆฌ๋ฉด ์ฌ์ฉ ์ค๋ฅ(์ข ๋ฃ 2).""" | |
| from poc_bundle.mirror import ULID_RE | |
| if value is None or ULID_RE.fullmatch(value) is not None: | |
| return value | |
| raise click.BadParameter("evaluation_run_id ๋ ULID(26์ Crockford base32)์ฌ์ผ ํ๋ค") | |
| async def _score_all(evaluator, judge_provider, results) -> tuple[list[dict], list[dict]]: | |
| """์ํ๋ณ ์ฑ์ . ์ฑ์ ์คํจ๋ ์ ์๊ฐ ์๋๋ผ ์คํจ๋ก ๋จ๊ธด๋ค โ aggregate() ์๋ ์ฑ๊ณต๋ถ๋ง ๋๊ธด๋ค.""" | |
| eval_results: list[dict[str, Any]] = [] | |
| eval_failures: list[dict[str, Any]] = [] | |
| for result in results: | |
| # ๋ฆฌ์คํธ ์์น๊ฐ ์๋๋ผ ์๋ณธ sample_idx ๊ฐ ์ ๋ณธ์ด๋ค(F6). | |
| sample_idx = result["sample_idx"] | |
| try: | |
| eval_sample = build_eval_sample_from_result(result, reference=result.get("golden")) | |
| evaluator.validate_required_fields(eval_sample) | |
| eval_result = await evaluator.evaluate(sample=eval_sample, provider=judge_provider) | |
| eval_results.append({**eval_result, "sample_idx": sample_idx}) | |
| except Exception as e: | |
| click.secho(f" Sample {sample_idx} eval failed: {e}", fg="yellow") | |
| eval_failures.append({"sample_idx": sample_idx, "error": str(e)}) | |
| return eval_results, eval_failures | |
| def _updated_snapshot( | |
| run_meta: dict[str, Any], evaluator_config: dict[str, Any] | None, judge_provider: Any | |
| ) -> dict[str, Any] | None: | |
| """์ค๋ ์ท์ด ์์ผ๋ฉด ์ ํ๊ฐ๊ธฐ๋ก ๋ค์ ์ด ์ค๋ ์ท, ์์ผ๋ฉด None(์ run ์ ์ค๋ ์ท์ ์๋ก ์ฐ์ง ์๋๋ค). | |
| ์ฑ์ **์ ์** ๋ถ๋ฅธ๋ค โ ๊ธฐ๋ก๋ ์ค๋ ์ท์ด ๊นจ์ก๊ฑฐ๋ ์ ํ๊ฐ๊ธฐ๋ก ์ง๋ฌธ์ ๋ค์ ๋ฐ ์ ์์ผ๋ฉด ์ ๋ฃ judge | |
| ํธ์ถ์ ํ๊ธฐ ์ ์ ๋ฉ์ถ๋ค. `judge_provider` ๋ ์ด๋ฒ ์ฌ์ฑ์ ์ด ์ค์ ๋ก ์ฐ๋ ๊ฒ์ด๋ค โ judge ๊ฐ ์๋ run ์ | |
| llm_judge ๋ก ๋ค์ ์ฑ์ ํ๋ฉด ๊ทธ ํ๋ก๋ฐ์ด๋ ๋ชจ๋์ด `runtime_impl_hash` ์ ๋ค์ด๊ฐ์ผ ํ๋ค(D7). | |
| `re_evaluated_at` ์ ์ฑ์ ์ด ๋๋ ๋ค `_stamp` ๊ฐ ๋ฎ์ด์ด๋ค. | |
| """ | |
| snapshot = run_meta.get("snapshot") | |
| if snapshot is None: | |
| return None | |
| started = datetime.now(timezone.utc).isoformat() | |
| try: | |
| # ๊ธฐ๋ก๋ ์ค๋ ์ท๋ถํฐ ๊ฒ์ฌํ๋ค โ `config` ๊ฐ ์๋ ์ค๋ ์ท์ด ๋งจ KeyError ๋ก ์ฃฝ์ง ์๊ฒ. | |
| self_check(snapshot) | |
| updated = with_evaluator( | |
| snapshot, evaluator_config, re_evaluated_at=started, judge_provider=judge_provider | |
| ) | |
| self_check(updated) | |
| except PREP_ERRORS as e: | |
| raise click.ClickException(f"cannot update the run snapshot's evaluator: {e}") from None | |
| return updated | |
| def _stamp(snapshot: dict[str, Any], at: str) -> dict[str, Any]: | |
| """`provenance.re_evaluated_at` ์ ์ฑ์ ์ด ๋๋ ์๊ฐ์ผ๋ก โ provenance ๋ ์ง๋ฌธ ๋ฐ์ด๋ค.""" | |
| return {**snapshot, "provenance": {**snapshot["provenance"], "re_evaluated_at": at}} | |
| def _make_evaluator(evaluator_config: dict[str, Any] | None): | |
| """๋ฌ๋ยท์ค๋ ์ท๊ณผ ๊ฐ์ ๊ท์น โ ํค๊ฐ ์์ ๋๋ง ๊ธฐ๋ณธ ํ๊ฐ๊ธฐ. ์๋ชป๋ ์ค์ ์ ์ฑ์ ์ ์ ๋ฉ์ถ๋ค.""" | |
| from solar_eval.evaluators.registry import DEFAULT_EVALUATOR_CONFIG, create_evaluator | |
| raw = DEFAULT_EVALUATOR_CONFIG if evaluator_config is None else evaluator_config | |
| try: | |
| return create_evaluator(raw) | |
| except ValueError as e: | |
| raise click.ClickException(f"invalid evaluator config: {e}") from None | |
| async def _evaluate_run(store, run_id, run_meta, task_config, results) -> dict[str, Any]: | |
| """์ฑ์ โ evaluation.jsonยทeval_detailsยทrun.json(์ํยท์ค๋ ์ท) ์ ์ฅ. ์ง๊ณ๋ฅผ ๋๋ ค์ค๋ค.""" | |
| evaluator_config = task_config.get("evaluator") | |
| evaluator = _make_evaluator(evaluator_config) | |
| # judge provider ๋ฅผ ์ ๋๊ธฐ๋ฉด llm_judge ๊ณ์ด์ ์ ์ํ์์ ์คํจํ๋ค -- ๊ทธ๋ฆฌ๊ณ | |
| # ๊ทธ ์คํจ๊ฐ ์์ ์ 0์ placeholder ๋ก ๋ฎ์ฌ ์กฐ์ฉํ completed ๋ก ๋๋ฌ๋ค(F8). | |
| _, judge_provider = run_meta_module.make_providers(evaluator_config) | |
| snapshot = _updated_snapshot(run_meta, evaluator_config, judge_provider) | |
| eval_results, eval_failures = await _score_all(evaluator, judge_provider, results) | |
| now = datetime.now(timezone.utc) | |
| if snapshot is not None: | |
| snapshot = _stamp(snapshot, now.isoformat()) | |
| aggregated = evaluator.aggregate(eval_results) | |
| # eval_model = ํ๊ฐ๊ธฐ๊ฐ ๋ณด๊ณ ํ๋ judge ๋ชจ๋ธ(์์ผ๋ฉด None) โ ์์ ์ "local" ์ ๊ฑฐ์ง์ด์๋ค(D11). | |
| doc = build_evaluation_doc( | |
| run_id, aggregated, eval_results, eval_failures, evaluator.judge_model | |
| ) | |
| eval_id = await store.create_evaluation(doc) | |
| await store.insert_eval_details(build_eval_detail_docs(eval_id, eval_results, eval_failures)) | |
| update: dict[str, Any] = {"status": "completed", "completed_at": now} | |
| if snapshot is not None: | |
| update["snapshot"] = snapshot | |
| await store.update_run(run_id, update) | |
| return {**aggregated, "overall_score": doc["overall_score"], "failed": len(eval_failures)} | |
| def _report(aggregated: Mapping[str, Any], run_dir: Path) -> None: | |
| click.echo() | |
| click.secho("Evaluation completed!", fg="green", bold=True) | |
| overall = aggregated.get("overall_score") | |
| if overall is None: | |
| # ์ฑ์ ์ด ํ ๊ฑด๋ ์ฑ๊ณตํ์ง ๋ชปํ๋ค -- ์ซ์๋ฅผ ์ฐ์ผ๋ฉด "0์ "์ผ๋ก ์ฝํ๋ค. | |
| click.secho(" Overall Score: ์ธก์ ์์ (์ ์ํ ์ฑ์ ์คํจ)", fg="red", bold=True) | |
| else: | |
| click.echo(f" Overall Score: {overall:.4f}") | |
| samples = aggregated.get(REPORT_SAMPLES) | |
| if samples is not None: | |
| click.echo(f" Samples: {samples} (์ฑํ๋ run ๋ฒ๋ค ๊ธฐ์ค)") | |
| failed = aggregated.get("failed", 0) | |
| if failed: | |
| click.secho(f" ์ฑ์ ์คํจ: {failed}๊ฑด (score: null ๋ก ๊ธฐ๋ก๋จ)", fg="yellow") | |
| scores = aggregated.get("scores", {}) | |
| if scores: | |
| click.echo(f" Scores: {format_scores(scores)}") | |
| click.echo(f"\nResults saved to: {run_dir}/") | |
| def eval_run( | |
| ctx: click.Context, | |
| project: str, | |
| run_id: str, | |
| no_ingest: bool, | |
| strict_ingest: bool, | |
| purpose: str | None, | |
| retry_of: str | None, | |
| ) -> None: | |
| """Run evaluation only on existing inference results (no new inference).""" | |
| cfg = ctx.obj["config"] | |
| if cfg.is_remote: | |
| raise click.ClickException("Eval is local-mode only") | |
| if purpose is not None and retry_of is not None: | |
| raise click.UsageError(USAGE_PURPOSE_WITH_RETRY) | |
| run_dir, store, run_meta = load_run(cfg, project, run_id) | |
| results = store.load_existing_results() | |
| if not results: | |
| raise click.ClickException(f"No inference results found in {run_dir}/results.jsonl") | |
| task = run_meta["task"] | |
| click.secho(f"Evaluating run: {run_id}", fg="blue", bold=True) | |
| _, task_config = find_task(cfg, project, task) | |
| run_meta_module.preflight_judge_config(task_config) | |
| route = decide_route(cfg, project, run_dir, run_meta, task_config) | |
| if route.family != FAMILY_JUDGE and (purpose is not None or retry_of is not None): | |
| raise click.UsageError(_judge_only_message(route)) | |
| if route.target is None: | |
| click.secho(f" Task: {task}, Samples: {len(results)}", fg="blue") | |
| _echo_route(route) | |
| # ์ด์ ํ๊ฐ๋ฅผ ๋ฏธ๋ฆฌ ์ง์ฐ์ง ์๋๋ค(F3) -- ์ฌํ๊ฐ๊ฐ ์คํจํด๋ ์ ์ ์๋ ๋จ๋๋ค. | |
| aggregated = asyncio.run(_evaluate_run(store, run_id, run_meta, task_config, results)) | |
| _report(aggregated, run_dir) | |
| return | |
| # ์ ๊ฒฝ๋ก์ ์ฑ์ ์ ๋ ฅ์ ์ฑํ๋ run ๋ฒ๋ค ์ฌ๋ณธ์ด๋ค โ ์๋ณธ ํ ์(์ค๋ณต ํฌํจ)๋ ์ฐ์ง ์๊ณ ์ฑ์ ๋ค ๋ณด๊ณ ์ ์ฃ๋๋ค(๋ฆฌ๋ทฐ ์ง์ 4). | |
| click.secho(f" Task: {task}, Samples: ์ฑํ๋ run ๋ฒ๋ค ๊ธฐ์ค(์๋ ๋ณด๊ณ )", fg="blue") | |
| strict = strict_requested(strict_ingest) | |
| judge_options = (purpose, retry_of) | |
| scored, code = _publish(route, project, run_dir, task_config, no_ingest, strict, judge_options) | |
| if scored is not None: | |
| _report(scored.aggregated, scored.path) | |
| if code: | |
| ctx.exit(code) | |
| def _judge_only_message(route: Route) -> str: | |
| """judge ์ ์ฉ ํ๋๊ทธ๋ฅผ ๋ค๋ฅธ ๊ฒฝ๋ก์์ ์ด ์ฌ์ฉ ์ค๋ฅ โ ๊ฒฝ๋ก ์๋ด(๋ฏธ์๋ฃ statusยท๋์ ์๋ ๋ฑ)๊ฐ ์์ผ๋ฉด ์ด์ ๋ก ๋ถ์ธ๋ค.""" | |
| if route.notice is None: | |
| return USAGE_JUDGE_ONLY | |
| return USAGE_JUDGE_ONLY + USAGE_NOT_JUDGE_REASON.format(notice=route.notice) | |
| def _echo_route(route: Route) -> None: | |
| """์ ๊ฒฝ๋ก ์๋ด ํ ์ค(standalone ์ ์๋ค).""" | |
| if route.notice is not None: | |
| click.secho(f" โ {route.notice}", fg="blue") | |
| def _publish( | |
| route: Route, | |
| project: str, | |
| run_dir: Path, | |
| task_config: dict[str, Any], | |
| no_ingest: bool, | |
| strict: bool, | |
| judge_options: tuple[str | None, str | None] = (None, None), | |
| ) -> tuple[Scored | None, int]: | |
| """์ ๊ฒฝ๋ก(`scoring_publish.publish_run_eval`) โ (์ฑ์ , ์ข ๋ฃ ์ฝ๋). ์ฑ์ ์ ๋ชป ํ์ผ๋ฉด ClickException(์ข ๋ฃ 1). | |
| `judge_options` = (`--purpose`, `--retry-of`) โ judge ๊ฒฝ๋ก๋ง ์ด๋ค. judge ๊ฐ์ง๋ `_judge_outcome` ์ด ์ข ๋ฃ ์ฝ๋๋ฅผ ์ ํ๋ค | |
| (retry ๋์ ์์์ด๋ฉด ์ฑ์ ์์ด ์ข ๋ฃ 0 โ ๊ทธ๋ ์ฑ์ ์ None). | |
| """ | |
| evaluator_config = task_config.get("evaluator") | |
| evaluator = _make_evaluator(evaluator_config) # ์๋ชป๋ ์ค์ ์ ์ ๊ธยท๋ฐฑํ ์ ์ ๋ฉ์ถ๋ค | |
| _, judge_provider = run_meta_module.make_providers(evaluator_config) | |
| purpose, retry_of = judge_options | |
| job = EvalJob( | |
| project=project, | |
| run_dir_name=run_dir.name, | |
| evaluator=evaluator, | |
| evaluator_config=evaluator_config, | |
| judge_provider=judge_provider, | |
| score_all=_score_all, | |
| no_ingest=no_ingest, | |
| strict=strict, | |
| family=route.family, | |
| purpose=purpose, | |
| retry_of=retry_of, | |
| ) | |
| outcome = publish_run_eval(route.target, job) | |
| if route.family == FAMILY_JUDGE: | |
| return _judge_outcome(outcome, strict) | |
| if outcome.scored is None: | |
| raise click.ClickException( | |
| "์ฑ์ ํ์ง ์์๋ค โ ์ ์์ฝ์ ์คํจ ๋ค๋ฆฌ์ ๋์ด๋ฆฌ๊ธฐ๋ฅผ ๋ณธ๋ค(run ๋๋ ํ ๋ฆฌ๋ ๊ทธ๋๋ก๋ค)" | |
| ) | |
| return outcome.scored, exit_code_for(outcome.result, strict) | |
| def _judge_outcome(outcome: PublishOutcome, strict: bool) -> tuple[Scored | None, int]: | |
| """judge ๊ฐ์ง์ ์ข ๋ฃ ์ฝ๋(ยง2.5 ยท ๊ฒํ F8). ์ฑ์ ํ์ผ๋ฉด L6b ๊ท์น(0, strict ์ด๊ณ ์คํจ ๋ค๋ฆฌ๋ฉด 3). | |
| ์ฑ์ ํ์ง ๋ชปํ์ผ๋ฉด: retry ๋์ ์์ โ ์๋ด ํ ์ค ๋ค (None, 0), ๊ทธ ๋ฐ(๋ฐํ ์ฌ์ ๊ฑฐ๋ถยท๋ฒ๋ค ์์ยท์ ๊ธ ๋ฑ) โ judge ์ ์ฉ ๋ฌธ๊ตฌ๋ก | |
| ์ข ๋ฃ 1. purposeยทretry ํ๋๊ทธ๋ ๊ทธ๊ฒ์ด ํธ๋ ์คํจ(`primary_exists`)์๋ง ๊ถํ๋ค โ ๋ฒ๋ค ์์ยท์ ๊ธ ์คํจ์๋ ์์ฝ์ ์๋ด๋ฅผ ๊ฐ๋ฆฌํจ๋ค. | |
| """ | |
| if outcome.scored is not None: | |
| return outcome.scored, exit_code_for(outcome.result, strict) | |
| leg = outcome.result.legs.get(LEG_PUBLISH) | |
| code = leg.code if leg is not None else None | |
| if leg is not None and (leg.status, code) == ( | |
| STATUS_SKIPPED, | |
| judge_publish.RETRY_NOTHING_TO_DO, | |
| ): | |
| click.secho(f" {JUDGE_RETRY_NOTHING}", fg="blue") | |
| return None, 0 | |
| raise click.ClickException(_judge_not_scored_message(code)) | |
| def _judge_not_scored_message(code: str | None) -> str: | |
| """๋ฐํ ๋ค๋ฆฌ ์ฝ๋ โ ์ข ๋ฃ 1 ๋ฌธ๊ตฌ. ์ฝ๋ ๋ฌธ์์ด์ `scoring_publish.judge`ยท`hub_hook.types` ์ ์์๋ง ์ด๋ค.""" | |
| if code == f"{judge_publish.PUBLISH_JUDGE_PREFIX}{judge_publish.JUDGE_PRIMARY_EXISTS}": | |
| return JUDGE_NOT_SCORED_PRIMARY_EXISTS | |
| if isinstance(code, str) and code.startswith(PUBLISH_NO_BUNDLE_PREFIX): | |
| return JUDGE_NOT_SCORED_NO_BUNDLE | |
| return JUDGE_NOT_SCORED | |