Spaces:
Running
Running
Download solar_eval/cli/commands/score.py from dev-strender/proofread-demo: direct link, hf CLI and curl.
- Browser
- Download file 8.67 kB
-
https://huggingface.co/spaces/dev-strender/proofread-demo/resolve/main/solar_eval/cli/commands/score.py
- Command line
-
hf download hf://spaces/dev-strender/proofread-demo/solar_eval/cli/commands/score.py
-
curl -L -o score.py https://huggingface.co/spaces/dev-strender/proofread-demo/resolve/main/solar_eval/cli/commands/score.py
8.67 kB
| """`solar-eval score` โ ์คํ๋ผ์ธ ์ฌ์ฑ์ ๋ช ๋ น ๊ทธ๋ฃน (๊ณํ 2026-09-26 L3b ยง2.1). | |
| `score proofread-diff` ๋ run ๋ฒ๋ค์ `results.jsonl` ์ ๋ ๊ฑฐ์ `lcs_diff` ์ ๊ฐ์ ํจ์๋ก ๋ค์ ์ฑ์ ํด ์ฑ์ ๋ฒ๋ค | |
| (`poc-scoring/v1`, `purpose=rescore_after_fix`)๋ก ๋ธ๋ค. ๋ ๋ชจ๋: | |
| - `<run_bundle_dir>โฆ` ๋ชจ๋: ๋ฒ๋ค์ด `โฆ/eval/bundles/<id>` ์ ์์ผ๋ฉด ๊ธฐ๋ณธ ์ถ๋ ฅ์ `โฆ/eval/evaluations`, ์๋๋ฉด `--out` ํ์. | |
| `--mirror` ๋ฉด ๊ฐ ๋ฒ๋ค ๋งค๋ํ์คํธ `case` ์ ๋ฏธ๋ฌ `evaluations` ์ ๋๋๋ค. | |
| - `--case a/c --all` ๋ชจ๋: ๋ฏธ๋ฌ์ ๊ทธ case run ๋ฒ๋ค ์ ๋ถ(`poc_bundle.adopt` ๋ก ์ฑํ). `--mirror` ๋๋ `--out` ์ด ํ์ํ๋ค. | |
| `--run <run_id>`(๋ฐ๋ณต)๋ฅผ ์ฃผ๋ฉด ๊ทธ run ๋ค๋ง(๊ณํ 2026-09-27 L6 ยง2.11 โ ์ฑํ ๊ท์น์ ๊ฐ๊ณ , ์์ญ์ ์๋ run ์ ์ค๋ฅ | |
| `run_not_in_mirror`). `--run` ์ ์ด ๋ชจ๋์์๋ง ๋ฐ๊ณ , ๋น ๊ฐยท๊ณต๋ฐฑ๋งยท์ ์ด ๋ฌธ์ยท512์ ์ด๊ณผ๋ ์ฌ์ฉ ์ค๋ฅ๋ค(ES | |
| `hub ingest --expect-run` ๊ณผ ๊ฐ์ ๋ฒ์ โ ์๋ run ์ด stderr ์ ๊ทธ๋๋ก ์ฐํ๋ฏ๋ก ์ค ์์กฐ๋ฅผ ๋ง๋๋ค). | |
| - `--out DIR` ๊ณผ `--mirror` ๋ ํจ๊ป ์ฐ์ง ์๋๋ค. `--mirror-root` ๋ ๋ฏธ๋ฌ **๋ฒํท์ ๋ถ๋ชจ**๋ค(ยง2.1 "๋ฏธ๋ฌ ๋ฃจํธ ์ธต"). | |
| ๋๋ ํ ๋ฆฌ ๋ชจ๋์์๋ `--mirror` ์ ํจ๊ป๋ง ๋ฐ๋๋ค. ์๋ run ๋ฒ๋ค ๊ฒฝ๋ก๋ ์ฌ์ฉ ์ค๋ฅ(์ข ๋ฃ 2)๋ค. | |
| stdout ์ run ๋ง๋ค ํ ์ค `written <path>` ยท `exists <path>` ยท `skip <run_id> <code>` ์ด๊ณ , case ๋ชจ๋๋ ๋ง์ง๋ง ์ค์ JSON ์์ฝ | |
| `{written, exists, skipped:{code:n}, errors, n_original_missing}` ์ ๋ธ๋ค. ์ค๋ฅ๋ stderr ์ `error <run_id> <code>` ํ ์ค์ฉ | |
| (case ์ ์ฒด ์ค๋ฅ๋ run_id ์๋ฆฌ์ `-`, `case_area_missing`ยท`case_area_empty` ๋ ๋ค์ ํด์ํ ์์ญ ๊ฒฝ๋ก๋ฅผ ๋ถ์ธ๋ค). | |
| ์ด๋ ์ชฝ์๋ ์ ๋ ฅยท์ถ๋ ฅยท์ ๋ต ์๋ฌธ์ ์ฃ์ง ์๋๋ค. | |
| ์ข ๋ฃ ์ฝ๋: 0 = ์ ๋ถ written/exists/๊ท์น skip ยท 1 = ์ค๋ฅ 1๊ฑด ์ด์ ยท 2 = ์ฌ์ฉ ์ค๋ฅ. | |
| import ๊ฒฝ๊ณ(D1): ์ด ๋ชจ๋์ click ๋ง ๋ชจ๋ ์ต์์์์ import ํ๋ค. `solar_eval.rescore`(๊ทธ๋ฆฌ๊ณ ๊ทธ๊ฒ์ด ๋์ด์ค๋ | |
| `poc_bundle`)๋ **๋ช ๋ น ํจ์ ์์์** import ํ๋ค โ `solar-eval` CLI ๋ฅผ ๋์ฐ๋ ๊ฒ๋ง์ผ๋ก ์ฌ์ฑ์ ๊ธฐยท๋ฒ๋ค writer ๊ฐ | |
| ๋ก๋ฉ๋์ง ์๊ฒ. | |
| """ | |
| from __future__ import annotations | |
| import json | |
| import sys | |
| from pathlib import Path | |
| from typing import Any | |
| import click | |
| _DIR = click.Path(file_okay=False, path_type=Path) | |
| # ์ ๋ ฅ run ๋ฒ๋ค์ ์์ด์ผ ํ๋ค โ ๊ฒฝ๋ก ์คํ๊ฐ ๊ท์น skip(`latest_bundle_invalid`)ยท์ข ๋ฃ 0 ์ผ๋ก ๋๋์ง ์๊ฒ click ์ด ์ข ๋ฃ 2 ๋ก ๋ง๋๋ค. | |
| _EXISTING_DIR = click.Path(exists=True, file_okay=False, path_type=Path) | |
| EXIT_ERRORS = 1 | |
| EXIT_USAGE = 2 | |
| def score_group() -> None: | |
| """Offline rescoring of run bundles (scoring bundles, poc-scoring/v1).""" | |
| def proofread_diff( | |
| run_bundle_dirs: tuple[Path, ...], | |
| case_key: str | None, | |
| all_runs: bool, | |
| out_dir: Path | None, | |
| mirror: bool, | |
| mirror_root: Path | None, | |
| run_ids: tuple[str, ...], | |
| ) -> None: | |
| """Rescore run bundles with proofread_diff@2 (same scoring as legacy lcs_diff).""" | |
| _check_usage(run_bundle_dirs, case_key, all_runs, out_dir, mirror, mirror_root, run_ids) | |
| if case_key is not None: | |
| code = _run_case(case_key, out_dir, mirror_root, frozenset(run_ids) or None) | |
| else: | |
| code = _run_dirs(run_bundle_dirs, out_dir, mirror_root if mirror else None, mirror) | |
| sys.exit(code) | |
| def _usage(message: str) -> None: | |
| click.echo(f"usage error: {message}", err=True) | |
| sys.exit(EXIT_USAGE) | |
| def _check_usage( | |
| dirs: tuple[Path, ...], | |
| case_key: str | None, | |
| all_runs: bool, | |
| out_dir: Path | None, | |
| mirror: bool, | |
| mirror_root: Path | None, | |
| run_ids: tuple[str, ...] = (), | |
| ) -> None: | |
| """๋ ๋ชจ๋์ ์ธ์ ์กฐํฉ โ ์ด๊ธ๋๋ฉด ์ข ๋ฃ 2.""" | |
| if out_dir is not None and mirror: | |
| _usage("--out and --mirror are mutually exclusive") | |
| if case_key is not None or all_runs: | |
| if dirs: | |
| _usage("give run bundle directories or --case/--all, not both") | |
| if case_key is None or not all_runs: | |
| _usage("--case a/c needs --all (and --all needs --case)") | |
| if out_dir is None and not mirror: | |
| _usage("--case --all needs --mirror or --out DIR") | |
| elif not dirs: | |
| _usage("give run bundle directories, or --case a/c --all") | |
| elif mirror_root is not None and not mirror: | |
| # ๋๋ ํ ๋ฆฌ ๋ชจ๋์์ --mirror-root ๋ --mirror ์ ์ถ๋ ฅ ์๋ฆฌ์ผ ๋ฟ์ด๋ค โ ์กฐ์ฉํ ๋ฒ๋ฆฌ์ง ์๋๋ค | |
| _usage("--mirror-root needs --mirror when rescoring run bundle directories") | |
| if run_ids: | |
| _check_run_filter(case_key, run_ids) | |
| def _check_run_filter(case_key: str | None, run_ids: tuple[str, ...]) -> None: | |
| """`--run` ์ case ๋ชจ๋์ run ํํฐ๋ค(L6 ยง2.11) โ ๋๋ ํ ๋ฆฌ ๋ชจ๋์์ ์กฐ์ฉํ ๋ฒ๋ฆฌ์ง ์๊ณ , ๊ฐ์ ๊ฒฝ๊ณ์์ ๊ฑฐ๋ฅธ๋ค. | |
| case ๋ชจ๋์ `--case`/`--all` ์ง์ `_check_usage` ๊ฐ ๋จผ์ ๋ง์ผ๋ฏ๋ก ์ฌ๊ธฐ์ `case_key is None` ์ด๋ฉด ๋๋ ํ ๋ฆฌ ๋ชจ๋๋ค. | |
| ๊ฐ ๊ท์น์ ๋ผ์ด๋ธ๋ฌ๋ฆฌ ๊ฒฝ๊ณ(`rescore_case`)์ ๊ฐ์ `is_valid_run_id` ๋ค. ๊ฑฐ๋ถ ๋ฌธ๊ตฌ์ ๊ฐ ์๋ฌธ์ ์ฃ์ง ์๋๋ค. | |
| """ | |
| from solar_eval.rescore.case import MAX_RUN_ID_LENGTH, is_valid_run_id | |
| if case_key is None: | |
| _usage("--run needs --case a/c --all (not run bundle directories)") | |
| if not all(is_valid_run_id(r) for r in run_ids): | |
| _usage( | |
| "--run needs a non-blank run_id without control characters," | |
| f" at most {MAX_RUN_ID_LENGTH} characters" | |
| ) | |
| def _run_case( | |
| case_key: str, | |
| out_dir: Path | None, | |
| mirror_root: Path | None, | |
| run_ids: frozenset[str] | None = None, | |
| ) -> int: | |
| from solar_eval.rescore.case import rescore_case | |
| from solar_eval.rescore.paths import RescorePathError, parse_case_key | |
| try: | |
| account, case = parse_case_key(case_key) | |
| except RescorePathError as exc: | |
| _usage(str(exc)) | |
| summary = rescore_case(account, case, mirror_root=mirror_root, out_dir=out_dir, run_ids=run_ids) | |
| for outcome in summary.outcomes: | |
| _echo_outcome(outcome) | |
| for failure in summary.failures: | |
| suffix = f" {failure.detail}" if failure.detail else "" | |
| click.echo(f"error {failure.run_id or '-'} {failure.code}{suffix}", err=True) | |
| report: dict[str, Any] = summary.to_json() | |
| click.echo(json.dumps(report, ensure_ascii=False, sort_keys=False)) | |
| return EXIT_ERRORS if report["errors"] else 0 | |
| def _run_dirs( | |
| dirs: tuple[Path, ...], out_dir: Path | None, mirror_root: Path | None, mirror: bool | |
| ) -> int: | |
| from solar_eval.rescore.evaluator_doc import current_evaluator_document | |
| from solar_eval.rescore.from_bundle import RescoreError, rescore_run_bundle | |
| from solar_eval.rescore.paths import default_evaluations_dir, resolve_mirror_root | |
| targets = [(d, out_dir or (None if mirror else default_evaluations_dir(d))) for d in dirs] | |
| missing = [d for d, target in targets if target is None and not mirror] | |
| if missing: | |
| _usage(f"{missing[0]} is not under โฆ/eval/bundles/ โ give --out DIR") | |
| root = resolve_mirror_root(mirror_root) if mirror else None | |
| evaluator = current_evaluator_document() | |
| errors = 0 | |
| for bundle_dir, target in targets: | |
| try: | |
| outcome = rescore_run_bundle(bundle_dir, target, mirror_root=root, evaluator=evaluator) | |
| except RescoreError as exc: | |
| click.echo(f"error {exc.run_id or bundle_dir.name} {exc.code}", err=True) | |
| errors += 1 | |
| continue | |
| _echo_outcome(outcome, fallback=bundle_dir.name) | |
| return EXIT_ERRORS if errors else 0 | |
| def _echo_outcome(outcome: Any, fallback: str = "-") -> None: | |
| if outcome.status == "skip": | |
| click.echo(f"skip {outcome.run_id or fallback} {outcome.code}") | |
| else: | |
| click.echo(f"{outcome.status} {outcome.path}") | |