Spaces:
Running
Running
File size: 8,673 Bytes
3dfc93b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 | """`solar-eval score` โ ์คํ๋ผ์ธ ์ฌ์ฑ์ ๋ช
๋ น ๊ทธ๋ฃน (๊ณํ 2026-09-26 L3b ยง2.1).
`score proofread-diff` ๋ run ๋ฒ๋ค์ `results.jsonl` ์ ๋ ๊ฑฐ์ `lcs_diff` ์ ๊ฐ์ ํจ์๋ก ๋ค์ ์ฑ์ ํด ์ฑ์ ๋ฒ๋ค
(`poc-scoring/v1`, `purpose=rescore_after_fix`)๋ก ๋ธ๋ค. ๋ ๋ชจ๋:
- `<run_bundle_dir>โฆ` ๋ชจ๋: ๋ฒ๋ค์ด `โฆ/eval/bundles/<id>` ์ ์์ผ๋ฉด ๊ธฐ๋ณธ ์ถ๋ ฅ์ `โฆ/eval/evaluations`, ์๋๋ฉด `--out` ํ์.
`--mirror` ๋ฉด ๊ฐ ๋ฒ๋ค ๋งค๋ํ์คํธ `case` ์ ๋ฏธ๋ฌ `evaluations` ์ ๋๋๋ค.
- `--case a/c --all` ๋ชจ๋: ๋ฏธ๋ฌ์ ๊ทธ case run ๋ฒ๋ค ์ ๋ถ(`poc_bundle.adopt` ๋ก ์ฑํ). `--mirror` ๋๋ `--out` ์ด ํ์ํ๋ค.
`--run <run_id>`(๋ฐ๋ณต)๋ฅผ ์ฃผ๋ฉด ๊ทธ run ๋ค๋ง(๊ณํ 2026-09-27 L6 ยง2.11 โ ์ฑํ ๊ท์น์ ๊ฐ๊ณ , ์์ญ์ ์๋ run ์ ์ค๋ฅ
`run_not_in_mirror`). `--run` ์ ์ด ๋ชจ๋์์๋ง ๋ฐ๊ณ , ๋น ๊ฐยท๊ณต๋ฐฑ๋งยท์ ์ด ๋ฌธ์ยท512์ ์ด๊ณผ๋ ์ฌ์ฉ ์ค๋ฅ๋ค(ES
`hub ingest --expect-run` ๊ณผ ๊ฐ์ ๋ฒ์ โ ์๋ run ์ด stderr ์ ๊ทธ๋๋ก ์ฐํ๋ฏ๋ก ์ค ์์กฐ๋ฅผ ๋ง๋๋ค).
- `--out DIR` ๊ณผ `--mirror` ๋ ํจ๊ป ์ฐ์ง ์๋๋ค. `--mirror-root` ๋ ๋ฏธ๋ฌ **๋ฒํท์ ๋ถ๋ชจ**๋ค(ยง2.1 "๋ฏธ๋ฌ ๋ฃจํธ ์ธต").
๋๋ ํ ๋ฆฌ ๋ชจ๋์์๋ `--mirror` ์ ํจ๊ป๋ง ๋ฐ๋๋ค. ์๋ run ๋ฒ๋ค ๊ฒฝ๋ก๋ ์ฌ์ฉ ์ค๋ฅ(์ข
๋ฃ 2)๋ค.
stdout ์ run ๋ง๋ค ํ ์ค `written <path>` ยท `exists <path>` ยท `skip <run_id> <code>` ์ด๊ณ , case ๋ชจ๋๋ ๋ง์ง๋ง ์ค์ JSON ์์ฝ
`{written, exists, skipped:{code:n}, errors, n_original_missing}` ์ ๋ธ๋ค. ์ค๋ฅ๋ stderr ์ `error <run_id> <code>` ํ ์ค์ฉ
(case ์ ์ฒด ์ค๋ฅ๋ run_id ์๋ฆฌ์ `-`, `case_area_missing`ยท`case_area_empty` ๋ ๋ค์ ํด์ํ ์์ญ ๊ฒฝ๋ก๋ฅผ ๋ถ์ธ๋ค).
์ด๋ ์ชฝ์๋ ์
๋ ฅยท์ถ๋ ฅยท์ ๋ต ์๋ฌธ์ ์ฃ์ง ์๋๋ค.
์ข
๋ฃ ์ฝ๋: 0 = ์ ๋ถ written/exists/๊ท์น skip ยท 1 = ์ค๋ฅ 1๊ฑด ์ด์ ยท 2 = ์ฌ์ฉ ์ค๋ฅ.
import ๊ฒฝ๊ณ(D1): ์ด ๋ชจ๋์ click ๋ง ๋ชจ๋ ์ต์์์์ import ํ๋ค. `solar_eval.rescore`(๊ทธ๋ฆฌ๊ณ ๊ทธ๊ฒ์ด ๋์ด์ค๋
`poc_bundle`)๋ **๋ช
๋ น ํจ์ ์์์** import ํ๋ค โ `solar-eval` CLI ๋ฅผ ๋์ฐ๋ ๊ฒ๋ง์ผ๋ก ์ฌ์ฑ์ ๊ธฐยท๋ฒ๋ค writer ๊ฐ
๋ก๋ฉ๋์ง ์๊ฒ.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
from typing import Any
import click
_DIR = click.Path(file_okay=False, path_type=Path)
# ์
๋ ฅ run ๋ฒ๋ค์ ์์ด์ผ ํ๋ค โ ๊ฒฝ๋ก ์คํ๊ฐ ๊ท์น skip(`latest_bundle_invalid`)ยท์ข
๋ฃ 0 ์ผ๋ก ๋๋์ง ์๊ฒ click ์ด ์ข
๋ฃ 2 ๋ก ๋ง๋๋ค.
_EXISTING_DIR = click.Path(exists=True, file_okay=False, path_type=Path)
EXIT_ERRORS = 1
EXIT_USAGE = 2
@click.group("score")
def score_group() -> None:
"""Offline rescoring of run bundles (scoring bundles, poc-scoring/v1)."""
@score_group.command("proofread-diff")
@click.argument("run_bundle_dirs", nargs=-1, type=_EXISTING_DIR)
@click.option("--case", "case_key", default=None, help="accounts/<a>/<c> as a/c (with --all).")
@click.option("--all", "all_runs", is_flag=True, default=False, help="Every adopted run bundle.")
@click.option("--out", "out_dir", type=_DIR, default=None, help="Parent dir for scoring bundles.")
@click.option("--mirror", is_flag=True, default=False, help="Write into the local mirror.")
@click.option(
"--mirror-root",
type=_DIR,
default=None,
help="Parent of the mirror bucket (default: data home blob-mirror).",
)
@click.option(
"--run",
"run_ids",
multiple=True,
help="Only this run_id (repeatable; with --case --all).",
)
def proofread_diff(
run_bundle_dirs: tuple[Path, ...],
case_key: str | None,
all_runs: bool,
out_dir: Path | None,
mirror: bool,
mirror_root: Path | None,
run_ids: tuple[str, ...],
) -> None:
"""Rescore run bundles with proofread_diff@2 (same scoring as legacy lcs_diff)."""
_check_usage(run_bundle_dirs, case_key, all_runs, out_dir, mirror, mirror_root, run_ids)
if case_key is not None:
code = _run_case(case_key, out_dir, mirror_root, frozenset(run_ids) or None)
else:
code = _run_dirs(run_bundle_dirs, out_dir, mirror_root if mirror else None, mirror)
sys.exit(code)
def _usage(message: str) -> None:
click.echo(f"usage error: {message}", err=True)
sys.exit(EXIT_USAGE)
def _check_usage(
dirs: tuple[Path, ...],
case_key: str | None,
all_runs: bool,
out_dir: Path | None,
mirror: bool,
mirror_root: Path | None,
run_ids: tuple[str, ...] = (),
) -> None:
"""๋ ๋ชจ๋์ ์ธ์ ์กฐํฉ โ ์ด๊ธ๋๋ฉด ์ข
๋ฃ 2."""
if out_dir is not None and mirror:
_usage("--out and --mirror are mutually exclusive")
if case_key is not None or all_runs:
if dirs:
_usage("give run bundle directories or --case/--all, not both")
if case_key is None or not all_runs:
_usage("--case a/c needs --all (and --all needs --case)")
if out_dir is None and not mirror:
_usage("--case --all needs --mirror or --out DIR")
elif not dirs:
_usage("give run bundle directories, or --case a/c --all")
elif mirror_root is not None and not mirror:
# ๋๋ ํ ๋ฆฌ ๋ชจ๋์์ --mirror-root ๋ --mirror ์ ์ถ๋ ฅ ์๋ฆฌ์ผ ๋ฟ์ด๋ค โ ์กฐ์ฉํ ๋ฒ๋ฆฌ์ง ์๋๋ค
_usage("--mirror-root needs --mirror when rescoring run bundle directories")
if run_ids:
_check_run_filter(case_key, run_ids)
def _check_run_filter(case_key: str | None, run_ids: tuple[str, ...]) -> None:
"""`--run` ์ case ๋ชจ๋์ run ํํฐ๋ค(L6 ยง2.11) โ ๋๋ ํ ๋ฆฌ ๋ชจ๋์์ ์กฐ์ฉํ ๋ฒ๋ฆฌ์ง ์๊ณ , ๊ฐ์ ๊ฒฝ๊ณ์์ ๊ฑฐ๋ฅธ๋ค.
case ๋ชจ๋์ `--case`/`--all` ์ง์ `_check_usage` ๊ฐ ๋จผ์ ๋ง์ผ๋ฏ๋ก ์ฌ๊ธฐ์ `case_key is None` ์ด๋ฉด ๋๋ ํ ๋ฆฌ ๋ชจ๋๋ค.
๊ฐ ๊ท์น์ ๋ผ์ด๋ธ๋ฌ๋ฆฌ ๊ฒฝ๊ณ(`rescore_case`)์ ๊ฐ์ `is_valid_run_id` ๋ค. ๊ฑฐ๋ถ ๋ฌธ๊ตฌ์ ๊ฐ ์๋ฌธ์ ์ฃ์ง ์๋๋ค.
"""
from solar_eval.rescore.case import MAX_RUN_ID_LENGTH, is_valid_run_id
if case_key is None:
_usage("--run needs --case a/c --all (not run bundle directories)")
if not all(is_valid_run_id(r) for r in run_ids):
_usage(
"--run needs a non-blank run_id without control characters,"
f" at most {MAX_RUN_ID_LENGTH} characters"
)
def _run_case(
case_key: str,
out_dir: Path | None,
mirror_root: Path | None,
run_ids: frozenset[str] | None = None,
) -> int:
from solar_eval.rescore.case import rescore_case
from solar_eval.rescore.paths import RescorePathError, parse_case_key
try:
account, case = parse_case_key(case_key)
except RescorePathError as exc:
_usage(str(exc))
summary = rescore_case(account, case, mirror_root=mirror_root, out_dir=out_dir, run_ids=run_ids)
for outcome in summary.outcomes:
_echo_outcome(outcome)
for failure in summary.failures:
suffix = f" {failure.detail}" if failure.detail else ""
click.echo(f"error {failure.run_id or '-'} {failure.code}{suffix}", err=True)
report: dict[str, Any] = summary.to_json()
click.echo(json.dumps(report, ensure_ascii=False, sort_keys=False))
return EXIT_ERRORS if report["errors"] else 0
def _run_dirs(
dirs: tuple[Path, ...], out_dir: Path | None, mirror_root: Path | None, mirror: bool
) -> int:
from solar_eval.rescore.evaluator_doc import current_evaluator_document
from solar_eval.rescore.from_bundle import RescoreError, rescore_run_bundle
from solar_eval.rescore.paths import default_evaluations_dir, resolve_mirror_root
targets = [(d, out_dir or (None if mirror else default_evaluations_dir(d))) for d in dirs]
missing = [d for d, target in targets if target is None and not mirror]
if missing:
_usage(f"{missing[0]} is not under โฆ/eval/bundles/ โ give --out DIR")
root = resolve_mirror_root(mirror_root) if mirror else None
evaluator = current_evaluator_document()
errors = 0
for bundle_dir, target in targets:
try:
outcome = rescore_run_bundle(bundle_dir, target, mirror_root=root, evaluator=evaluator)
except RescoreError as exc:
click.echo(f"error {exc.run_id or bundle_dir.name} {exc.code}", err=True)
errors += 1
continue
_echo_outcome(outcome, fallback=bundle_dir.name)
return EXIT_ERRORS if errors else 0
def _echo_outcome(outcome: Any, fallback: str = "-") -> None:
if outcome.status == "skip":
click.echo(f"skip {outcome.run_id or fallback} {outcome.code}")
else:
click.echo(f"{outcome.status} {outcome.path}")
|