dev-strender's picture
feat: switch demo to FT + Opus article review (pipeline_exp_261021_ft_hybrid)
3dfc93b verified
Raw History Blame Contribute Delete
16.2 kB
"""๋ฒˆ๋“ค ๋‹ค๋ฆฌ (๊ณ„ํš 2026-09-27 L6 ยง2.5 ยท D-L6-2 ยท D-L6-8 ยท D-L6-9).
case ์ž ๊ธˆ **์•ˆ**์—์„œ hook ์ด ๋ถ€๋ฅธ๋‹ค. ์ˆœ์„œ:
1. **๋ฐฑํ•„** โ€” `BackfillOptions(case=a/c, root=target.projects_dir, mirror_root=blob_mirror_root(), index_root=hub_index_root(),
repo_root=target.repo_root, run_ids=frozenset({run_id}))` โ†’ `backfill_case(opts, target.context)`(wrap โ†’ ๋ฏธ๋Ÿฌ โ†’ ๋ฒˆ๋“ค ์ƒ‰์ธ โ†’
eval-wrap). `CaseReport.runs` ๋Š” ์ •ํ™•ํžˆ ํ•˜๋‚˜์—ฌ์•ผ ํ•œ๋‹ค(0 ์ด๋ฉด `run_not_in_archive`). ๋ถ€๋ฅ˜ โ†’ ์ฝ”๋“œ:
wrap `writtenยทreusedยทrewrapped_input_fields` โ†’ ๋ฒˆ๋“ค ์žˆ์Œ ยท `frozen:<c>` โ†’ `bundle_frozen:<c>`(hub ๋Š” `no_bundle`) ยท
`error:<c>` โ†’ `bundle_error:<c>` ยท ๋ฒˆ๋“ค ์ƒ‰์ธ `error:<c>` โ†’ `bundle_index:<c>`(๊ณ„์†) ยท eval-wrap `error:*` โ†’
`eval_wrap:<error_code>`(๊ณ„์† โ€” ๋ฒˆ๋“ค์€ ์žˆ๋‹ค) ยท eval-wrap `skip:*`ยท`frozen:*` ๋Š” ์ •๋ณด ยท `BackfillError` โ†’ `mirror_unadoptable`.
2. **์žฌ์ฑ„์ **(๋ฒˆ๋“ค์ด ์žˆ์„ ๋•Œ๋งŒ) โ€” lcs_diff ๊ณ„์—ด(D-L6-8: run.json ์Šค๋ƒ…์ƒท `config.evaluator_config.config.type`, ์Šค๋ƒ…์ƒท์ด ์—†์œผ๋ฉด run
๋””๋ ‰ํ† ๋ฆฌ ํ‰๊ฐ€ ํŒŒ์ผ์˜ `poc_bundle.legacy_run.evaluator_family`)์ด๋ฉด `rescore_case(a, c, run_ids={run_id})`(case ๋ชจ๋“œ ์ฑ„ํƒ ๊ทœ์น™,
D-L6-9) โ†’ `written|exists` ๊ฐ€ ok ยท `no_results` skip โ†’ `rescore_empty` ยท ๊ทธ ๋ฐ– skip โ†’ `rescore_skip:<code>` ยท ์˜ค๋ฅ˜ โ†’
`rescore_error:<code>`. ๊ณ„์—ด์ด ์•„๋‹ˆ๋ฉด `not_proofread_family`(์ •๋ณด โ€” "์ด ํ‰๊ฐ€๊ธฐ๋Š” M2 ์ฑ„์  ๋ฒˆ๋“ค ์ƒ์‚ฐ์ž๊ฐ€ ์—†๋‹ค", Q-M2-1).
3. **์ƒ‰์ธ** โ€” `indexes.refresh`(lockยทํ…œํ”Œ๋ฆฟยทํ…์ŠคํŠธ ๋ชจ๋“œยท์ถ”์ถœ๊ธฐยท์ƒ‰์ธ ์ฝ”๋“œ ํ‚ค๊ฐ€ ๋ฐ”๋€Œ์—ˆ์„ ๋•Œ๋งŒ, D-L6-7).
๋‹ค๋ฆฌ ๊ฒฐ๊ณผ๋Š” ํ•˜๋‚˜์˜ `LegResult` ๋‹ค. ์‹คํŒจ๊ฐ€ ์—ฌ๋Ÿฟ์ด๋ฉด **์ˆœ์„œ์ƒ ์ฒซ ์‹คํŒจ**๊ฐ€ `code` ์ด๊ณ , ๋‹จ๊ณ„๋งˆ๋‹ค์˜ ๊ฒฐ๊ณผ๋Š” `detail` ์— ๋ชจ๋‘ ๋‚จ๋Š”๋‹ค
(์ƒํƒœ ํŒŒ์ผ ยง2.7): `{"backfill": run_entry, "rescore": {"status", "code"?, "bundle_id"?}, "indexes": <์ฝ”๋“œ>}`. ๋ฒˆ๋“ค์ด ์žˆ์œผ๋ฉด ์‹คํŒจ์—ฌ๋„
`detail.backfill.wrap` ์ด ๋‚จ์•„ hook ์ด `types.bundle_ready` ๋กœ hub ๋‹ค๋ฆฌ๋ฅผ ๋ˆ๋‹ค(D-L6-2). ์ƒ‰์ธ CLI ์˜ ์ •ํ•ฉ์„ฑ ๊ฒฝ๊ณ ๊ฐ€ ์žˆ์œผ๋ฉด
`index_warnings: <๊ฑด์ˆ˜>`, ์น˜์šด ์˜› ์ƒ‰์ธ ๋””๋ ‰ํ† ๋ฆฌ๊ฐ€ ์žˆ์œผ๋ฉด `index_removed: [<์ด๋ฆ„>โ€ฆ]` ๋ฅผ ๋”ํ•œ๋‹ค(๋ฆฌ๋ทฐ ์ง€์  2ยท4 โ€” ์ค„ ์ž์ฒด๋Š”
`indexes` ๊ฐ€ stderr ๋กœ ๋‚ธ๋‹ค).
**๋‹จ๊ณ„ ์˜ˆ์™ธ ๋ด‰์‡„.** ์žฌ์ฑ„์ ยท์ƒ‰์ธ ๋‹จ๊ณ„์˜ ์˜ˆ๊ธฐ์น˜ ์•Š์€ ์˜ˆ์™ธ๋Š” ๊ทธ ๋‹จ๊ณ„๋งŒ `rescore_exception:<Type>`ยท`indexes_exception:<Type>` ์‹คํŒจ๋กœ
๊ฐ•๋“ฑํ•œ๋‹ค(ํƒ€์ž… ์ด๋ฆ„๋งŒ) โ€” ๋ฐฑํ•„์ด ๋งŒ๋“  ๋ฒˆ๋“ค์„ ๋ฒ„๋ฆฌ์ง€ ์•Š๊ณ  hub ๋‹ค๋ฆฌ๊นŒ์ง€ ๊ฐ€๊ฒŒ. ๋ฐฑํ•„ ์ž์ฒด์˜ ์˜ˆ์™ธ๋Š” hook ์ด `bundle_exception:<Type>`
์œผ๋กœ ๊ฐ•๋“ฑํ•œ๋‹ค(๋ฒˆ๋“ค ์œ ๋ฌด๋ฅผ ๋ชจ๋ฅธ๋‹ค). ์›๋ฌธยทlocals ๋Š” ์‹ฃ์ง€ ์•Š์ง€๋งŒ **๋ฐœ์ƒ ์œ„์น˜**๋Š” ๋‚จ๊ธด๋‹ค(๋ฆฌ๋ทฐ ์ง€์  5): ๊ฐ€์žฅ ์•ˆ์ชฝ ํ”„๋ ˆ์ž„์˜
`<ํŒŒ์ผ basename>:<์ค„>:<ํ•จ์ˆ˜>` ๋ฅผ `detail.exception_at.<๋‹จ๊ณ„>` ์—(๊ณ ๊ฐ ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ์ฝ”๋“œ ์ขŒํ‘œ), ํŠธ๋ ˆ์ด์Šค๋ฐฑ ์ „์ฒด๋Š” ์ด ๋ชจ๋“ˆ ๋กœ๊ฑฐ์—
(`hook` ๊ณผ ๊ฐ™์€ ๊ทœ์น™ โ€” ๊ธฐ๋ณธ DEBUG, `SOLAR_EVAL_HOOK_DEBUG=1` ์ด๋ฉด WARNING).
`poc_bundle`ยท`solar_eval.rescore` ๋Š” ํ•จ์ˆ˜ ์•ˆ์—์„œ๋งŒ import ํ•œ๋‹ค(D-L6-14). ์ถœ๋ ฅยทdetail ์—๋Š” idยท๋ถ€๋ฅ˜ยท์ฝ”๋“œ๋งŒ(๊ณ ๊ฐ ํ…์ŠคํŠธ ์—†์Œ).
"""
from __future__ import annotations
import logging
import traceback
from collections.abc import Mapping
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from solar_eval.hub_hook import indexes, optin, target
from solar_eval.hub_hook.types import (
BUNDLE_ERROR_PREFIX,
BUNDLE_FROZEN_PREFIX,
BUNDLE_INDEX_FAILED_PREFIX,
EVAL_WRAP_FAILED_PREFIX,
MIRROR_UNADOPTABLE,
NO_BUNDLE,
NOT_PROOFREAD_FAMILY,
PROOFREAD_EVALUATOR_TYPE,
RESCORE_EMPTY,
RESCORE_ERROR_PREFIX,
RESCORE_NO_RESULTS,
RESCORE_OK,
RESCORE_SKIP_PREFIX,
RUN_NOT_IN_ARCHIVE,
STATUS_FAILED,
STATUS_NOT_RUN,
STATUS_SKIPPED,
STEP_BACKFILL,
STEP_INDEXES,
STEP_RESCORE,
WRAP_BUNDLE_READY,
WRAP_ERROR_PREFIX,
WRAP_FROZEN_PREFIX,
LegResult,
Target,
exception_code,
)
_RUN_JSON = "run.json"
_LOG = logging.getLogger(__name__)
# detail ํ‚ค `index_warnings`ยท`index_removed`ยท`exception_at`(๋ฆฌ๋ทฐ ์ง€์  2ยท4ยท5)๋Š” ์ฝ”๋“œ ๋ชจ์–‘ ๋ฌธ์ž์—ด์ด๋ผ ์ด ๋ชจ๋“ˆ ์ตœ์ƒ์œ„ ์ƒ์ˆ˜๋กœ ๋‘์ง€
# ์•Š๋Š”๋‹ค(`test_types_home` โ€” ์ฝ”๋“œ ๋ฌธ์ž์—ด์€ `types.py` ํ•œ ๊ณณ). ์“ฐ๋Š” ์ž๋ฆฌ์— ํ•œ ๋ฒˆ์”ฉ๋งŒ ์ ์—ˆ๋‹ค โ€” l6a-verify ๊ฐ€ `types.py` ๋กœ ์˜ฎ๊ธด๋‹ค
# (๊ณ„ํš ยง2.5 ๊ตฌํ˜„ ์‹œ ๋ณ€๊ฒฝ ๋ฉ”๋ชจ).
@dataclass(frozen=True)
class _Backfilled:
"""๋ฐฑํ•„ ๊ฒฐ๊ณผ โ€” `entry` ๋Š” ๋ณด๊ณ ์„œ `run_entry`(์—†์œผ๋ฉด run ์ด ์—†๊ฑฐ๋‚˜ case ์˜ค๋ฅ˜), `failures` ๋Š” ์ˆœ์„œ๋Œ€๋กœ์˜ ์‹คํŒจ ์ฝ”๋“œ."""
entry: Mapping[str, Any] | None
run_id: str | None = None
bundle_id: str | None = None
failures: tuple[str, ...] = ()
@property
def ready(self) -> bool:
return self.entry is not None and self.entry.get("wrap") in WRAP_BUNDLE_READY
@dataclass
class _Steps:
"""๋‹ค๋ฆฌ ์•ˆ ๋‹จ๊ณ„๋“ค์˜ ๊ฒฐ๊ณผ โ€” ์ฒซ ์‹คํŒจ์™€ ๊ทธ hint ๋ฅผ ๋ชจ์€๋‹ค."""
detail: dict[str, Any] = field(default_factory=dict)
failures: list[str] = field(default_factory=list)
hint: str | None = None
def fail(self, code: str, hint: str | None = None) -> None:
if not self.failures:
self.hint = hint
self.failures.append(code)
def result(self) -> LegResult:
if self.failures:
return LegResult.failed(self.failures[0], self.hint, self.detail)
return LegResult.ok(self.detail)
def contain(self, step: str, exc: Exception) -> str:
"""๋‹จ๊ณ„ ์˜ˆ์™ธ โ†’ ์ฝ”๋“œ(ํƒ€์ž… ์ด๋ฆ„๋งŒ). ์œ„์น˜ ์ขŒํ‘œ๋Š” detail ์—, ํŠธ๋ ˆ์ด์Šค๋ฐฑ์€ ๋กœ๊ฑฐ๋กœ(๋ชจ๋“ˆ ์„ค๋ช… ยท ๋ฆฌ๋ทฐ ์ง€์  5)."""
code = exception_code(step, exc)
where = exception_location(exc)
if where is not None:
self.detail.setdefault("exception_at", {})[step] = where
level = logging.WARNING if optin.hook_debug_requested() else logging.DEBUG
_LOG.log(level, "Experiment Hub ํ›… ๋ฒˆ๋“ค ๋‹ค๋ฆฌ ๋‹จ๊ณ„ ์˜ˆ์™ธ ๋ด‰์‡„ โ€” %s", code, exc_info=exc)
return code
def exception_location(exc: BaseException) -> str | None:
"""๊ฐ€์žฅ ์•ˆ์ชฝ ํ”„๋ ˆ์ž„์˜ `<ํŒŒ์ผ basename>:<์ค„>:<ํ•จ์ˆ˜>` โ€” ๋ฉ”์‹œ์ง€ยทlocals ๋Š” ์‹ฃ์ง€ ์•Š๋Š”๋‹ค(๊ณ ๊ฐ ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ์ฝ”๋“œ ์ขŒํ‘œ๋งŒ)."""
frames = traceback.extract_tb(exc.__traceback__)
if not frames:
return None
last = frames[-1]
return f"{Path(last.filename).name}:{last.lineno}:{last.name}"
def run_bundle_leg(target: Target, run_dir: Path, run_id: str) -> LegResult:
"""run ํ•˜๋‚˜์˜ ๋ฐฑํ•„ โ†’ ์žฌ์ฑ„์  โ†’ ์ƒ‰์ธ(ยง2.5). ๋ฐฑํ•„ ์˜ˆ์™ธ๋Š” hook ์ด `bundle_exception:<Type>` ์œผ๋กœ ๊ฐ•๋“ฑํ•œ๋‹ค."""
backfilled = _backfill(target, run_id)
if backfilled.entry is None:
return LegResult.failed(backfilled.failures[0])
steps = _Steps({STEP_BACKFILL: dict(backfilled.entry)})
for code in backfilled.failures:
steps.fail(code)
if backfilled.ready:
_rescore_step(target, Path(run_dir), backfilled, steps)
else:
steps.detail[STEP_RESCORE] = {"status": STATUS_NOT_RUN, "code": NO_BUNDLE}
_indexes_step(target, steps)
return steps.result()
def ensure_run_bundle(target: Target, run_id: str) -> tuple[LegResult, Path | None]:
"""๋ฐฑํ•„ ๋ถ€๋ถ„๋งŒ(L6b `runs eval` ์ƒˆ ๊ฒฝ๋กœ โ‘ ) โ€” ์ฑ„ํƒ๋œ ๋ฏธ๋Ÿฌ run ๋ฒˆ๋“ค ๊ฒฝ๋กœ๋ฅผ ํ•จ๊ป˜ ๋Œ๋ ค์ค€๋‹ค. ๋ฒˆ๋“ค์ด ์—†์œผ๋ฉด None.
**์ „์ œ: ํ˜ธ์ถœ์ž๊ฐ€ `lock.case_lock(target.account, target.case)` ๋ฅผ ์ฅ” ์ฑ„ ๋ถ€๋ฅธ๋‹ค**(๊ณ„ํš ยง2.12 B4 "new_path (case ์ž ๊ธˆ ์•ˆ)" ยท
๋ฆฌ๋ทฐ ์ง€์  6) โ€” ๋ฐฑํ•„์€ ๋ฏธ๋Ÿฌยท๋ฒˆ๋“ค ์ƒ‰์ธ ์‚ฌ์ด๋“œ์นดยทeval-wrap ์„ ์“ฐ๋ฏ€๋กœ ๊ฐ™์€ case ์˜ ํ›…๊ณผ ๊ฒน์น˜๋ฉด ์•ˆ ๋œ๋‹ค. ์—ฌ๊ธฐ์„œ ์ž ๊ทธ์ง€ ์•Š๋Š”
๊นŒ๋‹ญ: L6b ๋Š” โ‘ ~โ‘ค(๋ฒˆ๋“ค ํ™•์ธยท์ฑ„์ ยท๋ฐœํ–‰ยทhubยท์ƒํƒœ ํŒŒ์ผ)๋ฅผ ํ•œ ์ž ๊ธˆ ์•ˆ์—์„œ ๋ˆ๋‹ค โ€” `flock` ์€ ๊ฐ™์€ ํ”„๋กœ์„ธ์Šค๋ผ๋„ ๋”ฐ๋กœ ์—ฐ ๊ธฐ์ˆ ์ž๋ผ๋ฆฌ
๋ง‰์œผ๋ฏ€๋กœ ์•ˆ์—์„œ ๋‹ค์‹œ ์žก์œผ๋ฉด ์ž๊ธฐ ์ž์‹ ์„ ๊ธฐ๋‹ค๋ฆฌ๋‹ค ํ•œ๋„์— ๊ฑธ๋ฆฐ๋‹ค.
ํŒ์ • ์ง‘ํ•ฉ์€ `run_bundle_leg` ์™€ ๊ฐ™๋‹ค(`WRAP_BUNDLE_READY`) โ€” ํ›…์—์„œ ํ†ต๊ณผํ•œ run ์ด ์—ฌ๊ธฐ์„œ ์‹คํŒจํ•˜์ง€ ์•Š๊ฒŒ(๊ฒ€ํ†  F12).
eval-wrapยท๋ฒˆ๋“ค ์ƒ‰์ธ ์‹คํŒจ๋Š” `run_bundle_leg` ์ฒ˜๋Ÿผ ์‹คํŒจ๋กœ ์‹ฃ๋˜ ๊ฒฝ๋กœ๋Š” ๋Œ๋ ค์ค€๋‹ค(๋ฒˆ๋“ค์€ ์žˆ๋‹ค).
"""
backfilled = _backfill(target, run_id)
if backfilled.entry is None:
return LegResult.failed(backfilled.failures[0]), None
detail = {STEP_BACKFILL: dict(backfilled.entry)}
path = _mirror_bundle_path(target, backfilled.bundle_id) if backfilled.ready else None
if backfilled.failures:
return LegResult.failed(backfilled.failures[0], detail=detail), path
return LegResult.ok(detail), path
# โ”€โ”€ ๋ฐฑํ•„ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
def _backfill(target: Target, run_id: str) -> _Backfilled:
"""`backfill_case` ๋ฅผ run ํ•˜๋‚˜๋กœ. ์ถฉ๋Œ ๊ฒ€์‚ฌ๋Š” PB ๊ฐ€ case ์˜ run ๋””๋ ‰ํ† ๋ฆฌ ์ „๋ถ€๋กœ ํ•œ๋‹ค."""
from poc_bundle.backfill.report import run_entry
from poc_bundle.backfill.runner import BackfillError, BackfillOptions, backfill_case
from poc_bundle.home import blob_mirror_root, hub_index_root
opts = BackfillOptions(
case=target.case_ref,
root=target.projects_dir,
mirror_root=blob_mirror_root(),
index_root=hub_index_root(),
repo_root=target.repo_root,
run_ids=frozenset({run_id}),
)
try:
report = backfill_case(opts, target.context)
except BackfillError:
return _Backfilled(None, failures=(MIRROR_UNADOPTABLE,))
if not report.runs:
return _Backfilled(None, failures=(RUN_NOT_IN_ARCHIVE,))
if len(report.runs) != 1:
raise RuntimeError(f"backfill returned {len(report.runs)} runs for one run directory")
run = report.runs[0]
return _Backfilled(run_entry(run), run.run_id, run.bundle_id, _backfill_failures(run))
def _backfill_failures(run: Any) -> tuple[str, ...]:
"""`RunReport` ๋ถ€๋ฅ˜ โ†’ ๋ฒˆ๋“ค ๋‹ค๋ฆฌ ์‹คํŒจ ์ฝ”๋“œ(์ˆœ์„œ: ์–ด๋Œ‘ํ„ฐยทwrap โ†’ ๋ฒˆ๋“ค ์ƒ‰์ธ โ†’ eval-wrap)."""
failures: list[str] = []
wrap_failure = _wrap_failure(run.adapter, run.wrap)
if wrap_failure is not None:
failures.append(wrap_failure)
if isinstance(run.index, str) and run.index.startswith(WRAP_ERROR_PREFIX):
failures.append(f"{BUNDLE_INDEX_FAILED_PREFIX}{run.index.removeprefix(WRAP_ERROR_PREFIX)}")
if isinstance(run.eval_wrap, str) and run.eval_wrap.startswith(WRAP_ERROR_PREFIX):
detail = run.error_code or run.eval_wrap.removeprefix(WRAP_ERROR_PREFIX)
failures.append(f"{EVAL_WRAP_FAILED_PREFIX}{detail}")
return tuple(failures)
def _wrap_failure(adapter: str | None, wrap: str | None) -> str | None:
"""wrap ์ด ๋ฒˆ๋“ค์„ ๋‚จ๊ธฐ์ง€ ๋ชปํ–ˆ์œผ๋ฉด ๊ทธ ์ฝ”๋“œ. ์–ด๋Œ‘ํ„ฐ๊ฐ€ ๋ง‰์•„ wrap ์— ๋‹ฟ์ง€ ์•Š์•˜์œผ๋ฉด ์–ด๋Œ‘ํ„ฐ ๋ถ€๋ฅ˜๋กœ."""
if wrap in WRAP_BUNDLE_READY:
return None
if isinstance(wrap, str) and wrap.startswith(WRAP_FROZEN_PREFIX):
return f"{BUNDLE_FROZEN_PREFIX}{wrap.removeprefix(WRAP_FROZEN_PREFIX)}"
stage = wrap if isinstance(wrap, str) else adapter
if isinstance(stage, str) and stage.startswith(WRAP_ERROR_PREFIX):
return f"{BUNDLE_ERROR_PREFIX}{stage.removeprefix(WRAP_ERROR_PREFIX)}"
return f"{BUNDLE_ERROR_PREFIX}{stage or 'none'}"
def _mirror_bundle_path(t: Target, bundle_id: str | None) -> Path | None:
"""๋ฏธ๋Ÿฌ์˜ `accounts/<a>/<c>/eval/bundles/<bundle_id>`(์žˆ์„ ๋•Œ๋งŒ)."""
from solar_eval.rescore.paths import BUNDLES_AREA, mirror_area_dir
if not bundle_id:
return None
path = mirror_area_dir(None, t.account, t.case, BUNDLES_AREA) / bundle_id
return path if path.is_dir() else None
# โ”€โ”€ ์žฌ์ฑ„์  โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
def _rescore_step(t: Target, run_dir: Path, backfilled: _Backfilled, steps: _Steps) -> None:
try:
if not is_proofread_family(run_dir):
steps.detail[STEP_RESCORE] = {"status": STATUS_SKIPPED, "code": NOT_PROOFREAD_FAMILY}
return
status, code, bundle_id = _rescore(t, backfilled.run_id or run_dir.name)
except Exception as exc: # ๋‹จ๊ณ„ ๊ฒฝ๊ณ„ โ€” ๋ฐฑํ•„์ด ๋งŒ๋“  ๋ฒˆ๋“ค๋กœ hub ๋‹ค๋ฆฌ๋Š” ๊ณ„์† ๊ฐ„๋‹ค(๋ชจ๋“ˆ ์„ค๋ช…)
status, code, bundle_id = STATUS_FAILED, steps.contain(STEP_RESCORE, exc), None
entry: dict[str, Any] = {"status": status}
if code is not None:
entry["code"] = code
if bundle_id is not None:
entry["bundle_id"] = bundle_id
steps.detail[STEP_RESCORE] = entry
if status == STATUS_FAILED and code is not None:
steps.fail(code)
def is_proofread_family(run_dir: Path) -> bool:
"""์žฌ์ฑ„์  ๋Œ€์ƒ์ธ๊ฐ€(D-L6-8). ์Šค๋ƒ…์ƒท ํ‰๊ฐ€๊ธฐ type ์ด ์žˆ์œผ๋ฉด ๊ทธ๊ฒƒ, ์—†์œผ๋ฉด run ๋””๋ ‰ํ† ๋ฆฌ ํ‰๊ฐ€ ํŒŒ์ผ์˜ ๊ณ„์—ด(๊ฒ€ํ†  F18).
run.jsonยทํ‰๊ฐ€ ํŒŒ์ผ์„ ๋ชป ์ฝ์œผ๋ฉด ๊ณ„์—ด์„ ๋ชจ๋ฅด๋ฏ€๋กœ ๋Œ€์ƒ์ด ์•„๋‹ˆ๋‹ค(`not_proofread_family` โ€” wrap ์ด ๊ทธ run ์„ ์ด๋ฏธ ๊ฐ€๋ฅธ๋‹ค).
"""
from poc_bundle.canonical import load_json_strict
from poc_bundle.legacy_run import (
FAMILY_LCS,
LegacyRunError,
evaluator_family,
read_eval_files,
split_rounds,
)
try:
run_json = load_json_strict((Path(run_dir) / _RUN_JSON).read_bytes(), safe_integers=False)
except (OSError, ValueError, RecursionError):
return False
if not isinstance(run_json, dict):
return False
kind = target.snapshot_evaluator_type(run_json)
if kind is not None:
return kind == PROOFREAD_EVALUATOR_TYPE
try:
evaluation, lines = read_eval_files(Path(run_dir))
except LegacyRunError:
return False
return evaluator_family(run_json, evaluation, split_rounds(evaluation, lines)) == FAMILY_LCS
def _rescore(t: Target, run_id: str) -> tuple[str, str | None, str | None]:
"""`rescore_case(run_ids={run_id})` โ†’ (์ƒํƒœ, ์ฝ”๋“œ, ์ฑ„์  ๋ฒˆ๋“ค id). ์ƒํƒœ๋Š” `written|exists`(ok) ๋˜๋Š” `failed`."""
from poc_bundle.home import blob_mirror_root
from solar_eval.rescore.case import rescore_case
summary = rescore_case(
t.account,
t.case,
mirror_root=blob_mirror_root(),
repo_root=t.repo_root,
run_ids=frozenset({run_id}),
)
if summary.failures:
return STATUS_FAILED, f"{RESCORE_ERROR_PREFIX}{summary.failures[0].code}", None
if len(summary.outcomes) != 1:
return STATUS_FAILED, RESCORE_EMPTY, None
outcome = summary.outcomes[0]
if outcome.status in RESCORE_OK:
bundle_id = None if outcome.path is None else Path(outcome.path).name
return outcome.status, None, bundle_id
if outcome.code == RESCORE_NO_RESULTS:
return STATUS_FAILED, RESCORE_EMPTY, None
return STATUS_FAILED, f"{RESCORE_SKIP_PREFIX}{outcome.code}", None
# โ”€โ”€ ์ƒ‰์ธ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
def _indexes_step(t: Target, steps: _Steps) -> None:
try:
outcome = indexes.refresh(t)
except Exception as exc: # ๋‹จ๊ณ„ ๊ฒฝ๊ณ„ โ€” ์ƒ‰์ธ์€ case ํŒŒ์ƒ๋ฌผ์ด๋ผ run ๋ฒˆ๋“คยทhub ๋‹ค๋ฆฌ๋ฅผ ๋ง‰์ง€ ์•Š๋Š”๋‹ค
code = steps.contain(STEP_INDEXES, exc)
steps.detail[STEP_INDEXES] = f"{STATUS_FAILED}:{code}"
steps.fail(code)
return
steps.detail[STEP_INDEXES] = outcome.code
if outcome.warnings:
steps.detail["index_warnings"] = len(outcome.warnings)
if outcome.removed:
steps.detail["index_removed"] = list(outcome.removed)
if outcome.failed:
steps.fail(outcome.code.removeprefix(f"{STATUS_FAILED}:"), outcome.reason)
__all__ = ["ensure_run_bundle", "is_proofread_family", "run_bundle_leg"]