agent.md

Agent Notes — Fin-EvalOps

Notes left by coding agents for future agents. Read this before making changes.

---

Architecture in 30 seconds

• Backend: FastAPI + SQLAlchemy + SQLite (WAL). Lives in backend/app/. Routers: testsets, runs, skills, dashboard, agent, annotations, sse. Background eval work in services/evaluator.py.
• Frontend: Vite + React + TS + AntD + ECharts. Lives in frontend/src/. Key page: pages/TestSets/index.tsx.
• Eval pipeline: TestCase → auto-route to a Skill (services/skill_router.py) → LLM judge (services/llm_client.py) → store on Run with weight_assignment, dimension_scores, caps, root_causes, narrative_review.
• Deployment:
  • Frontend → GitHub Pages (auto via .github/workflows/deploy.yml on push to main touching frontend/**).
  • Backend → HF Space appqqq/Fin-EvalOps-v2 (Docker SDK). NOT auto-synced from GitHub unless the user manually enables GitHub integration in the Space settings.
• Persistence: SQLite DB lives in /data on the Space, snapshot-pushed to a HF Dataset repo via persistence.py.

---

Recent work (2026-06 → 2026-07)

Feature: custom business categories

Business asked: don't force every eval batch into one of the 13 seeded self-eval categories. Let users define their own.

• Backend (routers/testsets.py): new POST /api/testsets/categories (with body validation: code charset = ^[\w一-鿿\-]+$, length 1-32) and DELETE /api/testsets/categories/{code} (refuses seed is_custom=false with 409; refuses in-use categories).
• Model (models.py): TestCategory.is_custom: Boolean + inline migration. Widened code / category_code to String(64) — SQLite type affinity means no DDL migration needed.
• Slug auto-derived as c-{slugified-code} to keep custom in its own namespace and avoid colliding with the canonical English slugs.
• Frontend (pages/TestSets/index.tsx): + 新建分类 button next to the category Select in all 3 modals (import / new / iwencai-pull); new top-toolbar 分类管理 modal with delete popconfirm; tree node title appends 🏷️ for custom rows.

Fix: tags lost across all JSON import paths

_normalize_raw (Chinese-keyed branch) didn't extract tags; import_file / scan_disk / import-from-iwencai never wrote tags=... to the TestCase ORM. Result: any upstream metadata packed into tags (e.g. by scripts/convert_part06.py) silently disappeared. Fixed in the same PR — tags=norm.get("tags") now propagates everywhere.

Fix: scan-disk 500 when testsets_root is missing

数据测试集/ is now uploaded at runtime via the web UI instead of bundled in the Docker image (see "Things you must know" below). Without data on disk, POST /api/testsets/scan-disk returned 500. Fixed: now returns ScanDiskResponse(scanned=0, inserted=0, updated=0, skipped=0) and the frontend's "磁盘同步" button just shows 0 instead of erroring.

Fix: eval pipeline — parallel-array root_causes + tool_use text-block recovery

Two recurring eval failures observed in user runs:

1. Judge outputs root_causes as a dict of parallel arrays (e.g. {l1: ["a","b"], l2: ["x","y"], raw_score: ["40","60"], …}) when the schema wants a list of objects. Same pattern occasionally on caps, skipped_dimensions. Validator throws SchemaValidationError.
2. minimax / other Anthropic-compatible APIs put JSON in a markdown text block instead of a tool_use block. The fallback tried to validate strictly; on failure the entire eval died.

Fix in services/llm_client.py:

• New _sanitize_judge_for_validation() runs before strict validation. Detects "dict of parallel arrays" shape and transposes to list of objects. Coerces raw_score / score_ceiling string-numbers to float. Splits string matched_golden_cases into list.
• call_with_schema now wraps validation: if it fails, fills missing required fields with empty defaults, re-validates; if still failing, returns partial data with a warning instead of raising. Downstream evaluator sanitizers can recover the rest.
• _call_anthropic text-block fallback: pre-sanitizes, then on residual mismatch logs a warning and returns data anyway.

Duplicate sanitizer in services/evaluator.py _sanitize_judge_output — defense in depth, handles cached / out-of-band payloads.

Frontend: explicit "开始导入" button on the JSON import modal

The Antd Upload.beforeUpload was firing the import the moment a file was selected, and Modal.footer={null} hid the default OK/Cancel. Users had no way to confirm the action. Now: file is staged in state; 开始导入 OK button (disabled until both category + file are set) triggers importFile; success message includes the destination category code.

---

Things you must know (operational gotchas)

HF Space sync — there is NO direct git push from this environment

git push to huggingface.co/spaces/... fails from this machine (network policy blocks port 443 to huggingface.co). The workaround is huggingface_hub.HfApi:

from huggingface_hub import HfApi
api = HfApi(token=open("/Users/appstore/.cache/huggingface/token").read().strip())
api.upload_file(
    path_or_fileobj=open("local/file", "rb").read(),
    path_in_repo="remote/path",
    repo_id="appqqq/Fin-EvalOps-v2",
    repo_type="space",
    commit_message="...",
)

HfApi uses an HTTP API path that is reachable; raw git over SSH or HTTPS to huggingface.co is not. Endpoints like model_info("gpt2") work without auth; authenticated writes need the cached token at ~/.cache/huggingface/token.

upload_folder is for many files at once (uses LFS for large ones); upload_file is for a single file commit. delete_files(repo_id, repo_type, delete_patterns=[...]) removes files with glob patterns.

SSL is intermittent — wrap writes in a retry loop. Each upload_file / delete_files creates its own commit, so failures mid-upload don't roll back earlier commits.

If a fresh user runs into the same git push failure, use the script at scripts/sync_to_hf_space.sh which clones locally + rsyncs + commits (assumes the user can reach huggingface.co from their machine).

Dockerfile: do NOT COPY 数据测试集/

The Docker image must not bundle test data. The pattern is:

COPY backend/ /app/backend/
COPY skills/  /app/skills/
# 数据测试集/ is NOT bundled — tests are uploaded via the web UI.

If you add the COPY line back, the build fails with "/数据测试集/自研评测测试集": not found (because the directory is excluded from the Space repo via the deletion commit 7d79f77a1f13).

.hfignore keeps test data out of HF uploads

/Users/appstore/AI-Code/Fin-EvalOps/.hfignore excludes 数据测试集/, *.jsonl, and _smoke_*.json. Also scripts/sync_to_hf_space.sh has matching rsync excludes. Don't re-add these — they were excluded for a reason (don't bundle customer test data into the Space image).

SQLite migration pattern is _run_inline_migrations()

Adding a column? Put it in the MIGRATIONS list in backend/app/db.py with a predicate + action SQL. Predicate should return 1+ rows when migration is needed. SQLite ALTER TABLE only supports ADD COLUMN and RENAME COLUMN — for anything more complex (e.g. widening column type), you need table rebuild or just leave the schema declaration alone (SQLite doesn't enforce VARCHAR length anyway).

TestCase is referenced everywhere — be careful with column changes

TestCase is the central model. Test cases are created from JSON import, manual API, scan-disk, iwencai import. If you change a column, update:

• The model in models.py
• _normalize_raw (handles Chinese-keyed JSON)
• All 4 creation paths: import_file, scan_disk, import-from-iwencai, create_testcase
• The serializer (schemas.py)

Missing any path → silent data loss (this is exactly what bit us with tags).

---

Testing

cd backend && python3 -m pytest tests/ -v

49 tests, all passing. Patterns:

• test_field_rename.py — DB-level migration tests (create temp DB with old schema, run migration, verify shape).
• test_custom_category.py — full-stack via TestClient with patched DB session.
• test_sanitize_judge_output.py — pure-function tests for LLM output reshaping (the 12 new cases cover parallel-array transpose + permissive fallback).
• test_skill_router.py, test_scorer.py, test_cache_headers.py, test_sanitize_judge_output.py (legacy) — unit tests.

For end-to-end smoke tests against a live Space, you'll need an LLM API key. The HF Space has provider keys configured; for local testing set MINIMAX_API_KEY (or others) in env and run uvicorn app.main:app --port 8000.

---

File map (for agents coming in cold)

Fin-EvalOps/
├── backend/app/
│   ├── main.py                   # FastAPI app, lifespan (HF persistence + skill sync)
│   ├── config.py                 # Settings, env-driven
│   ├── db.py                     # SQLite engine, migrations
│   ├── models.py                 # ORM models (TestCase, Run, Skill, TestCategory, ...)
│   ├── schemas.py                # Pydantic I/O schemas
│   ├── persistence.py            # HF Datasets snapshot push/pull
│   ├── routers/
│   │   ├── testsets.py           # ← most custom-category work lives here
│   │   ├── runs.py               # Run creation + polling
│   │   ├── skills.py             # Skill catalog API
│   │   ├── dashboard.py
│   │   ├── agent.py              # Data Agent (LLM chat over the DB)
│   │   ├── annotations.py
│   │   └── sse.py                # SSE channels for live progress
│   └── services/
│       ├── llm_client.py         # ← sanitizer fixes live here
│       ├── evaluator.py          # ← 5-step eval pipeline; _sanitize_judge_output here
│       ├── scorer.py             # weighted-sum + cap rule application
│       ├── skill_router.py       # auto-route question → skill
│       ├── skill_loader.py       # parses 自研评测Skill/ markdown into Skill rows
│       ├── data_agent.py         # Data Agent LLM prompt
│       └── fetch_iwencai.py      # iwencai EvalOps backend adapter
├── frontend/src/
│   ├── pages/TestSets/           # ← UX overhaul for custom categories lives here
│   ├── api/testsets.ts           # frontend API client (createCategory, deleteCategory)
│   └── components/, pages/{Agent,Annotations,Compare,Dashboard,Runs,Skills,Settings}/
├── skills/                       # 13 self-eval + 14 competitor + 14 e2e skill markdown
├── scripts/
│   ├── convert_part06.py         # JSONL → platform-importable JSON converter
│   └── sync_to_hf_space.sh       # git-based HF Space sync (only works for users with network access)
├── 数据测试集/                   # ⚠️ NOT shipped to Space (excluded in .hfignore)
│   ├── 自研评测测试集/             # 13 canonical test set files
│   ├── 竞品对比测试集/
│   ├── part_06.jsonl             # new test data (user's)
│   └── part_06-importable.json   # generated by scripts/convert_part06.py
├── .hfignore                     # excludes data from HF uploads
└── Dockerfile

---

Common tasks

Add a new eval metric dimension

1. Update the skill's markdown in skills/自研评测Skill/<NN>-<slug>/ (add dim to dimensions table, caps table, root_causes table).
2. Update the LLM schema in backend/app/services/evaluator.py EVAL_OUTPUT_SCHEMA dimension_scores.additionalProperties if you want strict validation.
3. The schema is shared across all 13 skills — don't break the others. If a dimension is skill-specific, define it per-skill via a relaxed schema.

Add a new business category via API

curl -X POST https://appqqq-fin-evalops-v2.hf.space/api/testsets/categories \
  -H 'Content-Type: application/json' \
  -d '{"code":"my-batch-2026q3","name_zh":"2026Q3 回归批次","description":"..."}'

Returns TestCategoryOut with is_custom: true, slug: c-my-batch-2026q3.

Import a test set into a custom category

curl -X POST 'https://appqqq-fin-evalops-v2.hf.space/api/testsets/import-file?category_code=my-batch-2026q3' \
  -F 'file=@my_set.json'

Response: {"inserted": N, "total_in_file": M}. Returns 400 if the category doesn't exist; the import endpoint accepts both Chinese-keyed ({问题, 答案, 链路数据, 上下文, id, 来源, tags}) and English-keyed ({question, agent_answer, reasoning_trace, context_history, source_id, source, tags}) JSON.

Trigger a single eval

curl -X POST https://appqqq-fin-evalops-v2.hf.space/api/runs \
  -H 'Content-Type: application/json' \
  -d '{"testcase_id":"<hex>","judge_model":"minimax-3"}'

Returns 201 with the run. Poll GET /api/runs/{run_id} for status. Auto-routes to the best matching skill unless skill_id is provided.

---

Known rough edges (not bugs, just be aware)

• cost_usd is always null — ModelSpec has no pricing fields. If you add pricing, update LLMResult and the evaluator to compute tokens_in * input_price + tokens_out * output_price.
• matched_golden_cases is often [] even when the question clearly matches a golden case — judge models tend to be conservative. The golden case list is provided in the prompt (truncated to 6000 chars).
• scan-disk always returns 0 on the Space (no bundled data). This is intentional after the 2026-07 cleanup.
• HF Space file count is ~1598 because the previous sync included .pytest_cache/ etc. These are harmless dead files but inflate the repo. Safe to ignore.