| # AGENTS.md |
|
|
| ## Repository Orientation |
|
|
| This repository has **two main purposes**: |
|
|
| 1. **Production-facing agents/tools** for Hugging Face workflows |
| 2. **Evaluation harnesses** (prompts, runners, scoring, reports, plots) |
|
|
| --- |
|
|
| ## 1) Production Surface |
|
|
| Use these for real user-facing behavior: |
|
|
| - **Hub Community agent/tooling** |
| - Card: `.fast-agent/tool-cards/hf_hub_community.md` |
| - Tool backend: `.fast-agent/tool-cards/hf_api_tool.py` |
| - Focus: users/orgs/followers/discussions/collections/recent activity workflows |
|
|
| - **Daily Papers search agent/tooling** |
| - Card: `.fast-agent/tool-cards/hf_paper_search.md` |
| - Tool backend: `.fast-agent/tool-cards/hf_papers_tool.py` |
| - Focus: `/api/daily_papers` retrieval + filtering |
|
|
| --- |
|
|
| ## 2) Evaluation Inputs |
|
|
| Canonical challenge/config files: |
|
|
| - `scripts/hf_hub_community_challenges.txt` |
| - `scripts/tool_routing_challenges.txt` |
| - `scripts/tool_routing_expected.json` |
| - `scripts/tool_description_variants.json` |
|
|
| --- |
|
|
| ## 3) Evaluation Runners / Scorers |
|
|
| - `scripts/score_hf_hub_community_challenges.py` |
| - Runs + scores the HF Hub community challenge pack |
|
|
| - `scripts/score_tool_routing_confusion.py` |
| - Scores routing/confusion quality for one model |
|
|
| - `scripts/run_tool_routing_batch.py` |
| - Batch wrapper for routing eval across multiple models |
|
|
| - `scripts/eval_tool_description_ab.py` |
| - A/B evaluation of tool description variants |
|
|
| - `scripts/plot_tool_description_eval.py` |
| - Plot/interpretation generation from summary outputs |
|
|
| - `scripts/run_all_evals.sh` |
| - Convenience orchestrator for the full evaluation flow |
|
|
| --- |
|
|
| ## 4) Evaluation Outputs |
|
|
| - Community challenge reports: |
| - `docs/hf_hub_community_challenge_report.md` |
| - `docs/hf_hub_community_challenge_report.json` |
|
|
| - Routing evaluation outputs: |
| - `docs/tool_routing_eval/` |
|
|
| - Tool-description A/B outputs: |
| - `docs/tool_description_eval/` |
|
|
| Top-level result index: |
|
|
| - `docs/RESULTS.md` |
|
|
| --- |
|
|
| ## 5) Key Context Docs |
|
|
| - `README.md` (quick start + layout) |
| - `docs/SPACE.md` (workspace map) |
| - `docs/hf_hub_community_challenge_pack.md` |
| - `docs/tool_description_eval_setup.md` |
| - `docs/tool_description_eval/tool_description_interpretation.md` |
| - `bench.md` |
|
|
| --- |
|
|
| ## Suggested First Steps for New Contributors |
|
|
| 1. Read `README.md` and `docs/SPACE.md` |
| 2. Run one production query for each tool card |
| 3. Run one eval script |
| 4. Open generated report(s) in `docs/` |
| 5. Then edit cards/scripts with context |
|
|
| --- |
|
|
| ## Space Deployment / Sync (HF CLI) |
|
|
| This project is hosted on Hugging Face Spaces at: |
|
|
| - `https://huggingface.co/spaces/evalstate/hf-papers/` |
|
|
| When publishing card/script updates, use the `hf` CLI (not ad-hoc manual edits) to keep deployment reproducible. |
|
|
| Typical flow: |
|
|
| 1. Authenticate: |
| - `hf auth login` |
| 2. Work in the local repo and validate changes. |
| 3. Push updates to the Space repo with `hf` CLI workflows (e.g., clone/upload/commit via `hf` commands) targeting: |
| - `spaces/evalstate/hf-papers` |
|
|
| Keep production card changes (`.fast-agent/tool-cards/`) and related eval/report updates in sync when publishing. |
|
|
