reference/README.md

reference/ — shared eval reference data

Cross-cutting data that more than one experiment depends on, kept in one place so the experiments can't drift. (Distinct from discovery-and-conformance/, which is the reference area for host-app/capability facts — this folder is for eval infrastructure: the models and providers we run experiments against.)

models.json — the model/provider registry

The single source of truth for the models we commonly test. Every experiment that calls models should consume this list rather than keep its own copy.

Each entry is minimal — just identity, not per-run knobs:

field	meaning
label	stable short name used in results/datasets (e.g. claude-fable-5). Unique.
provider	how it's reached: huggingface (the HF Inference Providers router, https://router.huggingface.co/v1, one HF_TOKEN — the default route for open-weight models), anthropic / openai / google (the vendor's native first-party SDK via ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY), moonshot (Moonshot's native first-party API — OpenAI-compatible chat completions at https://api.moonshot.ai/v1, key MOONSHOT_API_KEY; the vendor's own route to the Kimi models, kept alongside the router route where both exist — kimi-k3 and kimi-k3-hf are the same model reached two ways), or lan (a local OpenAI-compatible server — Ollama/llama.cpp/vLLM — via LAN_BASE_URL/LAN_API_KEY).
model	the provider's model id: an HF Hub id (zai-org/GLM-5.2), a native vendor id (claude-fable-5), or a local model id (llama3.2:3b).

Run-specific concerns (temperature, HF provider pinning via the <hub-id>:<provider> suffix, tool_choice, retries) belong in the consuming experiment, keyed by label — not here. Keep this file identity-only.

Adding a model

Add one line with a unique label, the right provider, and the model id. Every consumer picks it up on its next run (a vendored copy may need re-syncing — see below).

inference-provider-capabilities.{md,jsonl} — provider capability snapshot

Which Hugging Face Inference Providers expose tool calling and structured output (plus pricing/context) per (model, provider), so experiments can pick a route that actually supports the feature they need. models.json is identity-only; this captures the orthogonal capability facts. Sourced from HF's Inference Providers catalog (https://router.huggingface.co/v1/models). It is a dated snapshot — see the .md for how to refresh, and re-pull if it's more than ~7 days old.

Consumers

• mcp/explicit-state-handles/harness/ (TypeScript) — loads this file directly (src/index.ts → loadModels() reads ../../../../reference/models.json); falls back to its built-in DEFAULT_MODELS if absent. Speaks OpenAI-compatible chat only, so it runs huggingface, openrouter, and lan entries and skips native-SDK entries with a warning.
• mcp/tool-schemas/companion-space/ (Python, deploys as its own Hugging Face Space) — because the Space deploys from its own subfolder, it can't read a repo-root file at runtime, so it keeps a vendored copy companion-space/models.json that its selftest.py asserts is byte-for-byte equal to this file (the same drift-check discipline as mcp/tool-schemas/schemas.json). Its proxy serves the huggingface entries through the HF router; direct_providers.py (local-only) runs the anthropic/openai/google/moonshot entries through each vendor's native SDK or first-party endpoint. After editing this file, re-copy it over companion-space/models.json or that drift check fails.

This mirrors the project's single-source-of-truth pattern: edit the canonical file here; vendored copies are byte-equal mirrors guarded by a drift check, never hand-edited.

provider-endpoint-notes.md — operational endpoint learnings

Dated, provenance-linked record of vendor-endpoint behavior discovered while running experiments here: schema-validation gates (Moonshot's flavored-schema items: false rejection, Anthropic's top-level combinator gate), tool_choice/thinking policies (Kimi K3's required-only forcing, the GPT-5.6 reasoning gate), per-model quirks, and gateway footguns (the probe_hf.py scoped-run clobber). Read it before wiring a new arm against any of these endpoints; add to it (with a date + provenance pointer) whenever an experiment surfaces a new endpoint fact. This file exists so learnings live in the repo rather than in any one machine's or agent's local memory.

Historical registries