# Local Intelligence Eval Harness

Canonical home for the reconstructed Use Talkie local-intelligence eval pack.

What lives here:
- `cards.json`: 24 machine-graded Talkie workflow tasks
- `run_eval.py`: provider-agnostic runner
- `run_hf_transformers.py`: Hugging Face compute runner using `transformers`
- `run_hf_matrix.py`: batch runner for the recommended HF model matrix
- `launch_hf_job.py`: Hugging Face Jobs launcher for non-notebook cloud runs
- `Dockerfile`: custom dependency image for repeat HF Jobs runs
- `job-requirements.txt`: pinned runtime deps baked into the custom image
- `build_image.sh`: local helper to build the custom image
- `.github/workflows/build-local-intelligence-image.yml`: GitHub Actions workflow that publishes the image to GHCR
- `sync_to_hf.py`: uploads just this eval harness to a Hugging Face repo
- `upload_results.py`: uploads result JSON files back to the HF repo
- `grader.py`: assertion-based JSON grader
- `template.py`: prompt rendering helpers
- `providers/`: Apple, MLX, Ollama, and Hugging Face adapters
- `push_to_hf.py`: export pack for notebook or dataset use
- `hf_compute_quickstart.md`: copy-paste guide for HF Pro compute
- `notebook.ipynb`: optional notebook entrypoint

Quick examples:

```bash
python3 eval/local_intelligence/run_eval.py \
  --provider replay \
  --replay-file /tmp/local-intelligence-replay.jsonl \
  --limit 1 \
  --verbose

python3 eval/local_intelligence/launch_hf_job.py \
  --sync-code \
  --image pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel \
  --flavor a10g-small \
  --timeout 2h \
  --limit 24
```

## Hugging Face Compute

Run a single model on HF compute:

```bash
python3 eval/local_intelligence/run_hf_transformers.py \
  --model google/gemma-4-E4B-it \
  --limit 24 \
  --verbose
```

The HF compute runners record poor benchmark scores without failing the whole job.
Use `--fail-on-incomplete` only when you explicitly want non-passing evals to exit nonzero.

## Custom Image

Build a reusable Jobs image that already contains the runtime dependencies:

```bash
chmod +x eval/local_intelligence/build_image.sh
eval/local_intelligence/build_image.sh arach/local-intelligence-eval:latest
docker push arach/local-intelligence-eval:latest
```

Or publish it from GitHub Actions to GHCR:

```text
ghcr.io/<your-github-owner>/training-lab-local-intelligence-eval:latest
```

The workflow lives at:
- [.github/workflows/build-local-intelligence-image.yml](/Users/arach/dev/training-lab/.github/workflows/build-local-intelligence-image.yml)

It publishes on:
- manual `workflow_dispatch`
- pushes to `main` when the image files change

Then launch Jobs with that image:

```bash
python3 eval/local_intelligence/launch_hf_job.py \
  --sync-code \
  --image ghcr.io/<your-github-owner>/training-lab-local-intelligence-eval:latest \
  --flavor a10g-small \
  --timeout 2h \
  --limit 24
```

Run the default model matrix:

```bash
python3 eval/local_intelligence/run_hf_matrix.py \
  --limit 24 \
  --keep-going
```

Launch the cloud batch job on HF Jobs:

```bash
python3 eval/local_intelligence/launch_hf_job.py \
  --sync-code \
  --image ghcr.io/<your-github-owner>/training-lab-local-intelligence-eval:latest \
  --flavor a10g-small \
  --timeout 2h \
  --limit 24
```

Recommended first pass:
- `google/gemma-4-E4B-it`
- `nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1`
- `Qwen/Qwen3.5-4B-Instruct-2507`

If you want results in a separate repo:

```bash
python3 eval/local_intelligence/launch_hf_job.py \
  --sync-code \
  --source-repo-id arach/training-lab \
  --results-repo-id arach/training-lab-results \
  --image ghcr.io/<your-github-owner>/training-lab-local-intelligence-eval:latest \
  --flavor a10g-small \
  --timeout 2h \
  --limit 24
```