README.md

---
pipeline_tag: other
tags:
- eval-results
- health-check
- monitoring
- tool-calling
- sakthai
- companion
- artifact-store
- evaluation
task_categories:
- artifact-store
library_name: generic
license: mit
datasets:
- Nanthasit/sakthai-bench-v2
- Nanthasit/sakthai-bench-v3
model-index:
- name: SakThai Eval Results
  results:
  - task:
      type: evaluation
      name: artifact-store
    metrics:
    - name: artifacts
      type: other
      value: "4 YAML health/audit artifacts in .eval_results as of 2026-08-01"
    - name: last_probe
      type: other
      value: "2026-07-31 router probe: 0/14 Nanthasit/* healthy"
---

<div align="center">

# 🏭 SakThai Eval Results

[![eval-results](https://img.shields.io/badge/eval--results-%23000?style=flat&logo=huggingface)](https://huggingface.co/Nanthasit/eval_results)
[![last-commit](https://img.shields.io/github/last-commit/Nanthasit/eval_results?label=last+updated&logo=git)](https://huggingface.co/Nanthasit/eval_results)

**Centralized evaluation and health-check artifact store** for the SakThai model family. This companion repository holds automated evaluation outputs — health-check YAML files, benchmark raw dumps, and inference-probe results — generated by the nightly cron-job pipeline.

> 🏠 **This is not an inference model.** This repo stores evaluation artifacts only. There are no weight files, no Python packages, and no inference endpoints here.

</div>

---

## Repo Contents

| Path | Type | Size | Description |
|------|------|------:|-------------|
| `.eval_results/health-2026-07-31.yaml` | YAML | ~5.5 KB | Full-family health check (14 models, router 404 scan) |
| `.eval_results/health-context-0.5b-tools-2026-07-31.yaml` | YAML | ~2.8 KB | Model-specific health report for `context-0.5b-tools` |
| `README.md` | MD | — | This card |

More health checks and benchmark artifacts are appended daily by the automated cron pipeline.

---

## Model Description

This repository does **not** contain a machine-learning model. It is the evaluation and artifact store for the SakThai model family on Hugging Face. The repo stores timestamped YAML outputs from automated nightly jobs so model cards, dashboards, and auditors can cite first-party evidence instead of inferred claims.

- **Artifacts:** router health checks, model-specific YAML audits, benchmark raw dumps, inference-probe results.
- **Update cadence:** nightly cron pipeline.
- **Scope:** public SakThai models on the `Nanthasit` account.

---

## Evaluation Pipeline

Every SakThai nightly cycle runs two artifact-producing jobs:

1. **Family health check** — scans every public SakThai model via the HF router chat/completions endpoint and records HTTP status, latency, and error class. A `0/N healthy` result is expected because SakThai models are GGUF/LLaMA.cpp-oriented and are not deployed on the HF serverless inference router.
2. **Per-model health report** — model-specific YAML with repo hygiene, card quality, weight verification, cross-link audit, and issue list. First-party evidence that model cards are accurate and repos are clean.

Results land in `.eval_results/` as timestamped YAML files so model cards can reference them as verifiable sources.

---

## How to consume these artifacts

```python
import urllib.request
import yaml

url = "https://huggingface.co/Nanthasit/eval_results/raw/main/.eval_results/health-2026-07-31.yaml"
try:
    with urllib.request.urlopen(url, timeout=30) as r:
        report = yaml.safe_load(r.read().decode("utf-8"))
except Exception as exc:
    raise SystemExit(f"Failed to load artifact: {exc}")

print(report.get("summary", "<missing summary>"))
for model, info in report.get("models", {}).items():
    status = info.get("status")
    err = (info.get("error") or "")[:80]
    print(f"{model}\t{status}\t{err}")
```

Model cards link directly to these YAML files for Evaluation & Status sections; if a link breaks, the artifact store is the canonical fallback.

---

## Purpose

The raw YAML outputs live here so downstream cards and dashboards can cite `.eval_results/*.yaml` as first-class evidence. This repo is the single source of truth for:

- **Router inference health** — whether each model is reachable via HF router
- **Responses API compatibility** — whether non-chat models respond through HF Responses API
- **Download & velocity tracking** — per-model counts, daily velocity, family rank
- **Repo hygiene audits** — card quality, broken links, missing metadata, weight checks

---

## Benchmarks

### Router probe on 2026-07-31

| Model | Router status | Latency | Interpretation |
|------|------:|------:|------|
| `Nanthasit/sakthai-context-0.5b-merged` | 404 | 0.18s | Not deployed on HF router |
| `Nanthasit/sakthai-context-0.5b-tools` | 404 | 0.18s | Not deployed on HF router |
| `Nanthasit/sakthai-context-1.5b-merged` | 404 | 0.20s | Not deployed on HF router |
| `Nanthasit/sakthai-context-1.5b-merged-v2` | 404 | 0.19s | Not deployed on HF router |
| `Nanthasit/sakthai-context-1.5b-tools` | 404 | 0.21s | Not deployed on HF router |
| `Nanthasit/sakthai-context-1.5b-tools-v2` | 404 | 0.18s | Not deployed on HF router |
| `Nanthasit/sakthai-context-7b-merged` | 404 | 0.19s | Not deployed on HF router |
| `Nanthasit/sakthai-context-7b-tools` | 404 | 0.25s | Not deployed on HF router |
| `Nanthasit/sakthai-plus-1.5b` | 404 | 0.27s | Not deployed on HF router |
| `Nanthasit/sakthai-coder-1.5b` | 404 | 0.28s | Not deployed on HF router |
| `Nanthasit/sakthai-coder-browser` | 404 | 0.23s | Not deployed on HF router |
| `Nanthasit/sakthai-vision-7b` | 404 | 0.21s | Not deployed on HF router |
| `Nanthasit/sakthai-tts-model` | 404 | 0.22s | Not deployed on HF router |
| `Nanthasit/sakthai-embedding-multilingual` | 404 | 0.19s | Not deployed on HF router |
| `google/gemma-3-4b-it` | 200 | 0.73s | Control probe: router is functional |

> Expected outcome: `0/14 healthy` for `Nanthasit/*` on the HF router. These models are built for local/offline inference with llama.cpp/Ollama, not serverless inference providers.

## Evaluation scope

### What this repo does and does not measure

- **Does:** router reachability, basic HTTP semantics, artifact hygiene, metadata completeness
- **Does not:** local CPU/GPU latency, GGUF quantization accuracy, offline inference benchmarks, end-to-end tool-call pass rates

---

## Related repos

| Repo | Type |
|------|------|
| [Nanthasit/sakthai-context-1.5b-merged](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged) | Flagship tool-calling model |
| [Nanthasit/sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v2) | Tool-calling benchmark v2 |
| [Nanthasit/sakthai-bench-v3](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v3) | Next-gen benchmark scaffold |
| [Nanthasit/sakthai-pipeline](https://huggingface.co/Nanthasit/sakthai-pipeline) | Training & inference pipeline |
| [Nanthasit/sakthai-model-family](https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02) | Full collection |

---

## Status summary

This repo now documents the four current eval artifacts for downstream cards: `health-2026-07-31.yaml`, `health-context-0.5b-tools-2026-07-31.yaml`, and two metadata-cron snapshots dated `2026-07-31` and `2026-08-01`.

## Limitations

- **No weights / no inference** — this is an artifact store only.
- **Router-centric** — health checks probe HF router endpoints; local/offline inference results are not produced here.
- **0/14 healthy today** — expected, but not a signal of model breakage.
- **Scope-limited artifacts** — current outputs are health checks and YAML metadata; benchmark result dumps are still being onboarded from v2/v3 runs.
- **API dependency** — download counts and router checks depend on HF API availability; maintenance of this repo does not require external compute.

---

## Citation

If these evaluation artifacts are useful in your research or tooling, please cite the SakThai family release:

```bibtex
@misc{sakthai-eval-results,
  author       = {Nanthasit Burankum},
  title        = {{SakThai Eval Results — Automated Health and Benchmark Artifact Store}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanthasit/eval_results}},
  note         = {Health-check YAML, metadata audits, family benchmark evidence, and router probe logs}
}
```

---

## License

MIT — data artifacts only, no model weights.