README.md

metadata

pipeline_tag: other
tags:
  - eval-results
  - health-check
  - monitoring
  - tool-calling
  - sakthai
  - companion
  - artifact-store
  - evaluation
task_categories:
  - artifact-store
library_name: generic
license: mit
datasets:
  - Nanthasit/sakthai-bench-v2
  - Nanthasit/sakthai-bench-v3
model-index:
  - name: SakThai Eval Results
    results:
      - task:
          type: evaluation
          name: artifact-store
        metrics:
          - name: artifacts
            type: other
            value: 4 YAML health/audit artifacts in .eval_results as of 2026-08-01
          - name: last_probe
            type: other
            value: '2026-07-31 router probe: 0/14 Nanthasit/* healthy'

🏭 SakThai Eval Results

Centralized evaluation and health-check artifact store for the SakThai model family. This companion repository holds automated evaluation outputs — health-check YAML files, benchmark raw dumps, and inference-probe results — generated by the nightly cron-job pipeline.

🏠 This is not an inference model. This repo stores evaluation artifacts only. There are no weight files, no Python packages, and no inference endpoints here.

---

Repo Contents

Path	Type	Size	Description
.eval_results/health-2026-07-31.yaml	YAML	~5.5 KB	Full-family health check (14 models, router 404 scan)
.eval_results/health-context-0.5b-tools-2026-07-31.yaml	YAML	~2.8 KB	Model-specific health report for context-0.5b-tools
README.md	MD	—	This card

More health checks and benchmark artifacts are appended daily by the automated cron pipeline.

---

Model Description

This repository does not contain a machine-learning model. It is the evaluation and artifact store for the SakThai model family on Hugging Face. The repo stores timestamped YAML outputs from automated nightly jobs so model cards, dashboards, and auditors can cite first-party evidence instead of inferred claims.

• Artifacts: router health checks, model-specific YAML audits, benchmark raw dumps, inference-probe results.
• Update cadence: nightly cron pipeline.
• Scope: public SakThai models on the Nanthasit account.

---

Evaluation Pipeline

Every SakThai nightly cycle runs two artifact-producing jobs:

1. Family health check — scans every public SakThai model via the HF router chat/completions endpoint and records HTTP status, latency, and error class. A 0/N healthy result is expected because SakThai models are GGUF/LLaMA.cpp-oriented and are not deployed on the HF serverless inference router.
2. Per-model health report — model-specific YAML with repo hygiene, card quality, weight verification, cross-link audit, and issue list. First-party evidence that model cards are accurate and repos are clean.

Results land in .eval_results/ as timestamped YAML files so model cards can reference them as verifiable sources.

---

How to consume these artifacts

import urllib.request
import yaml

url = "https://huggingface.co/Nanthasit/eval_results/raw/main/.eval_results/health-2026-07-31.yaml"
try:
    with urllib.request.urlopen(url, timeout=30) as r:
        report = yaml.safe_load(r.read().decode("utf-8"))
except Exception as exc:
    raise SystemExit(f"Failed to load artifact: {exc}")

print(report.get("summary", "<missing summary>"))
for model, info in report.get("models", {}).items():
    status = info.get("status")
    err = (info.get("error") or "")[:80]
    print(f"{model}\t{status}\t{err}")

Model cards link directly to these YAML files for Evaluation & Status sections; if a link breaks, the artifact store is the canonical fallback.

---

Purpose

The raw YAML outputs live here so downstream cards and dashboards can cite .eval_results/*.yaml as first-class evidence. This repo is the single source of truth for:

• Router inference health — whether each model is reachable via HF router
• Responses API compatibility — whether non-chat models respond through HF Responses API
• Download & velocity tracking — per-model counts, daily velocity, family rank
• Repo hygiene audits — card quality, broken links, missing metadata, weight checks

---

Benchmarks

Router probe on 2026-07-31

Model	Router status	Latency	Interpretation
Nanthasit/sakthai-context-0.5b-merged	404	0.18s	Not deployed on HF router
Nanthasit/sakthai-context-0.5b-tools	404	0.18s	Not deployed on HF router
Nanthasit/sakthai-context-1.5b-merged	404	0.20s	Not deployed on HF router
Nanthasit/sakthai-context-1.5b-merged-v2	404	0.19s	Not deployed on HF router
Nanthasit/sakthai-context-1.5b-tools	404	0.21s	Not deployed on HF router
Nanthasit/sakthai-context-1.5b-tools-v2	404	0.18s	Not deployed on HF router
Nanthasit/sakthai-context-7b-merged	404	0.19s	Not deployed on HF router
Nanthasit/sakthai-context-7b-tools	404	0.25s	Not deployed on HF router
Nanthasit/sakthai-plus-1.5b	404	0.27s	Not deployed on HF router
Nanthasit/sakthai-coder-1.5b	404	0.28s	Not deployed on HF router
Nanthasit/sakthai-coder-browser	404	0.23s	Not deployed on HF router
Nanthasit/sakthai-vision-7b	404	0.21s	Not deployed on HF router
Nanthasit/sakthai-tts-model	404	0.22s	Not deployed on HF router
Nanthasit/sakthai-embedding-multilingual	404	0.19s	Not deployed on HF router
google/gemma-3-4b-it	200	0.73s	Control probe: router is functional

Expected outcome: 0/14 healthy for Nanthasit/* on the HF router. These models are built for local/offline inference with llama.cpp/Ollama, not serverless inference providers.

Evaluation scope

What this repo does and does not measure

• Does: router reachability, basic HTTP semantics, artifact hygiene, metadata completeness
• Does not: local CPU/GPU latency, GGUF quantization accuracy, offline inference benchmarks, end-to-end tool-call pass rates

---

Related repos

Repo	Type
Nanthasit/sakthai-context-1.5b-merged	Flagship tool-calling model
Nanthasit/sakthai-bench-v2	Tool-calling benchmark v2
Nanthasit/sakthai-bench-v3	Next-gen benchmark scaffold
Nanthasit/sakthai-pipeline	Training & inference pipeline
Nanthasit/sakthai-model-family	Full collection

---

Status summary

This repo now documents the four current eval artifacts for downstream cards: health-2026-07-31.yaml, health-context-0.5b-tools-2026-07-31.yaml, and two metadata-cron snapshots dated 2026-07-31 and 2026-08-01.

Limitations

• No weights / no inference — this is an artifact store only.
• Router-centric — health checks probe HF router endpoints; local/offline inference results are not produced here.
• 0/14 healthy today — expected, but not a signal of model breakage.
• Scope-limited artifacts — current outputs are health checks and YAML metadata; benchmark result dumps are still being onboarded from v2/v3 runs.
• API dependency — download counts and router checks depend on HF API availability; maintenance of this repo does not require external compute.

---

Citation

If these evaluation artifacts are useful in your research or tooling, please cite the SakThai family release:

@misc{sakthai-eval-results,
  author       = {Nanthasit Burankum},
  title        = {{SakThai Eval Results — Automated Health and Benchmark Artifact Store}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanthasit/eval_results}},
  note         = {Health-check YAML, metadata audits, family benchmark evidence, and router probe logs}
}

---

License

MIT — data artifacts only, no model weights.