Clarify model limitations and eval status

README.md

@@ -1,146 +1,75 @@
-# HackIDLE-NIST-Coder (GGUF)
-
-A specialized cybersecurity LLM fine-tuned on 568 NIST publications, optimized for Ollama and llama.cpp.
-
-## Model Details
-
-**Base Model:** Qwen2.5-Coder-7B-Instruct
-**Fine-tuning:** LoRA (11.5M parameters, 0.151% of base)
-**Training Data:** 568 NIST cybersecurity documents (523,706 examples)
-**Context Length:** 32,768 tokens
-**License:** Apache 2.0
-
-## Quantization Variants
-
-| File | Size | Use Case | Perplexity |
-|------|------|----------|------------|
-| `hackidle-nist-coder-f16.gguf` | 14GB | Reference/source | Baseline |
-| `hackidle-nist-coder-q8_0.gguf` | 7.5GB | Highest quality | ~0.1% loss |
-| `hackidle-nist-coder-q5_k_m.gguf` | 5.1GB | High quality | ~0.5% loss |
-| **`hackidle-nist-coder-q4_k_m.gguf`** | **4.4GB** | **Recommended** | ~1% loss |
-
-## Usage
-
-### With Ollama
-
-Download and run:
-```bash
-ollama run ethanolivertroy/hackidle-nist-coder
-```
-
-Or create from this repo:
-```bash
-# Download GGUF
-wget https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-GGUF/resolve/main/hackidle-nist-coder-q4_k_m.gguf
-
-# Create Modelfile
-cat > Modelfile << 'EOF'
-FROM ./hackidle-nist-coder-q4_k_m.gguf
+---
+license: apache-2.0
+license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/blob/main/LICENSE
+base_model: mlx-community/Qwen2.5-Coder-7B-Instruct-4bit
+tags:
+- gguf
+- cybersecurity
+- nist
+- security-controls
+- compliance
+- fine-tuned
+- llama-cpp
+language:
+- en
+quantized_by: ethanolivertroy
+---
 
-SYSTEM """You are HackIDLE-NIST-Coder, a cybersecurity expert with deep knowledge of NIST standards, frameworks, and best practices."""
-
-PARAMETER temperature 0.7
-PARAMETER num_ctx 32768
-EOF
+# HackIDLE-NIST-Coder (GGUF)
 
-# Create model
-ollama create hackidle-nist-coder -f Modelfile
-```
+This is the first GGUF build of HackIDLE-NIST-Coder, a NIST-focused local model built from Qwen2.5-Coder-7B-Instruct and fine-tuned on a NIST cybersecurity corpus.
 
-### With llama.cpp
+This repo is kept for reproducibility. For new testing, start with the v1.1 GGUF build:
 
-```bash
-# Download GGUF
-wget https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-GGUF/resolve/main/hackidle-nist-coder-q4_k_m.gguf
+- [ethanolivertroy/HackIDLE-NIST-Coder-v1.1-GGUF](https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-v1.1-GGUF)
 
-# Run inference
-./llama-cli -m hackidle-nist-coder-q4_k_m.gguf \
-    -p "What is Zero Trust Architecture according to NIST?" \
-    -n 200 \
-    --temp 0.7
-```
+Use this model as a helper. Do not treat it as a source of truth for exact control names, RMF step lists, or reference-architecture component names without checking the source publication.
 
-### With LM Studio
+## Training data
 
-1. Search for "hackidle-nist-coder" in LM Studio
-2. Download Q4_K_M variant
-3. Start chatting!
+This first build used `523,706` examples from `568` NIST cybersecurity documents.
 
-Or use the [MLX version](https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-MLX-4bit) for native Apple Silicon support.
+Training dataset:
 
-## Expertise Areas
+- [ethanolivertroy/nist-cybersecurity-training](https://huggingface.co/datasets/ethanolivertroy/nist-cybersecurity-training)
 
-- NIST Cybersecurity Framework (CSF)
-- Risk Management Framework (RMF)
-- SP 800 series security controls (AC, AU, CA, CM, CP, IA, IR, MA, MP, PE, PL, PS, RA, SA, SC, SI, SR)
-- FIPS cryptographic standards
-- Zero Trust Architecture (SP 800-207)
-- Cloud security (SP 800-210, SP 800-144)
-- Supply chain risk management (SP 800-161)
-- Privacy Framework
+## Current eval status
 
-## Example Queries
+The dated smoke eval from April 22, 2026 was run against the Ollama `latest` tag, which matched the v1.1 line in the local install used for that check. I have not rerun that exact eval against this older GGUF build.
 
-```
-"What is Zero Trust Architecture according to NIST SP 800-207?"
-"Explain control AC-1 from NIST SP 800-53."
-"What are the core components of the NIST Cybersecurity Framework?"
-"How does NIST recommend implementing secure cloud architecture?"
-"What is the Risk Management Framework process?"
-```
+The v1.1 result matters for this older build too because it sets the right expectation for the model family: the model can stay in-domain while still missing exact NIST structure.
 
-## Training Details
+Be careful with:
 
-**Dataset:** [`ethanolivertroy/nist-cybersecurity-training`](https://huggingface.co/datasets/ethanolivertroy/nist-cybersecurity-training)
-- 523,706 training examples
-- 568 source documents
-- Smart chunking with sentence boundaries
-- 5 extraction strategies: sections, controls, definitions, tables, semantic chunks
+- exact control names
+- exact RMF step ordering
+- exact SP 800-207 component naming
+- source-level answers that need to be right on the first pass
 
-**Fine-tuning:**
-- Method: LoRA with MLX (Apple Silicon)
-- Training time: 3.5 hours on M4 Max
-- Iterations: 1000
-- Validation loss improvement: 45%
-- Base model: Qwen2.5-Coder-7B-Instruct-4bit
+## Available quantizations
 
-## Performance
+| Quantization | Approx. size | Use case |
+| --- | ---: | --- |
+| F16 | 14 GB | Full precision reference build |
+| Q8_0 | 7.5 GB | Higher quality local inference |
+| Q5_K_M | 5.1 GB | Balanced size and quality |
+| Q4_K_M | 4.4 GB | Small local default for most machines |
 
-**Ollama (M4 Max, Q4_K_M):**
-- Inference: 80-100 tokens/sec
-- Memory: ~6GB
-- Prompt processing: 50-100 tokens/sec
+## Run with llama.cpp
 
-**llama.cpp (M4 Max, Q4_K_M):**
-- Inference: 70-90 tokens/sec
-- Memory: ~5GB
-
-## Related Models
-
-- **MLX Format:** [`ethanolivertroy/HackIDLE-NIST-Coder-MLX-4bit`](https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-MLX-4bit)
-- **LM Studio:** [`ethanolivertroy/hackidle-nist-coder`](https://lmstudio.ai/ethanolivertroy/hackidle-nist-coder)
-- **Ollama Library:** `ethanolivertroy/hackidle-nist-coder` (coming soon)
-
-## Citation
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+cd llama.cpp
+make
 
-If you use this model in your research or applications, please cite:
+wget https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-GGUF/resolve/main/hackidle-nist-coder-q4_k_m.gguf
 
-```bibtex
-@software{hackidle_nist_coder,
-  author = {Ethan Oliver Troy},
-  title = {HackIDLE-NIST-Coder: A Fine-Tuned LLM for NIST Cybersecurity Standards},
-  year = {2025},
-  url = {https://huggingface.co/ethanolivertroy/HackIDLE-NIST-Coder-GGUF}
-}
+./llama-cli \
+  -m hackidle-nist-coder-q4_k_m.gguf \
+  -p "Which NIST docs would you start with for contractor remote access?" \
+  -n 500
 ```
 
 ## License
 
-This model is released under the Apache 2.0 license. NIST publications are in the public domain.
-
-## Acknowledgments
-
-- **NIST** for publishing comprehensive cybersecurity guidance
-- **Qwen Team** for the exceptional Qwen2.5-Coder base model
-- **llama.cpp** team for GGUF format and quantization
-- **Ollama** for making local LLM deployment accessible
+The base model is Qwen2.5-Coder-7B-Instruct, released under Apache 2.0. The NIST source publications used for the dataset are public domain U.S. government works. This model card uses Apache 2.0 for the model artifact and documents the NIST data source separately.