Add GGUF quantized builds (Q4_K_M, Q8_0) for llama.cpp on-device inference

Converted from merged-vuln-detector/model.safetensors with convert_hf_to_gguf.py and quantized with llama-quantize. Greedy-decoding outputs verified identical to the original transformers model. Model card updated with GGUF usage and measured Apple M1 performance.

.gitattributes

@@ -34,3 +34,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 merged-vuln-detector/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+vuln_detector-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+vuln_detector-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -13,6 +13,8 @@ metrics:
 - BLEU
 tags:
 - llama-3.2-1B-Instruct
+- gguf
+- llama.cpp
 ---
 
 # Model Card for `merged-vuln-detector`
@@ -129,6 +131,40 @@ int main() {
 **Model Output:**
 > The code has a buffer overflow vulnerability due to the lack of bounds checking on the destination buffer size.
 
+## GGUF / llama.cpp (On-device Inference)
+
+Quantized GGUF builds of `merged-vuln-detector` are provided for on-device inference with [llama.cpp](https://github.com/ggml-org/llama.cpp). They were converted from `merged-vuln-detector/model.safetensors` with `convert_hf_to_gguf.py` and quantized with `llama-quantize`.
+
+| File | Quantization | Size | Notes |
+|------|--------------|------|-------|
+| `vuln_detector-Q4_K_M.gguf` | Q4_K_M | 0.81 GB | Recommended for on-device use |
+| `vuln_detector-Q8_0.gguf`   | Q8_0   | 1.32 GB | Near-lossless |
+
+Outputs were verified against the original safetensors model: under greedy decoding, the Q8_0 and Q4_K_M builds produce identical analyses to the `transformers` model.
+
+**Measured performance** (Apple M1, Metal backend, Q4_K_M): ~47 tokens/s generation, ~178 tokens/s prompt processing — real-time interactive inference on a consumer laptop.
+
+### Run with llama.cpp
+
+```bash
+llama-cli -hf cycloevan/vuln_detector:Q4_K_M \
+  -p "Analyze the security vulnerabilities in the following code.\n\n<CODE>\n\nAnalysis:\n" \
+  -n 256 --temp 0
+```
+
+### Run with llama-cpp-python
+
+```python
+from llama_cpp import Llama
+
+llm = Llama.from_pretrained("cycloevan/vuln_detector", filename="vuln_detector-Q4_K_M.gguf")
+
+code = "def login(u, p): cursor.execute(f\"SELECT * FROM users WHERE name='{u}' AND pw='{p}'\")"
+prompt = f"Analyze the security vulnerabilities in the following code.\n\n{code}\n\nAnalysis:\n"
+out = llm(prompt, max_tokens=256, temperature=0)
+print(out["choices"][0]["text"])
+```
+
 ## Model Card Authors
 
 [Seokhee Chang]

vuln_detector-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a03d80a47ae5504ca1e7d3a51d62bbf90b0f2e3a7d261671236f37368dd0e656
+size 807693984

vuln_detector-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff758f786ff0f6dbeca282e88fe1cd99b587105e6454b0d8c6f0cd7a9b0dd1c6
+size 1321082528