docs: add model card with eval metrics on held-out test set

README.md

@@ -1,207 +1,209 @@
 ---
+license: apache-2.0
 base_model: Qwen/Qwen3.5-2B
 library_name: peft
 pipeline_tag: text-generation
+language:
+  - en
 tags:
-- base_model:adapter:Qwen/Qwen3.5-2B
-- lora
-- transformers
+  - lora
+  - peft
+  - qwen
+  - guardrails
+  - code-detection
+  - language-identification
+  - multi-label-classification
+  - quantization
+  - 8-bit
+metrics:
+  - accuracy
+  - f1
+  - precision
+  - recall
+model-index:
+  - name: PromptInjection-Qwen3.5-2B-v5
+    results:
+      - task:
+          type: text-classification
+          name: Multi-label Programming Language Identification
+        dataset:
+          name: LangID Guard Held-out Test Set
+          type: custom
+        metrics:
+          - type: accuracy
+            name: is_valid accuracy
+            value: 1.0000
+          - type: accuracy
+            name: language-set exact match
+            value: 0.9600
+          - type: f1
+            name: binary F1 (positive=contains code)
+            value: 1.0000
+          - type: f1
+            name: macro F1 over languages
+            value: 0.9696
+          - type: precision
+            name: binary precision (positive=contains code)
+            value: 1.0000
+          - type: recall
+            name: binary recall (positive=contains code)
+            value: 1.0000
 ---
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
+# PromptInjection-Qwen3.5-2B-v5
+LoRA adapter for **Qwen/Qwen3.5-2B** that identifies which programming languages are embedded in a user prompt across **25 languages and configuration formats**. Trained on a combined dataset of Rosetta Code snippets and curated config-language samples (Dockerfile, YAML, Terraform, Makefile, SQL).
+The model is fine-tuned to emit a strict JSON object describing the languages found:
+
+```json
+{"is_valid": true, "category": {"Python": true, "Bash": true}}
+```
+
+`is_valid` is `true` when at least one code/config snippet is present and `false` for natural-language-only prompts. `category` contains only the detected languages, each mapped to `true`; if no code is present `category` is `{}`.
+## Quick start
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch, json, re
+
+BASE = "Qwen/Qwen3.5-2B"
+ADAPTER = "Accuknoxtechnologies/PromptInjection-Qwen3.5-2B-v5"
+
+SYSTEM_MSG = """You are a code language identifier. For the given user prompt, decide whether it contains any embedded source code (program source or recognizable code-like configuration). Output exactly one JSON object and nothing else: {"is_valid": <true|false>, "category": {"<Lang>": true, ...}}.
+No preamble. No explanation. No <think> tags. No markdown code fences. No trailing prose.
+Rules:
+  - is_valid is TRUE when the prompt contains at least one code/config snippet, FALSE when the prompt is plain natural-language only.
+  - category contains ONLY the languages that appear, each mapped to true. If no code is present, category is the empty object {}.
+  - When multiple languages appear, list every distinct one (still only true).
+Allowed language keys (use these exact spellings):
+  Python, JavaScript, Java, C, C++, C#, Go, Rust, Kotlin, Swift, Ruby, R, Scala, Perl, Lua, Bash, PowerShell, Batch, SQL, Dockerfile, YAML, Makefile, Terraform, AWK, jq
+
+Examples:
+
+Input: What's the weather forecast today?
+Output: {"is_valid": false, "category": {}}
+
+Input: Run this for me: print('hello world')
+Output: {"is_valid": true, "category": {"Python": true}}
+
+Input: Compare these — SELECT * FROM users vs the snippet: console.log(users)
+Output: {"is_valid": true, "category": {"SQL": true, "JavaScript": true}}"""
+
+tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    BASE, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
+)
+model = PeftModel.from_pretrained(model, ADAPTER); model.eval()
+
+def langid(prompt: str) -> dict:
+    chat = tokenizer.apply_chat_template(
+        [{"role":"system","content":SYSTEM_MSG},
+         {"role":"user","content":prompt}],
+        tokenize=False, add_generation_prompt=True, enable_thinking=False)
+    inputs = tokenizer(chat, return_tensors="pt").to(model.device)
+    out = model.generate(**inputs, max_new_tokens=220, do_sample=False)
+    text = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+    return json.loads(re.search(r'\{.*\}', text, re.DOTALL).group(0))
+```
+
+## System prompt
+The model was trained with the exact system prompt below. Pass it verbatim at inference time — the output schema depends on this prompt.
+
+```text
+You are a code language identifier. For the given user prompt, decide whether it contains any embedded source code (program source or recognizable code-like configuration). Output exactly one JSON object and nothing else: {"is_valid": <true|false>, "category": {"<Lang>": true, ...}}.
+No preamble. No explanation. No <think> tags. No markdown code fences. No trailing prose.
+Rules:
+  - is_valid is TRUE when the prompt contains at least one code/config snippet, FALSE when the prompt is plain natural-language only.
+  - category contains ONLY the languages that appear, each mapped to true. If no code is present, category is the empty object {}.
+  - When multiple languages appear, list every distinct one (still only true).
+Allowed language keys (use these exact spellings):
+  Python, JavaScript, Java, C, C++, C#, Go, Rust, Kotlin, Swift, Ruby, R, Scala, Perl, Lua, Bash, PowerShell, Batch, SQL, Dockerfile, YAML, Makefile, Terraform, AWK, jq
+
+Examples:
+
+Input: What's the weather forecast today?
+Output: {"is_valid": false, "category": {}}
+
+Input: Run this for me: print('hello world')
+Output: {"is_valid": true, "category": {"Python": true}}
+
+Input: Compare these — SELECT * FROM users vs the snippet: console.log(users)
+Output: {"is_valid": true, "category": {"SQL": true, "JavaScript": true}}
+```
 ## Evaluation
+Evaluated on **200 held-out prompts** drawn from `test_dataset_langid.csv` (same single + multi + benign composition as training).
+
+- Evaluation timestamp: `2026-05-22 00:42 UTC`
+- GPU: `NVIDIA A10G`
+- Source adapter: `Accuknoxtechnologies/PromptInjection-Qwen3.5-2B-v5`
+- JSON parse errors: `0/200` (`0.0%`)
+### Top-level metrics
+| Metric | Value |
+|---|---:|
+| `is_valid` accuracy | **1.0000** |
+| Language-set exact match | **0.9600** |
+| Binary F1 (positive = contains code) | **1.0000** |
+| Binary precision | 1.0000 |
+| Binary recall | 1.0000 |
+| Macro F1 across languages | **0.9696** |
+### Confusion matrix — binary `is_valid` decision
+Positive class = the prompt **contains code** (`is_valid=True`).
+
+| | predicted contains-code | predicted no-code |
+|---|---:|---:|
+| **actual contains-code** | TP = 181 | FN = 0 |
+| **actual no-code**       | FP = 0 | TN = 19 |
+### Per-language metrics
+Only languages that appear in either the actual or predicted labels are listed.
+
+| Language | support | precision | recall | F1 |
+|---|---:|---:|---:|---:|
+| `Python` | 14 | 1.000 | 1.000 | 1.000 |
+| `Terraform` | 14 | 1.000 | 1.000 | 1.000 |
+| `Java` | 12 | 1.000 | 1.000 | 1.000 |
+| `C` | 12 | 1.000 | 1.000 | 1.000 |
+| `Rust` | 12 | 1.000 | 1.000 | 1.000 |
+| `AWK` | 12 | 1.000 | 0.917 | 0.957 |
+| `Ruby` | 11 | 0.917 | 1.000 | 0.957 |
+| `R` | 11 | 1.000 | 1.000 | 1.000 |
+| `Go` | 10 | 1.000 | 0.900 | 0.947 |
+| `Swift` | 10 | 1.000 | 0.900 | 0.947 |
+| `Scala` | 10 | 1.000 | 0.800 | 0.889 |
+| `SQL` | 10 | 1.000 | 1.000 | 1.000 |
+| `jq` | 10 | 0.909 | 1.000 | 0.952 |
+| `JavaScript` | 9 | 0.900 | 1.000 | 0.947 |
+| `Kotlin` | 9 | 1.000 | 1.000 | 1.000 |
+| `Perl` | 9 | 1.000 | 1.000 | 1.000 |
+| `PowerShell` | 9 | 1.000 | 1.000 | 1.000 |
+| `Batch` | 9 | 1.000 | 1.000 | 1.000 |
+| `YAML` | 9 | 1.000 | 0.889 | 0.941 |
+| `C++` | 7 | 1.000 | 0.857 | 0.923 |
+| `C#` | 7 | 0.875 | 1.000 | 0.933 |
+| `Lua` | 7 | 1.000 | 0.857 | 0.923 |
+| `Bash` | 7 | 1.000 | 1.000 | 1.000 |
+| `Dockerfile` | 6 | 0.857 | 1.000 | 0.923 |
+| `Makefile` | 6 | 1.000 | 1.000 | 1.000 |
+
+### Inference latency
+- Mean: **0.99 s/prompt**
+- Median: 0.94 s/prompt
+- p95: 1.35 s/prompt
+- Max: 1.63 s/prompt
+
+## Training setup
+- Base model: `Qwen/Qwen3.5-2B` (loaded in full precision (bf16 / fp16, no `bitsandbytes` quantization))
+- LoRA: r=16, alpha=32, dropout=0.05, target modules = {q,k,v,o,gate,up,down}_proj
+- Optimizer: adamw_torch, lr=1e-4, cosine schedule, warmup 5%
+- Precision: bf16 if available, else fp16
+- Effective batch size: 8 (per-device 1 + grad-accum 8), gradient checkpointing on
+- Max sequence length: 3200 tokens
+- Training data: 10,000 rows  (7,000 single-language + 2,000 multi-language + 1,000 benign)
+- Languages: 25 (programming + config formats)
+
+## Supported languages
+The model emits one or more of these keys in the `category` map of its JSON output:
+
+```
+Python, JavaScript, Java, C, C++, C#, Go, Rust, Kotlin, Swift, Ruby, R, Scala, Perl, Lua, Bash, PowerShell, Batch, SQL, Dockerfile, YAML, Makefile, Terraform, AWK, jq
+```
 
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
-### Framework versions
-
-- PEFT 0.19.1
+---
+*Model card generated automatically by `eval_and_push_card.py` on 2026-05-22 00:42 UTC.*