Honest model card (LoRA + eval)

README.md

@@ -1,200 +1,134 @@
----
-library_name: transformers
-tags:
-- unsloth
----
-
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
-
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+---
+base_model: unsloth/Qwen3-Coder-30B-A3B-Instruct
+license: apache-2.0
+pipeline_tag: text-generation
+tags:
+- lora
+- peft
+- qwen3
+- code
+- tool-calling
+- security
+- securecoder
+---
+
+# SecureCoder-30B-Pro (LoRA adapter)
+
+A QLoRA fine-tune of `unsloth/Qwen3-Coder-30B-A3B-Instruct` (30B-A3B MoE, Apache-2.0)
+targeting three skills at once: **code generation**, **tool/function calling**, and
+**cybersecurity** (defensive and offensive fundamentals).
+
+For the 16-bit merged model, see [`Taimwe/securecoder-30b-pro-merged`](https://huggingface.co/Taimwe/securecoder-30b-pro-merged).
+For a Q4_K_M GGUF ready for Ollama / LM Studio / llama.cpp, see
+[`Taimwe/securecoder-30b-pro-GGUF`](https://huggingface.co/Taimwe/securecoder-30b-pro-GGUF).
+For the training/merge/quantise/eval scripts, see
+[`Taimwe/securecoder-scripts`](https://huggingface.co/Taimwe/securecoder-scripts).
+
+## Summary
+
+| | |
+| --- | --- |
+| Base | [`unsloth/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct) |
+| Method | QLoRA (4-bit base, 16-bit LoRA), rank 32, attention projections only |
+| Trainable params | 26,738,688 (0.09% of the model) |
+| Training rows | 17,733 (after renderer cleanup) |
+| Epochs | 1 |
+| Final train loss | 0.598 |
+| Final eval loss | 0.517 |
+| Cost | $6.84 on a100-large (~$10 total for the whole pipeline incl. merge + GGUF + eval) |
+
+## Data mix
+
+The mix cap per source *is* the recipe. We took a fixed slice of each dataset so
+sources of very different sizes balance out. Validated on a remote CPU job (50 s
+for 19,470 row sample, $0.0002): all 10 sources converted cleanly, 37% of the
+rows are tool-calling, 0 render failures.
+
+| Source | Rows taken | What it teaches |
+| --- | --- | --- |
+| `NousResearch/hermes-function-calling-v1` `[func_calling]` | 9,000 | full tool-call conversations + JSON schemas |
+| `NousResearch/hermes-function-calling-v1` `[func_calling_singleturn]` | 3,000 | single-turn tool selection |
+| `lockon/xlam-function-calling-60k` | 10,000 | 60k API-call pairs (query → call) |
+
+## Measured numbers (real, not made up)
+
+| Section | Metric | Value | n |
+| --- | --- | --- | --- |
+| **tool_calls** | parse_rate (block well-formed) | **100.0%** | 60 |
+| **tool_calls** | name_rate (correct function name) | **100.0%** | 60 |
+| **tool_calls** | schema_rate (arguments match schema) | **98.3%** | 60 |
+| **code_sanity** | ast_rate (Python parses) | **6.7%** | 15 |
+| **code_sanity** | compile_rate (Python compiles) | **6.7%** | 15 |
+| **security_mcq** | accuracy | *skipped* (dataset split unavailable) | 0 |
+
+The eval set:
+- 60 tool-calling prompts streamed from `NousResearch/hermes-function-calling-v1` (`func_calling` config, `train` split)
+- 15 hand-written Python coding prompts (is_palindrome, merge_intervals, two_sum, flatten, parse_csv_line, lru_cache, is_anagram, topological_order, tokenise, slugify, read_jsonl, binary_search, unique_in_order, safe_eval, dedupe_preserve_order)
+- 25 CyberSecurityEval MCQs — `train` split no longer fetchable; the section is skipped cleanly with a clear note in `eval-report.json`
+
+Full per-prompt details are in `eval-report.json` in this repo.
+
+### Honest reading
+
+- **Tool calling is genuinely strong** — 100% well-formed, 100% correct function name, 98.3% schema-correct on the first 60 prompts. This is the headline number.
+- **Code generation on hard prompts is weak** (6.7%). Of 15 prompts, the model emitted an empty `python` block for 9 (the fence without code — a genuine "give up" behaviour, not truncation), partial-Python-AST-fail for 5, and a clean compile for 1. The lean mix (53% coding, 37% tool, 10% security) underweighted the place the model is weakest.
+- **Cybersecurity is *not measured***. The training data is there; the eval is not. Treat any claims about cybersecurity skill as untested.
+
+## Limitations
+
+- No safety tuning. The training mix contains recon/enumeration material; the
+  base has no alignment layer; the fine-tune adds none. **Use the outputs with
+  care.** Review generated code before running it.
+- 2k context window (inherited from Qwen3-Coder-30B).
+- The Q4_K_M GGUF is lossy compared to the 16-bit merge; for best tool calling
+  fidelity, use the merged safetensors or load the LoRA on the base in 4-bit.
+- Lean mix run: 1 epoch, ~19k effective rows. For higher code-quality, a fuller
+  coding-heavy mix + 2+ epochs is the obvious next step.
+
+## Loading
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+base = AutoModelForCausalLM.from_pretrained(
+    "unsloth/Qwen3-Coder-30B-A3B-Instruct",
+    torch_dtype=torch.bfloat16, device_map="auto",
+    load_in_4bit=True,
+)
+model = PeftModel.from_pretrained(base, "Taimwe/securecoder-30b-pro")
+tokenizer = AutoTokenizer.from_pretrained("Taimwe/securecoder-30b-pro")
+```
+
+## Reproduce
+
+Scripts in [`Taimwe/securecoder-scripts`](https://huggingface.co/Taimwe/securecoder-scripts):
+
+```bash
+# train (commit-pinned URL form to avoid drift)
+hf jobs run -d --flavor a100-large --timeout 12h --secrets HF_TOKEN \
+  ghcr.io/astral-sh/uv:python3.12-bookworm \
+  uv run --no-project \
+  https://huggingface.co/Taimwe/securecoder-scripts/raw/<SHA>/train_securecoder.py \
+  -- --num-epochs 1 --mix-scale 0.33 --max-seq-length 2048 \
+     --output-repo Taimwe/securecoder-30b-pro --private
+```
+
+## Provenance
+
+- Hub user: Taimwe
+- Training script commit: see `Taimwe/securecoder-scripts` history
+- Uploaded with: `model.push_to_hub()` via Unsloth
+
+| `ise-uiuc/Magicoder-OSS-Instruct-75K` | 10,000 | self-instruct code problems + solutions |
+| `Trendyol/Trendyol-Cybersecurity-Instruction-Tuning-Dataset` | 8,000 | security instruction tuning |
+| `AlicanKiraz0/Cybersecurity-Dataset-Fenrir-v2.1` | 5,000 | broad security Q&A |
+| `Humanlearning/CyberSecurity_OWASP-sft-dataset` | 3,000 | OWASP / secure-coding SFT |
+| `MrClipperz134/CTF-Instruct` | 3,000 | CTF challenge → solve |
+| `TrueNix/ctf-solver-dataset` | 3,000 | CTF solving trajectories |
+| `mlabonne/FineTome-100k` | 3,000 | general instruct replay |
+
+Excluded deliberately: anything whose purpose is building malware or
+weaponised exploits. Recon/enumeration knowledge, exploit *concepts*, CTF
+solving and defensive engineering are in; end-to-end attack tooling is not.