Upload README.md with huggingface_hub

README.md

@@ -20,7 +20,7 @@ tags:
   - sft
   - mlx
 model-index:
-  - name: marvy-14B
+  - name: marvy-1-14B
     results:
       - task:
           type: text-generation
@@ -37,17 +37,17 @@ model-index:
             name: Test cross-entropy loss
 ---
 
-# marvy-14B
+# marvy-1-14B
 
 **The first open, fine-tuned LLM for the full ServiceNow delivery lifecycle — from business analysis to validation.**
 
-marvy-14B is an open-source language model fine-tuned for the complete ServiceNow delivery lifecycle: business analysis, requirements, stakeholder mapping, systems inventory, Solution Design Documents, user stories with acceptance criteria, implementation planning, test cases, and validation. Where general-purpose models treat ServiceNow as one topic among many, marvy is built to draft the actual artifacts a delivery team produces — in the structure and sequence real engagements follow. It is a first-draft specialist, not a consultant replacement, and it is not an agentic or tool-use fine-tune.
+marvy-1-14B is an open-source language model fine-tuned for the complete ServiceNow delivery lifecycle: business analysis, requirements, stakeholder mapping, systems inventory, Solution Design Documents, user stories with acceptance criteria, implementation planning, test cases, and validation. Where general-purpose models treat ServiceNow as one topic among many, marvy is built to draft the actual artifacts a delivery team produces — in the structure and sequence real engagements follow. It is a first-draft specialist, not a consultant replacement, and it is not an agentic or tool-use fine-tune.
 
 It was built by [MainStack](https://huggingface.co/MainStack), a consultancy specializing in ServiceNow Agentic Delivery. marvy is a LoRA SFT fine-tune of [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) (Apache-2.0), trained on ~1,958 anonymized artifacts from real engagements (~887k tokens), rigorously redacted to zero residual PII per an automated leakage scanner. Its test perplexity of 13.107 was measured on a project- and customer-disjoint held-out split — the model generalizes to unseen work rather than memorizing the training set.
 
 > Released under **Apache-2.0**. Built with Qwen — see `NOTICE`.
 
-## Why marvy-14B
+## Why marvy-1-14B
 
 - **Drafts the full lifecycle, not just snippets.** Business analysis through validation — the artifacts and sequence real delivery teams actually work in.
 - **OOTB-first and implementation-grade.** Tuned to favor out-of-the-box correctness and produce drafts you can review, not rewrite.
@@ -68,7 +68,7 @@ It was built by [MainStack](https://huggingface.co/MainStack), a consultancy spe
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
-model_id = "MainStack/marvy-14B"
+model_id = "MainStack/marvy-1-14B"
 tok = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")
 
@@ -93,22 +93,22 @@ print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
 
 ```bash
 pip install vllm
-vllm serve MainStack/marvy-14B
+vllm serve MainStack/marvy-1-14B
 ```
 
 ### Ollama (via GGUF)
 
-Use the companion repo [`MainStack/marvy-14B-GGUF`](https://huggingface.co/MainStack/marvy-14B-GGUF):
+Use the companion repo [`MainStack/marvy-1-14B-GGUF`](https://huggingface.co/MainStack/marvy-1-14B-GGUF):
 
 ```bash
-ollama run hf.co/MainStack/marvy-14B-GGUF:Q4_K_M
+ollama run hf.co/MainStack/marvy-1-14B-GGUF:Q4_K_M
 ```
 
 ### MLX (Apple Silicon native)
 
 ```bash
 pip install mlx-lm
-python -m mlx_lm generate --model MainStack/marvy-14B \
+python -m mlx_lm generate --model MainStack/marvy-1-14B \
   --system-prompt "You are a senior ServiceNow delivery consultant..." \
   --prompt "Draft the Platform Architecture section of an ITSM SDD." \
   --max-tokens 1024 --temp 0.4
@@ -116,13 +116,13 @@ python -m mlx_lm generate --model MainStack/marvy-14B \
 
 ### LoRA-only (apply on top of the base)
 
-If you prefer a tiny adapter (~175 MB) on top of the BF16 base, see [`MainStack/marvy-14B-lora`](https://huggingface.co/MainStack/marvy-14B-lora).
+If you prefer a tiny adapter (~175 MB) on top of the BF16 base, see [`MainStack/marvy-1-14B-lora`](https://huggingface.co/MainStack/marvy-1-14B-lora).
 
 ---
 
 ## Intended use
 
-marvy-14B is designed to produce implementation-grade first drafts across the ServiceNow delivery lifecycle — accelerating the artifacts a practitioner would otherwise write from scratch, then review and refine. Built for solution architects, business analysts, technical consultants, and project managers. Typical tasks:
+marvy-1-14B is designed to produce implementation-grade first drafts across the ServiceNow delivery lifecycle — accelerating the artifacts a practitioner would otherwise write from scratch, then review and refine. Built for solution architects, business analysts, technical consultants, and project managers. Typical tasks:
 
 | Task family            | What it produces                                                                |
 |------------------------|---------------------------------------------------------------------------------|
@@ -206,18 +206,47 @@ Train / val / test are split **by project**, so no customer appears in more than
 
 ## Evaluation
 
-Test-set evaluation on the **project-disjoint** test split (252 records from two customers never seen in training/val), 50 batches:
+### Fine-tuned vs. base — efficiency on the held-out test set
 
-| Metric | Value |
-|---|---|
-| Test cross-entropy loss | **2.573** |
-| Test perplexity | **13.107** |
+The cleanest measure of the fine-tune's value is to score the **same base
+model twice** — plain vs. with the marvy adapter — on the **project-disjoint**
+test split (252 records from two customers never seen in training/val), using
+per-token cross-entropy/perplexity on the **assistant tokens only**
+(prompt-masked, the same objective used in training). Lower perplexity = the
+model assigns higher probability to the real, human-authored delivery artifact.
+
+![marvy-1-14B vs base — perplexity by task](./marvy_vs_base_ppl.png)
+
+![How much fine-tuning improved each task](./marvy_improvement.png)
+
+**Overall: perplexity 8.91 → 6.03, a 32.3% reduction** on unseen customers.
+
+| Task | Base ppl | marvy-1-14B ppl | Improvement |
+|---|---:|---:|---:|
+| Systems inventory | 77.07 | 10.53 | **−86.3%** |
+| Requirements extraction | 46.76 | 9.39 | **−79.9%** |
+| Stakeholder mapping | 27.81 | 6.91 | **−75.2%** |
+| Story authoring | 15.38 | 7.86 | **−48.9%** |
+| Validation / critique | 9.72 | 8.23 | −15.3% |
+| Business analysis | 7.14 | 6.66 | −6.6% |
+| SDD design | 4.48 | 4.40 | −1.7% |
+| **Overall** | **8.91** | **6.03** | **−32.3%** |
+
+The gains are largest on **structured, format-heavy artifacts** (inventories,
+requirements, stakeholder registers, stories) where the base model wanders from
+the expected schema; they are smaller on long-form prose (SDD sections, business
+analysis) where the base was already competent. This is the honest, expected
+shape of a domain SFT.
 
-> Note: two test sequences exceed 2,048 tokens and are truncated by the MLX eval harness. The reported figure is therefore a slight upper bound on true loss. Full-length scoring is planned for v2.
+> Notes: the test customers (`Customer-CHEM-01`, `Customer-FININST-01`) appear in
+> neither train nor val, so this reflects generalization, not memorization. The
+> test split happens to cover 7 of the 11 task families. An earlier MLX
+> batch-eval reported aggregate ppl ≈ 13.1 with 2,048-token truncation; the
+> figures above recompute per-task with full assistant-token masking, so the
+> base-vs-marvy **delta** is the result of interest.
 
-To reproduce or validate these results yourself — including a base-vs-marvy
-comparison and qualitative task probes — see [`VALIDATION.md`](./VALIDATION.md)
-and run [`validate.sh`](./validate.sh).
+Reproduce it yourself: `bash benchmark/run_benchmark.sh` (see
+[`VALIDATION.md`](./VALIDATION.md) for qualitative probes too).
 
 ---
 
@@ -228,7 +257,7 @@ and run [`validate.sh`](./validate.sh).
 - **Synthetic instructions.** User prompts are templated paraphrases (3–5 variants per task); assistant outputs are the original human-authored artifacts.
 - **English-only.** The corpus is English.
 - **Not a replacement for a consultant.** Output is first-draft, implementation-grade content that requires expert review before client delivery or production use.
-- **No tool use / function calling fine-tune.** `marvy-14B` is a text-completion specialist; agentic tool use is left to the orchestrator.
+- **No tool use / function calling fine-tune.** `marvy-1-14B` is a text-completion specialist; agentic tool use is left to the orchestrator.
 - **Hallucination risk on instance-specific facts.** The model will confidently invent `sys_id`s, plugin IDs, and table fields if asked about specifics it has not seen. Always verify against an actual ServiceNow instance.
 - **No safety fine-tune beyond the base.** Inherits Qwen2.5-14B-Instruct safety behavior; no additional RLHF.
 
@@ -244,10 +273,10 @@ This model is a derivative of **Qwen2.5-14B-Instruct** (Apache-2.0). See `NOTICE
 
 ```bibtex
 @software{marvy_14b_2026,
-  title  = {marvy-14B: A ServiceNow delivery lifecycle fine-tune of Qwen2.5-14B-Instruct},
+  title  = {marvy-1-14B: A ServiceNow delivery lifecycle fine-tune of Qwen2.5-14B-Instruct},
   author = {MainStack},
   year   = {2026},
-  url    = {https://huggingface.co/MainStack/marvy-14B},
+  url    = {https://huggingface.co/MainStack/marvy-1-14B},
   license= {Apache-2.0}
 }