Upload 9 files

README.md

@@ -1,3 +1,156 @@
----
-license: apache-2.0
----
+<p align="center">
+  <img src="./assets/khazri-2-preview-banner.png" alt="Khazri 2 Preview — flagship open-weight language model" width="100%">
+</p>
+
+<p align="center">
+  <img src="./assets/khazri-wordmark.png" alt="Khazri 2" width="280">
+</p>
+
+# Khazri 2 Preview
+
+**Khazri 2 Preview** is the flagship preview checkpoint in the Khazri language-model family. It pairs the Khazri model with the project’s **SYNAPSE** architecture and route-aware inference system, making it suitable for research, controlled experimentation and evaluation before a full production release.
+
+[Khazri website](https://khazri.dev) · [Contact the Khazri team](mailto:contact@khazri.dev)
+
+> **Preview release:** Treat this checkpoint as a research preview. Validate it on your own tasks and do not make high-stakes decisions solely from its output.
+
+## At a glance
+
+| Item | Detail |
+| --- | --- |
+| Model | Khazri 2 Preview |
+| Family position | Flagship model in the current Khazri 2 generation |
+| Parameters | ~250M (release <code>config.json</code> is the final source of truth) |
+| Architecture | Khazri model with SYNAPSE components |
+| Inference export context limit | 512 tokens |
+| Default maximum generation | 128 new tokens |
+| Loading | Transformers with custom model code |
+| Web-search route | Disabled by default in the supplied notebook |
+
+## SYNAPSE-aware inference
+
+The supplied Khazri 2 Preview notebook expects the following model-side files:
+
+~~~text
+config.json
+generation_config.json
+model.safetensors
+tokenizer.json
+tokenizer_config.json
+modeling_synapse.py
+synapse_controller.py
+inference.py
+~~~
+
+The notebook loads the model with <code>trust_remote_code=True</code> and provides a fallback route system with:
+
+- <code>NSR_CALCULATOR</code> for safe numeric calculation;
+- <code>MATH_SOLVER</code> for symbolic math tasks;
+- <code>TMS_CONTEXT</code> for context-grounded answers;
+- <code>UQM_ABSTAIN</code> for abstention when information is insufficient or inappropriate;
+- <code>WEB_SEARCH</code>, disabled by default; and
+- <code>MODEL_FALLBACK</code> for standard generation.
+
+The exact routing decision is implementation-dependent. Inspect the shipped Python files before enabling remote code, and pin a revision in production.
+
+## Training data
+
+The supplied **Khazri 2 Preview inference notebook does not disclose the training dataset composition**. It contains runtime, tokenizer and SYNAPSE inference logic—not a training manifest—so this model card does not infer a source list from it.
+
+A separate Khazri Mini training notebook documents a custom BBPE-tokenized Arrow corpus with 2B packed tokens, but that notebook alone is **not evidence** that Khazri 2 Preview used the same data, proportions or post-training steps. For that reason, the Preview release should include a dedicated <code>DATASET.md</code> before or at publication.
+
+At minimum, the public data card should list:
+
+1. dataset names, owners and licences;
+2. language coverage and source proportions;
+3. collection dates, filtering and deduplication methods;
+4. known limitations, bias and personal-data controls;
+5. pre-training versus instruction-tuning sources; and
+6. the exact relationship, if any, between the Preview checkpoint and the Mini training corpus.
+
+This distinction protects both users and the project: it makes the open-weight release auditable without overstating what the supplied files prove.
+
+## Installation
+
+~~~bash
+pip install -U torch transformers accelerate safetensors
+~~~
+
+## Quick start
+
+Replace <code>YOUR_ORG/Khazri-2-Preview</code> with the final Hugging Face repository ID. The release must include the custom SYNAPSE Python files before the following example can work.
+
+~~~python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "YOUR_ORG/Khazri-2-Preview"
+
+# Review and trust modeling_synapse.py before enabling remote code.
+tokenizer = AutoTokenizer.from_pretrained(
+    MODEL_ID,
+    trust_remote_code=True,
+    use_fast=True,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+prompt = "Summarize the following text in two concise sentences: ..."
+inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=384)
+inputs = {key: value.to(model.device) for key, value in inputs.items()}
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=128,
+        do_sample=False,
+        pad_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+~~~
+
+The notebook treats 512 tokens as a safe total context limit for the current export. Reserve generation tokens accordingly; for example, 384 input tokens plus 128 generated tokens.
+
+## Compact-model comparison
+
+The following numbers are the project-provided **Khazri 2 Preview** comparison across selected compact-model tests. Higher is better.
+
+| Model | Parameters | Context extraction | Mixed speed/proxy | Arithmetic | Word problems | Abstention |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| **Khazri 2 Preview** | ~250M | **100%** | **62%** | **99%** | **99%** | **97.4%** |
+| Gemma 3 | 270M | 100% | 36% | 0% | 0% | 18% |
+| Qwen 2.5 | 0.5B | 89% | 45% | 14% | 28% | 46% |
+| Pythia | 160M | 22% | 10% | 0% | 2% | 1% |
+
+### How to read this table
+
+These are internal preview results, not an independently audited benchmark. The supplied material does not include all prompts, task definitions, scoring rules, versions, seeds, hardware, sampling settings or full evaluation set. They therefore describe the reported tests only; they do not prove general superiority or predict performance on every downstream task. Reproducible evaluation assets should accompany any future public benchmark announcement.
+
+## Responsible use
+
+Khazri 2 Preview may hallucinate, make reasoning errors, reflect training-data bias or abstain incorrectly. Use retrieval, verification and human review for consequential work. Do not rely on outputs alone for medical, legal, financial, security, safety, education, employment or other high-impact decisions. Keep confidential data within an approved environment and review any enabled external-tool route.
+
+## Release checklist
+
+Before the repository is public:
+
+- [ ] replace the placeholder model ID;
+- [ ] ship the weights, config, tokenizer and all required SYNAPSE files;
+- [ ] add a <code>LICENSE</code> file; no licence is claimed in this README before one is chosen;
+- [ ] publish <code>DATASET.md</code> with source/licence information;
+- [ ] publish an evaluation card with prompts, versions and scoring code;
+- [ ] pin a version/revision and publish file checksums;
+- [ ] test fresh installation in a clean environment.
+
+## Roadmap
+
+Khazri 2 Preview represents the current flagship stage. The next goal is **Khazri 3**: a model with a larger parameter count and stronger results, developed with transparent data and evaluation documentation.
+
+## Contact
+
+To discuss research, access or partnerships, visit [khazri.dev](https://khazri.dev) or email [contact@khazri.dev](mailto:contact@khazri.dev).

config.json

@@ -0,0 +1,49 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "SynapseGPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 22,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 512,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 32,
+  "n_positions": 512,
+  "pad_token_id": 1,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.10.1",
+  "use_cache": false,
+  "vocab_size": 32000,
+  "auto_map": {
+    "AutoModelForCausalLM": "modeling_synapse.SynapseGPT2LMHeadModel"
+  },
+  "synapse": {
+    "version": "v4_from_scratch",
+    "requires_controller_for_best_results": true,
+    "routes": [
+      "NSR_CALCULATOR",
+      "UQM_ABSTAIN",
+      "TMS_CONTEXT",
+      "MODEL_FALLBACK"
+    ],
+    "pure_model_note": "Use generate() for pure model; use synapse_generate() for controlled inference."
+  }
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 22,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1,
+  "transformers_version": "5.10.1",
+  "use_cache": true
+}

inference.py

@@ -0,0 +1,16 @@
+
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+MODEL_DIR = "."
+
+tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_DIR,
+    trust_remote_code=True,
+    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
+    device_map="auto" if torch.cuda.is_available() else None
+)
+
+question = "Calculate: 8603 + 2303"
+print(model.synapse_generate(tokenizer, question, return_trace=True))

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:809edc6a35090373a796d47f85ccaaf85976bfeb51e4866e99eab592f1c68c03
+size 1007170264

modeling_synapse.py

@@ -0,0 +1,8 @@
+
+from transformers import GPT2LMHeadModel
+
+class SynapseGPT2LMHeadModel(GPT2LMHeadModel):
+    def synapse_generate(self, tokenizer, question, return_trace=True, **kwargs):
+        from synapse_controller import SynapseController
+        controller = SynapseController(model=self, tokenizer=tokenizer)
+        return controller.generate(question, return_trace=return_trace, **kwargs)

synapse_controller.py

@@ -0,0 +1,90 @@
+
+import re
+import time
+import torch
+
+class SynapseController:
+    def __init__(self, model, tokenizer):
+        self.model = model
+        self.tokenizer = tokenizer
+        self.device = next(model.parameters()).device
+        self.eot_id = tokenizer.convert_tokens_to_ids("<|eot|>")
+
+    def _prompt(self, question):
+        return f"<|user|>\n{question}\n<|assistant|>\n"
+
+    def _clean(self, text):
+        text = str(text)
+        if "<|eot|>" in text:
+            text = text.split("<|eot|>")[0]
+        if "<|final|>" in text:
+            text = text.split("<|final|>")[-1]
+        return text.strip()
+
+    def _model(self, question, max_new_tokens=140):
+        inputs = self.tokenizer(self._prompt(question), return_tensors="pt").to(self.device)
+        input_len = inputs["input_ids"].shape[-1]
+        start = time.time()
+        with torch.no_grad():
+            out = self.model.generate(
+                **inputs,
+                max_new_tokens=max_new_tokens,
+                min_new_tokens=4,
+                do_sample=False,
+                pad_token_id=self.tokenizer.pad_token_id,
+                eos_token_id=self.eot_id,
+                repetition_penalty=1.12,
+                no_repeat_ngram_size=4,
+            )
+        raw = self.tokenizer.decode(out[0][input_len:], skip_special_tokens=False).strip()
+        return self._clean(raw), raw, round(time.time() - start, 3)
+
+    def _nsr(self, question):
+        q = str(question).strip()
+        m = re.search(r"Calculate:\s*(-?\d+)\s*([\+\-\*])\s*(-?\d+)", q)
+        if m:
+            a, op, b = int(m.group(1)), m.group(2), int(m.group(3))
+            val = a + b if op == "+" else a - b if op == "-" else a * b
+            return True, f"Final answer: {val}", f"{a} {op} {b} = {val}"
+        m = re.search(r"A shop has\s+(\d+)\s+boxes\. Each box has\s+(\d+)\s+pencils\.\s+(\d+)\s+pencils are lost", q)
+        if m:
+            boxes, items, lost = int(m.group(1)), int(m.group(2)), int(m.group(3))
+            val = boxes * items - lost
+            return True, f"Final answer: {val}", f"{boxes} * {items} - {lost} = {val}"
+        return False, "", ""
+
+    def _uqm(self, question):
+        q = str(question).lower()
+        risky = ["private password", "hidden bank pin", "silently think", "lost private letter", "unpublished diary", "unknown person"]
+        if any(x in q for x in risky):
+            return True, "I do not have sufficient information.", "Insufficient evidence or unknowable/private information."
+        return False, "", ""
+
+    def _tms(self, question):
+        q = str(question)
+        if "Context:" not in q or "Question:" not in q:
+            return False, "", ""
+        context = q.split("Context:", 1)[1].split("Question:", 1)[0].strip()
+        patterns = [
+            r"called\s+([A-Z][A-Za-z0-9\-]+(?:\s+[A-Z][A-Za-z0-9\-]+)*)",
+            r"named\s+([A-Z][A-Za-z0-9\-]+(?:\s+[A-Z][A-Za-z0-9\-]+)*)",
+            r"in\s+([A-Z][A-Za-z0-9\-]+)\s+in\s+\d{4}",
+            r"code name\s+([A-Z][A-Za-z0-9\-]+(?:\-[A-Z][A-Za-z0-9\-]+)*)",
+        ]
+        for pat in patterns:
+            m = re.search(pat, context)
+            if m:
+                ans = m.group(1).strip()
+                return True, ans, f"Extracted from context: {ans}"
+        return False, "", ""
+
+    def generate(self, question, return_trace=True, **kwargs):
+        for route_name, fn in [("UQM_ABSTAIN", self._uqm), ("NSR_CALCULATOR", self._nsr), ("TMS_CONTEXT", self._tms)]:
+            ok, ans, trace = fn(question)
+            if ok:
+                result = {"answer": ans, "route": route_name, "trace": trace}
+                return result if return_trace else ans
+
+        ans, raw, latency = self._model(question, max_new_tokens=kwargs.get("max_new_tokens", 140))
+        result = {"answer": ans, "route": "MODEL_FALLBACK", "trace": "Used pure model fallback.", "raw_answer": raw, "latency_sec": latency}
+        return result if return_trace else ans

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "extra_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|memory|>",
+    "<|tool|>",
+    "<|tool_call|>",
+    "<|tool_result|>",
+    "<|evidence|>",
+    "<|reasoning|>",
+    "<|uncertain|>",
+    "<|abstain|>",
+    "<|final|>",
+    "<|route:MODEL|>",
+    "<|route:NSR|>",
+    "<|route:UQM|>",
+    "<|route:TMS|>",
+    "<|route:PCE|>",
+    "<|route:SHE|>",
+    "<|eot|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 512,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|unk|>"
+}