Update README.md

README.md

@@ -1,3 +1,492 @@
----
-license: mit
----
+---
+license: mit
+language:
+- en
+tags:
+- code
+- markdown
+- tiny
+- small
+- quick
+- fast
+- 28M
+- mistral
+- text-generation-inference
+---
+
+# **Mini-MD**
+
+Mini-MD is a **\~28M parameter transformer-decoder** trained on \~200k markdown files from Github.
+
+## Architecture
+
+| Key | Value |
+| :---: | :---: |
+| `hidden_size` | 384 |
+| `num_layers` | 8 |
+| `num_heads` | 6 |
+| `num_kv_heads` | 2 |
+| `head_dim` | 64 |
+| `intermediate_size` | 1536 |
+| `vocab_size` | 14002 |
+| `sliding_window` | 640 |
+| `rope_theta` | 10000.0 |
+| `tie_embeddings` | True |
+| `total_params` | 28061568 |
+
+## Training
+
+### Training Parameters
+
+| Key | Value |
+| :---: | :---: |
+| `num_epochs` | 3 |
+| `batch_size` | 5 |
+| `stride` | 620 |
+| `seq_len` | 640 |
+| `val_split` | 0.09 |
+| `learning_rate` | 2e-4 |
+
+### Training Results
+
+| `train_loss` | `val_loss` | `step` | `epoch` |
+| :---: | :---: | :---: | :---: |
+| 6.8138 | 5.7706 | 1200 | 0.02 |
+| 2.4274 | 2.5915 | 12000 | 0.24 |
+| 2.1519 | 2.2091 | 30000 | 0.59 |
+| 2.0411 | 2.0464 | 48000 | 0.95 |
+| 1.7728 | 1.8912 | 84000 | 1.66 |
+| 1.7304 | 1.8494 | 100800 | 1.99 |
+| 1.6394 | 1.7599 | 132000 | 2.60 |
+| 1.6794 | 1.7234 | 151200 | 2.98 |
+
+### Hardware
+
+GPU: One singular NVIDIA RTX 2060 6GB (14GB Shared RAM).  
+CPU: Ryzen 5 2600  
+RAM: 16GB
+
+## Generations
+
+Input:
+
+```
+# README
+```
+
+Output:
+
+```
+Cover Evolution
+
+Official documentation for dis curated documentation: cosu:
+
+**README.md** (Recommended for reference)
+- **Repository**:
+- **Bloomberg**:
+
+**Welcome to Open Questions about Apache Dis curriculum**
+- **Security:** All files are securely required through social media.
+- **International (FONTLOG)**: Only include relevant code and resources when necessary to see new or better info.
+```
+
+---
+
+Input:
+
+```
+# Getting Started with Rust
+```
+
+Output:
+
+```
+You need a Rust crate for your operating system or platform environment, but you can
+install the Rust organization for Rust programs, and it's enough you'd really want it
+to work with your operating system.
+
+### Building Tools
+
+#### FFI Compatibility
+
+Rust's Rust has a compiler that can handle the following:
+- Using a toolchain with the AVX-based kernels
+- FFI Compatibility on ARM for iOS, iOS, Robot, etc
+- FFI Support from Linux for XPC platforms
+```
+
+---
+
+Input:
+
+```
+## Installation
+```
+
+Output:
+
+```
+in this project
+
+For building the Heft and French benchmarks, there are two popular tools you can use to build
+various benchmarks:
+
+* [Jim]( and [iL2](
+* [Lucid]( is supported, with [t5]( the
+[`LuxSrc32`]( CLI to get a better benchmark
+
+## Usage in this project:
+
+Npm install
+[LuxSrc32]( It is recommended to use precompiled GopherLM files
+```
+
+---
+
+Input:
+
+```
+The quick brown fox
+```
+
+Output:
+
+```
+ispepper is the skype of the games and you just want to generate them to your be.
+To use this you will need a Galicu sensor that has 2x 3x2 pesus.
+
+The other four mandaracters share it - you want the German light/highest food wey no
+your browses for game. Geometric, like grove, grove, and bus are pencilumo.
+```
+
+---
+
+Input:
+
+```
+Hello world
+```
+
+Output:
+
+```
+I can find a brown, I know it may need the full-fledip data into my phone
+buying you...
+my first time I went my email, we're going through it.
+You should now see you.
+Thank you the wonderful! Thanks! It's very cool when the dog would bed.
+We can't go through the world until you've started it!
+```
+
+---
+
+Input:
+
+```
+Once upon a time,
+```
+
+Output:
+
+```
+all data are moved to any place the user is aware of their operation and then removed.
+
+When the following is present:
+
+* The ability to move state through the same service, which allows
+  the ability of clients to move state across a network interface.
+
+During a [DaemonSet controller](/docs/concepts/cluster-administration/deployment/),
+all webhooks of the DaemonSet controller may be invoked from a user through a
+"Service Cluster" resource without any actions.
+```
+
+## Capabilities & Restrictions
+
+### Capabilities
+
+- Continues GitHub-style markdown documents plausibly, particularly README sections, changelogs, installation guides, and API documentation
+- Produces syntactically well-formed code blocks across multiple languages (Python, Rust, Go, C++, JavaScript)
+- Sustains a single topic for several paragraphs when the prompt closely matches training distribution (e.g. `# Getting Started with <common language>`)
+
+### Restrictions
+
+- Not an instruction-following model — treats all input as a document prefix to continue, not a query to answer
+- Out-of-distribution prompts (natural language, fiction, conversation) produce incoherent or nonsensical output
+- Prone to topic drift over longer generations, gradually sliding into unrelated documentation
+- Prone to repetition loops, particularly on short or ambiguous prompts
+- Generates hallucinated URLs, package names, library names, and version numbers with no grounding
+- Multilingual output may appear mid-generation, inherited from non-English READMEs in the training corpus; coherence in non-English output is lower than English
+- Not suitable for any production use
+
+## Inference
+
+```python
+#!/usr/bin/env python3
+"""
+Tiny Mistral REPL demo — streaming tokens (TextStreamer if available, else manual sampling).
+Commands: :quit, :help, :show, :set <param> <value> (max_new_tokens, temperature, top_p, full_output)
+"""
+from __future__ import annotations
+import shlex
+import time
+import torch
+from typing import Optional
+
+from transformers import AutoTokenizer, MistralForCausalLM
+
+# --------- CONFIG ----------
+MODEL_DIR = "Harley-ml/Mini-MD"
+TOKENIZER_DIR = MODEL_DIR
+DEFAULT_MAX_NEW_TOKENS = 640
+DEFAULT_TEMPERATURE = 0.9
+DEFAULT_TOP_P = 0.9
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+PROMPT = ">>> "
+# ---------------------------
+
+def load_tokenizer(path: str):
+    print("Loading tokenizer...", path)
+    tok = AutoTokenizer.from_pretrained(path, use_fast=True, local_files_only=False)
+    if tok.pad_token is None:
+        if getattr(tok, "eos_token", None) is not None:
+            tok.add_special_tokens({"pad_token": tok.eos_token})
+        else:
+            tok.add_special_tokens({"pad_token": "<pad>", "eos_token": "</s>"})
+    print("Tokenizer ready. vocab_size=", getattr(tok, "vocab_size", "N/A"))
+    return tok
+
+def load_model(path: str, device: str):
+    print("Loading model...", path)
+    model = None
+    try:
+        desired_dtype = torch.float16 if device.startswith("cuda") else torch.float32
+        model = MistralForCausalLM.from_pretrained(path, local_files_only=False, dtype=desired_dtype)
+        print("Loaded with dtype arg.")
+    except TypeError:
+        model = MistralForCausalLM.from_pretrained(path, local_files_only=False)
+        print("Loaded without dtype; will convert.")
+    except Exception as e:
+        print("Load warning, retrying without dtype:", e)
+        model = MistralForCausalLM.from_pretrained(path, local_files_only=False)
+
+    try:
+        model.to(device)
+        if device.startswith("cuda") and next(model.parameters()).dtype != torch.float16:
+            model.half()
+        if not device.startswith("cuda") and next(model.parameters()).dtype != torch.float32:
+            model.to(torch.float32)
+    except Exception as e:
+        print("Model move/convert warning:", e)
+
+    model.config.pad_token_id = getattr(model.config, "pad_token_id", None)
+    model.eval()
+    return model
+
+# Simple nucleus/top-p filtering for a single logits vector
+def top_p_filtering(logits: torch.Tensor, top_p: float, min_keep: int = 1) -> torch.Tensor:
+    if top_p <= 0 or top_p >= 1.0:
+        return logits
+    sorted_logits, sorted_idx = torch.sort(logits, descending=True)
+    probs = torch.softmax(sorted_logits, dim=-1)
+    cumprobs = torch.cumsum(probs, dim=-1)
+    cutoff = (cumprobs > top_p).nonzero(as_tuple=False)
+    if cutoff.numel() > 0:
+        idx = int(cutoff[0].item())
+        cutoff_idx = max(idx + 1, min_keep)
+    else:
+        cutoff_idx = sorted_logits.size(-1)
+    mask = torch.ones_like(sorted_logits, dtype=torch.bool)
+    mask[cutoff_idx:] = False
+    filtered = sorted_logits.masked_fill(~mask, -float("inf"))
+    return torch.empty_like(filtered).scatter_(0, sorted_idx, filtered)
+
+# Manual streaming generator (single-batch)
+def manual_stream_generate(model, tokenizer, prompt: str, device: str,
+                           max_new_tokens: int = 64, temperature: float = 1.0, top_p: float = 0.9,
+                           eos_token_id: Optional[int] = None):
+    inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
+    input_ids = inputs["input_ids"].to(device)
+    attention_mask = inputs.get("attention_mask", None)
+    if attention_mask is not None:
+        attention_mask = attention_mask.to(device)
+
+    past = None
+    with torch.no_grad():
+        out = model(input_ids=input_ids, attention_mask=attention_mask, use_cache=True)
+        past = getattr(out, "past_key_values", None)
+
+    # start sampling tokens
+    next_input = input_ids[:, -1:].to(device) if past is not None else input_ids.to(device)
+    for _ in range(max_new_tokens):
+        with torch.no_grad():
+            out = model(input_ids=next_input, past_key_values=past, use_cache=True)
+            logits = out.logits[:, -1, :]  # (batch, vocab)
+            past = getattr(out, "past_key_values", past)
+
+            if temperature != 1.0:
+                logits = logits / max(temperature, 1e-8)
+
+            filtered = top_p_filtering(logits[0].cpu(), top_p).to(device)
+            probs = torch.nn.functional.softmax(filtered.unsqueeze(0), dim=-1)
+            next_token = torch.multinomial(probs, num_samples=1)
+            token_id = int(next_token[0, 0].item())
+
+        token_text = tokenizer.decode([token_id], clean_up_tokenization_spaces=False)
+        yield token_id, token_text
+
+        if eos_token_id is not None and token_id == eos_token_id:
+            break
+        next_input = torch.tensor([[token_id]], dtype=torch.long, device=device)
+
+def has_text_streamer():
+    try:
+        from transformers import TextStreamer  # type: ignore
+        return True
+    except Exception:
+        return False
+
+# tiny REPL state
+class State:
+    def __init__(self):
+        self.max_new_tokens = DEFAULT_MAX_NEW_TOKENS
+        self.temperature = DEFAULT_TEMPERATURE
+        self.top_p = DEFAULT_TOP_P
+        self.full_output = False
+        self.stream = True
+
+def handle_generation(model, tokenizer, prompt: str, device: str, state: State):
+    eos = getattr(tokenizer, "eos_token_id", None)
+    try:
+        if has_text_streamer():
+            from transformers import TextStreamer
+            streamer = TextStreamer(tokenizer, skip_prompt=not state.full_output, skip_special_tokens=True)
+            inputs = tokenizer(prompt, return_tensors="pt", truncation=True, add_special_tokens=False)
+            inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)}
+            inputs.pop("token_type_ids", None)
+            model.generate(**inputs,
+                           max_new_tokens=state.max_new_tokens,
+                           do_sample=True,
+                           temperature=state.temperature,
+                           top_p=state.top_p,
+                           pad_token_id=tokenizer.pad_token_id,
+                           eos_token_id=tokenizer.eos_token_id,
+                           streamer=streamer)
+            print("")  # newline after streamer
+            return
+        # fallback: manual streaming
+        gen = manual_stream_generate(model, tokenizer, prompt, device,
+                                     max_new_tokens=state.max_new_tokens,
+                                     temperature=state.temperature,
+                                     top_p=state.top_p,
+                                     eos_token_id=eos)
+        if state.full_output:
+            print("PROMPT:", prompt)
+            print("GENERATING:", end=" ", flush=True)
+        else:
+            print("GENERATING:", end=" ", flush=True)
+
+        count = 0
+        t0 = time.time()
+        for _tok_id, tok_text in gen:
+            count += 1
+            print(tok_text, end="", flush=True)
+        print()
+        print(f"(generated {count} tokens in {time.time()-t0:.2f}s)")
+    except KeyboardInterrupt:
+        print("\n[interrupted] Generation aborted by user.")
+    except Exception as e:
+        print("Generation error:", e)
+
+def repl(model, tokenizer, device):
+    state = State()
+    help_text = (
+        "Commands:\n"
+        " :quit\n"
+        " :help\n"
+        " :show\n"
+        " :set <param> <value>  # params: max_new_tokens, temperature, top_p, full_output, stream\n"
+        " (blank line repeats last prompt)\n"
+    )
+    print("Tiny Mistral REPL — device:", device)
+    print(help_text)
+    last = ""
+    while True:
+        try:
+            raw = input(PROMPT).strip()
+        except (EOFError, KeyboardInterrupt):
+            print("\nExiting.")
+            break
+        if not raw:
+            raw = last
+            if not raw:
+                continue
+
+        if raw.startswith(":"):
+            toks = shlex.split(raw)
+            cmd = toks[0].lower()
+            if cmd == ":quit":
+                print("bye.")
+                break
+            if cmd == ":help":
+                print(help_text); continue
+            if cmd == ":show":
+                print(f"max_new_tokens={state.max_new_tokens}, temperature={state.temperature}, top_p={state.top_p}, full_output={state.full_output}, stream={state.stream}")
+                continue
+            if cmd == ":set":
+                if len(toks) < 3:
+                    print("usage: :set <param> <value>"); continue
+                k, v = toks[1], toks[2]
+                try:
+                    if k == "max_new_tokens":
+                        state.max_new_tokens = int(v)
+                    elif k == "temperature":
+                        state.temperature = float(v)
+                    elif k == "top_p":
+                        state.top_p = float(v)
+                    elif k in ("full_output", "full"):
+                        state.full_output = v.lower() in ("1", "true", "yes", "y")
+                    elif k == "stream":
+                        state.stream = v.lower() in ("1", "true", "yes", "y")
+                    else:
+                        print("unknown param:", k)
+                        continue
+                    print("OK.")
+                except Exception as e:
+                    print("set error:", e)
+                continue
+            print("unknown command")
+            continue
+
+        last = raw
+        if state.stream:
+            handle_generation(model, tokenizer, raw, device, state)
+        else:
+            # non-streaming generate
+            try:
+                inputs = tokenizer(raw, return_tensors="pt", truncation=True, add_special_tokens=False)
+                inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)}
+                inputs.pop("token_type_ids", None)
+                out = model.generate(**inputs,
+                                     max_new_tokens=state.max_new_tokens,
+                                     do_sample=True,
+                                     temperature=state.temperature,
+                                     top_p=state.top_p,
+                                     pad_token_id=tokenizer.pad_token_id,
+                                     eos_token_id=tokenizer.eos_token_id)
+                seq = out[0]
+                input_len = inputs["input_ids"].shape[1] if "input_ids" in inputs else 0
+                text = tokenizer.decode(seq if state.full_output else seq[input_len:], skip_special_tokens=True)
+                print("\nOUTPUT\n", text)
+            except Exception as e:
+                print("Generation failed:", e)
+
+def main():
+    device = DEVICE
+    tokenizer = load_tokenizer(TOKENIZER_DIR)
+    model = load_model(MODEL_DIR, device)
+    repl(model, tokenizer, device)
+
+if __name__ == "__main__":
+    main()
+```