--- license: mit language: - en tags: - code - markdown - tiny - small - quick - fast - 28M - mistral - text-generation-inference --- # **Mini-MD** Mini-MD is a **\~28M parameter transformer-decoder** trained on \~200k markdown files from Github. ## Architecture | Key | Value | | :---: | :---: | | `hidden_size` | 384 | | `num_layers` | 8 | | `num_heads` | 6 | | `num_kv_heads` | 2 | | `head_dim` | 64 | | `intermediate_size` | 1536 | | `vocab_size` | 14002 | | `sliding_window` | 640 | | `rope_theta` | 10000.0 | | `tie_embeddings` | True | | `total_params` | 28061568 | ## Training ### Training Parameters | Key | Value | | :---: | :---: | | `num_epochs` | 3 | | `batch_size` | 5 | | `stride` | 620 | | `seq_len` | 640 | | `val_split` | 0.09 | | `learning_rate` | 2e-4 | ### Training Results | `train_loss` | `val_loss` | `step` | `epoch` | | :---: | :---: | :---: | :---: | | 6.8138 | 5.7706 | 1200 | 0.02 | | 2.4274 | 2.5915 | 12000 | 0.24 | | 2.1519 | 2.2091 | 30000 | 0.59 | | 2.0411 | 2.0464 | 48000 | 0.95 | | 1.7728 | 1.8912 | 84000 | 1.66 | | 1.7304 | 1.8494 | 100800 | 1.99 | | 1.6394 | 1.7599 | 132000 | 2.60 | | 1.6794 | 1.7234 | 151200 | 2.98 | ### Hardware GPU: One singular NVIDIA RTX 2060 6GB (14GB Shared RAM). CPU: Ryzen 5 2600 RAM: 16GB ## Generations Input: ``` # README ``` Output: ``` Cover Evolution Official documentation for dis curated documentation: cosu: **README.md** (Recommended for reference) - **Repository**: - **Bloomberg**: **Welcome to Open Questions about Apache Dis curriculum** - **Security:** All files are securely required through social media. - **International (FONTLOG)**: Only include relevant code and resources when necessary to see new or better info. ``` --- Input: ``` # Getting Started with Rust ``` Output: ``` You need a Rust crate for your operating system or platform environment, but you can install the Rust organization for Rust programs, and it's enough you'd really want it to work with your operating system. ### Building Tools #### FFI Compatibility Rust's Rust has a compiler that can handle the following: - Using a toolchain with the AVX-based kernels - FFI Compatibility on ARM for iOS, iOS, Robot, etc - FFI Support from Linux for XPC platforms ``` --- Input: ``` ## Installation ``` Output: ``` in this project For building the Heft and French benchmarks, there are two popular tools you can use to build various benchmarks: * [Jim]( and [iL2]( * [Lucid]( is supported, with [t5]( the [`LuxSrc32`]( CLI to get a better benchmark ## Usage in this project: Npm install [LuxSrc32]( It is recommended to use precompiled GopherLM files ``` --- Input: ``` The quick brown fox ``` Output: ``` ispepper is the skype of the games and you just want to generate them to your be. To use this you will need a Galicu sensor that has 2x 3x2 pesus. The other four mandaracters share it - you want the German light/highest food wey no your browses for game. Geometric, like grove, grove, and bus are pencilumo. ``` --- Input: ``` Hello world ``` Output: ``` I can find a brown, I know it may need the full-fledip data into my phone buying you... my first time I went my email, we're going through it. You should now see you. Thank you the wonderful! Thanks! It's very cool when the dog would bed. We can't go through the world until you've started it! ``` --- Input: ``` Once upon a time, ``` Output: ``` all data are moved to any place the user is aware of their operation and then removed. When the following is present: * The ability to move state through the same service, which allows the ability of clients to move state across a network interface. During a [DaemonSet controller](/docs/concepts/cluster-administration/deployment/), all webhooks of the DaemonSet controller may be invoked from a user through a "Service Cluster" resource without any actions. ``` ## Capabilities & Restrictions ### Capabilities - Continues GitHub-style markdown documents plausibly, particularly README sections, changelogs, installation guides, and API documentation - Produces syntactically well-formed code blocks across multiple languages (Python, Rust, Go, C++, JavaScript) - Sustains a single topic for several paragraphs when the prompt closely matches training distribution (e.g. `# Getting Started with `) ### Restrictions - Not an instruction-following model — treats all input as a document prefix to continue, not a query to answer - Out-of-distribution prompts (natural language, fiction, conversation) produce incoherent or nonsensical output - Prone to topic drift over longer generations, gradually sliding into unrelated documentation - Prone to repetition loops, particularly on short or ambiguous prompts - Generates hallucinated URLs, package names, library names, and version numbers with no grounding - Multilingual output may appear mid-generation, inherited from non-English READMEs in the training corpus; coherence in non-English output is lower than English - Not suitable for any production use ## Inference ```python #!/usr/bin/env python3 """ Tiny Mistral REPL demo — streaming tokens (TextStreamer if available, else manual sampling). Commands: :quit, :help, :show, :set (max_new_tokens, temperature, top_p, full_output) """ from __future__ import annotations import shlex import time import torch from typing import Optional from transformers import AutoTokenizer, MistralForCausalLM # --------- CONFIG ---------- MODEL_DIR = "Harley-ml/Mini-MD" TOKENIZER_DIR = MODEL_DIR DEFAULT_MAX_NEW_TOKENS = 640 DEFAULT_TEMPERATURE = 0.9 DEFAULT_TOP_P = 1.4 DEVICE = "cuda" if torch.cuda.is_available() else "cpu" PROMPT = ">>> " # --------------------------- def load_tokenizer(path: str): print("Loading tokenizer...", path) tok = AutoTokenizer.from_pretrained(path, use_fast=True, local_files_only=False) if tok.pad_token is None: if getattr(tok, "eos_token", None) is not None: tok.add_special_tokens({"pad_token": tok.eos_token}) else: tok.add_special_tokens({"pad_token": "", "eos_token": ""}) print("Tokenizer ready. vocab_size=", getattr(tok, "vocab_size", "N/A")) return tok def load_model(path: str, device: str): print("Loading model...", path) model = None try: desired_dtype = torch.float16 if device.startswith("cuda") else torch.float32 model = MistralForCausalLM.from_pretrained(path, local_files_only=False, dtype=desired_dtype) print("Loaded with dtype arg.") except TypeError: model = MistralForCausalLM.from_pretrained(path, local_files_only=False) print("Loaded without dtype; will convert.") except Exception as e: print("Load warning, retrying without dtype:", e) model = MistralForCausalLM.from_pretrained(path, local_files_only=False) try: model.to(device) if device.startswith("cuda") and next(model.parameters()).dtype != torch.float16: model.half() if not device.startswith("cuda") and next(model.parameters()).dtype != torch.float32: model.to(torch.float32) except Exception as e: print("Model move/convert warning:", e) model.config.pad_token_id = getattr(model.config, "pad_token_id", None) model.eval() return model # Simple nucleus/top-p filtering for a single logits vector def top_p_filtering(logits: torch.Tensor, top_p: float, min_keep: int = 1) -> torch.Tensor: if top_p <= 0 or top_p >= 1.0: return logits sorted_logits, sorted_idx = torch.sort(logits, descending=True) probs = torch.softmax(sorted_logits, dim=-1) cumprobs = torch.cumsum(probs, dim=-1) cutoff = (cumprobs > top_p).nonzero(as_tuple=False) if cutoff.numel() > 0: idx = int(cutoff[0].item()) cutoff_idx = max(idx + 1, min_keep) else: cutoff_idx = sorted_logits.size(-1) mask = torch.ones_like(sorted_logits, dtype=torch.bool) mask[cutoff_idx:] = False filtered = sorted_logits.masked_fill(~mask, -float("inf")) return torch.empty_like(filtered).scatter_(0, sorted_idx, filtered) # Manual streaming generator (single-batch) def manual_stream_generate(model, tokenizer, prompt: str, device: str, max_new_tokens: int = 64, temperature: float = 1.0, top_p: float = 0.9, eos_token_id: Optional[int] = None): inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False) input_ids = inputs["input_ids"].to(device) attention_mask = inputs.get("attention_mask", None) if attention_mask is not None: attention_mask = attention_mask.to(device) past = None with torch.no_grad(): out = model(input_ids=input_ids, attention_mask=attention_mask, use_cache=True) past = getattr(out, "past_key_values", None) # start sampling tokens next_input = input_ids[:, -1:].to(device) if past is not None else input_ids.to(device) for _ in range(max_new_tokens): with torch.no_grad(): out = model(input_ids=next_input, past_key_values=past, use_cache=True) logits = out.logits[:, -1, :] # (batch, vocab) past = getattr(out, "past_key_values", past) if temperature != 1.0: logits = logits / max(temperature, 1e-8) filtered = top_p_filtering(logits[0].cpu(), top_p).to(device) probs = torch.nn.functional.softmax(filtered.unsqueeze(0), dim=-1) next_token = torch.multinomial(probs, num_samples=1) token_id = int(next_token[0, 0].item()) token_text = tokenizer.decode([token_id], clean_up_tokenization_spaces=False) yield token_id, token_text if eos_token_id is not None and token_id == eos_token_id: break next_input = torch.tensor([[token_id]], dtype=torch.long, device=device) def has_text_streamer(): try: from transformers import TextStreamer # type: ignore return True except Exception: return False # tiny REPL state class State: def __init__(self): self.max_new_tokens = DEFAULT_MAX_NEW_TOKENS self.temperature = DEFAULT_TEMPERATURE self.top_p = DEFAULT_TOP_P self.full_output = False self.stream = True def handle_generation(model, tokenizer, prompt: str, device: str, state: State): eos = getattr(tokenizer, "eos_token_id", None) try: if has_text_streamer(): from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip_prompt=not state.full_output, skip_special_tokens=True) inputs = tokenizer(prompt, return_tensors="pt", truncation=True, add_special_tokens=False) inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)} inputs.pop("token_type_ids", None) model.generate(**inputs, max_new_tokens=state.max_new_tokens, do_sample=True, temperature=state.temperature, top_p=state.top_p, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id, streamer=streamer) print("") # newline after streamer return # fallback: manual streaming gen = manual_stream_generate(model, tokenizer, prompt, device, max_new_tokens=state.max_new_tokens, temperature=state.temperature, top_p=state.top_p, eos_token_id=eos) if state.full_output: print("PROMPT:", prompt) print("GENERATING:", end=" ", flush=True) else: print("GENERATING:", end=" ", flush=True) count = 0 t0 = time.time() for _tok_id, tok_text in gen: count += 1 print(tok_text, end="", flush=True) print() print(f"(generated {count} tokens in {time.time()-t0:.2f}s)") except KeyboardInterrupt: print("\n[interrupted] Generation aborted by user.") except Exception as e: print("Generation error:", e) def repl(model, tokenizer, device): state = State() help_text = ( "Commands:\n" " :quit\n" " :help\n" " :show\n" " :set # params: max_new_tokens, temperature, top_p, full_output, stream\n" " (blank line repeats last prompt)\n" ) print("Tiny Mistral REPL — device:", device) print(help_text) last = "" while True: try: raw = input(PROMPT).strip() except (EOFError, KeyboardInterrupt): print("\nExiting.") break if not raw: raw = last if not raw: continue if raw.startswith(":"): toks = shlex.split(raw) cmd = toks[0].lower() if cmd == ":quit": print("bye.") break if cmd == ":help": print(help_text); continue if cmd == ":show": print(f"max_new_tokens={state.max_new_tokens}, temperature={state.temperature}, top_p={state.top_p}, full_output={state.full_output}, stream={state.stream}") continue if cmd == ":set": if len(toks) < 3: print("usage: :set "); continue k, v = toks[1], toks[2] try: if k == "max_new_tokens": state.max_new_tokens = int(v) elif k == "temperature": state.temperature = float(v) elif k == "top_p": state.top_p = float(v) elif k in ("full_output", "full"): state.full_output = v.lower() in ("1", "true", "yes", "y") elif k == "stream": state.stream = v.lower() in ("1", "true", "yes", "y") else: print("unknown param:", k) continue print("OK.") except Exception as e: print("set error:", e) continue print("unknown command") continue last = raw if state.stream: handle_generation(model, tokenizer, raw, device, state) else: # non-streaming generate try: inputs = tokenizer(raw, return_tensors="pt", truncation=True, add_special_tokens=False) inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)} inputs.pop("token_type_ids", None) out = model.generate(**inputs, max_new_tokens=state.max_new_tokens, do_sample=True, temperature=state.temperature, top_p=state.top_p, pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id) seq = out[0] input_len = inputs["input_ids"].shape[1] if "input_ids" in inputs else 0 text = tokenizer.decode(seq if state.full_output else seq[input_len:], skip_special_tokens=True) print("\nOUTPUT\n", text) except Exception as e: print("Generation failed:", e) def main(): device = DEVICE tokenizer = load_tokenizer(TOKENIZER_DIR) model = load_model(MODEL_DIR, device) repl(model, tokenizer, device) if __name__ == "__main__": main() ```