Mini-MD
Mini-MD is a ~28M parameter transformer-decoder trained on ~200k markdown files from Github.
Architecture
| Key | Value |
|---|---|
hidden_size |
384 |
num_layers |
8 |
num_heads |
6 |
num_kv_heads |
2 |
head_dim |
64 |
intermediate_size |
1536 |
vocab_size |
14002 |
sliding_window |
640 |
rope_theta |
10000.0 |
tie_embeddings |
True |
total_params |
28061568 |
Training
Training Parameters
| Key | Value |
|---|---|
num_epochs |
3 |
batch_size |
5 |
stride |
620 |
seq_len |
640 |
val_split |
0.09 |
learning_rate |
2e-4 |
Training Results
train_loss |
val_loss |
step |
epoch |
|---|---|---|---|
| 6.8138 | 5.7706 | 1200 | 0.02 |
| 2.4274 | 2.5915 | 12000 | 0.24 |
| 2.1519 | 2.2091 | 30000 | 0.59 |
| 2.0411 | 2.0464 | 48000 | 0.95 |
| 1.7728 | 1.8912 | 84000 | 1.66 |
| 1.7304 | 1.8494 | 100800 | 1.99 |
| 1.6394 | 1.7599 | 132000 | 2.60 |
| 1.6794 | 1.7234 | 151200 | 2.98 |
Hardware
GPU: One singular NVIDIA RTX 2060 6GB (14GB Shared RAM).
CPU: Ryzen 5 2600
RAM: 16GB
Generations
Input:
# README
Output:
Cover Evolution
Official documentation for dis curated documentation: cosu:
**README.md** (Recommended for reference)
- **Repository**:
- **Bloomberg**:
**Welcome to Open Questions about Apache Dis curriculum**
- **Security:** All files are securely required through social media.
- **International (FONTLOG)**: Only include relevant code and resources when necessary to see new or better info.
Input:
# Getting Started with Rust
Output:
You need a Rust crate for your operating system or platform environment, but you can
install the Rust organization for Rust programs, and it's enough you'd really want it
to work with your operating system.
### Building Tools
#### FFI Compatibility
Rust's Rust has a compiler that can handle the following:
- Using a toolchain with the AVX-based kernels
- FFI Compatibility on ARM for iOS, iOS, Robot, etc
- FFI Support from Linux for XPC platforms
Input:
## Installation
Output:
in this project
For building the Heft and French benchmarks, there are two popular tools you can use to build
various benchmarks:
* [Jim]( and [iL2](
* [Lucid]( is supported, with [t5]( the
[`LuxSrc32`]( CLI to get a better benchmark
## Usage in this project:
Npm install
[LuxSrc32]( It is recommended to use precompiled GopherLM files
Input:
The quick brown fox
Output:
ispepper is the skype of the games and you just want to generate them to your be.
To use this you will need a Galicu sensor that has 2x 3x2 pesus.
The other four mandaracters share it - you want the German light/highest food wey no
your browses for game. Geometric, like grove, grove, and bus are pencilumo.
Input:
Hello world
Output:
I can find a brown, I know it may need the full-fledip data into my phone
buying you...
my first time I went my email, we're going through it.
You should now see you.
Thank you the wonderful! Thanks! It's very cool when the dog would bed.
We can't go through the world until you've started it!
Input:
Once upon a time,
Output:
all data are moved to any place the user is aware of their operation and then removed.
When the following is present:
* The ability to move state through the same service, which allows
the ability of clients to move state across a network interface.
During a [DaemonSet controller](/docs/concepts/cluster-administration/deployment/),
all webhooks of the DaemonSet controller may be invoked from a user through a
"Service Cluster" resource without any actions.
Capabilities & Restrictions
Capabilities
- Continues GitHub-style markdown documents plausibly, particularly README sections, changelogs, installation guides, and API documentation
- Produces syntactically well-formed code blocks across multiple languages (Python, Rust, Go, C++, JavaScript)
- Sustains a single topic for several paragraphs when the prompt closely matches training distribution (e.g.
# Getting Started with <common language>)
Restrictions
- Not an instruction-following model — treats all input as a document prefix to continue, not a query to answer
- Out-of-distribution prompts (natural language, fiction, conversation) produce incoherent or nonsensical output
- Prone to topic drift over longer generations, gradually sliding into unrelated documentation
- Prone to repetition loops, particularly on short or ambiguous prompts
- Generates hallucinated URLs, package names, library names, and version numbers with no grounding
- Multilingual output may appear mid-generation, inherited from non-English READMEs in the training corpus; coherence in non-English output is lower than English
- Not suitable for any production use
Inference
#!/usr/bin/env python3
"""
Tiny Mistral REPL demo — streaming tokens (TextStreamer if available, else manual sampling).
Commands: :quit, :help, :show, :set <param> <value> (max_new_tokens, temperature, top_p, full_output)
"""
from __future__ import annotations
import shlex
import time
import torch
from typing import Optional
from transformers import AutoTokenizer, MistralForCausalLM
# --------- CONFIG ----------
MODEL_DIR = "Harley-ml/Mini-MD"
TOKENIZER_DIR = MODEL_DIR
DEFAULT_MAX_NEW_TOKENS = 640
DEFAULT_TEMPERATURE = 0.9
DEFAULT_TOP_P = 1.4
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
PROMPT = ">>> "
# ---------------------------
def load_tokenizer(path: str):
print("Loading tokenizer...", path)
tok = AutoTokenizer.from_pretrained(path, use_fast=True, local_files_only=False)
if tok.pad_token is None:
if getattr(tok, "eos_token", None) is not None:
tok.add_special_tokens({"pad_token": tok.eos_token})
else:
tok.add_special_tokens({"pad_token": "<pad>", "eos_token": "</s>"})
print("Tokenizer ready. vocab_size=", getattr(tok, "vocab_size", "N/A"))
return tok
def load_model(path: str, device: str):
print("Loading model...", path)
model = None
try:
desired_dtype = torch.float16 if device.startswith("cuda") else torch.float32
model = MistralForCausalLM.from_pretrained(path, local_files_only=False, dtype=desired_dtype)
print("Loaded with dtype arg.")
except TypeError:
model = MistralForCausalLM.from_pretrained(path, local_files_only=False)
print("Loaded without dtype; will convert.")
except Exception as e:
print("Load warning, retrying without dtype:", e)
model = MistralForCausalLM.from_pretrained(path, local_files_only=False)
try:
model.to(device)
if device.startswith("cuda") and next(model.parameters()).dtype != torch.float16:
model.half()
if not device.startswith("cuda") and next(model.parameters()).dtype != torch.float32:
model.to(torch.float32)
except Exception as e:
print("Model move/convert warning:", e)
model.config.pad_token_id = getattr(model.config, "pad_token_id", None)
model.eval()
return model
# Simple nucleus/top-p filtering for a single logits vector
def top_p_filtering(logits: torch.Tensor, top_p: float, min_keep: int = 1) -> torch.Tensor:
if top_p <= 0 or top_p >= 1.0:
return logits
sorted_logits, sorted_idx = torch.sort(logits, descending=True)
probs = torch.softmax(sorted_logits, dim=-1)
cumprobs = torch.cumsum(probs, dim=-1)
cutoff = (cumprobs > top_p).nonzero(as_tuple=False)
if cutoff.numel() > 0:
idx = int(cutoff[0].item())
cutoff_idx = max(idx + 1, min_keep)
else:
cutoff_idx = sorted_logits.size(-1)
mask = torch.ones_like(sorted_logits, dtype=torch.bool)
mask[cutoff_idx:] = False
filtered = sorted_logits.masked_fill(~mask, -float("inf"))
return torch.empty_like(filtered).scatter_(0, sorted_idx, filtered)
# Manual streaming generator (single-batch)
def manual_stream_generate(model, tokenizer, prompt: str, device: str,
max_new_tokens: int = 64, temperature: float = 1.0, top_p: float = 0.9,
eos_token_id: Optional[int] = None):
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
input_ids = inputs["input_ids"].to(device)
attention_mask = inputs.get("attention_mask", None)
if attention_mask is not None:
attention_mask = attention_mask.to(device)
past = None
with torch.no_grad():
out = model(input_ids=input_ids, attention_mask=attention_mask, use_cache=True)
past = getattr(out, "past_key_values", None)
# start sampling tokens
next_input = input_ids[:, -1:].to(device) if past is not None else input_ids.to(device)
for _ in range(max_new_tokens):
with torch.no_grad():
out = model(input_ids=next_input, past_key_values=past, use_cache=True)
logits = out.logits[:, -1, :] # (batch, vocab)
past = getattr(out, "past_key_values", past)
if temperature != 1.0:
logits = logits / max(temperature, 1e-8)
filtered = top_p_filtering(logits[0].cpu(), top_p).to(device)
probs = torch.nn.functional.softmax(filtered.unsqueeze(0), dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
token_id = int(next_token[0, 0].item())
token_text = tokenizer.decode([token_id], clean_up_tokenization_spaces=False)
yield token_id, token_text
if eos_token_id is not None and token_id == eos_token_id:
break
next_input = torch.tensor([[token_id]], dtype=torch.long, device=device)
def has_text_streamer():
try:
from transformers import TextStreamer # type: ignore
return True
except Exception:
return False
# tiny REPL state
class State:
def __init__(self):
self.max_new_tokens = DEFAULT_MAX_NEW_TOKENS
self.temperature = DEFAULT_TEMPERATURE
self.top_p = DEFAULT_TOP_P
self.full_output = False
self.stream = True
def handle_generation(model, tokenizer, prompt: str, device: str, state: State):
eos = getattr(tokenizer, "eos_token_id", None)
try:
if has_text_streamer():
from transformers import TextStreamer
streamer = TextStreamer(tokenizer, skip_prompt=not state.full_output, skip_special_tokens=True)
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, add_special_tokens=False)
inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)}
inputs.pop("token_type_ids", None)
model.generate(**inputs,
max_new_tokens=state.max_new_tokens,
do_sample=True,
temperature=state.temperature,
top_p=state.top_p,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
streamer=streamer)
print("") # newline after streamer
return
# fallback: manual streaming
gen = manual_stream_generate(model, tokenizer, prompt, device,
max_new_tokens=state.max_new_tokens,
temperature=state.temperature,
top_p=state.top_p,
eos_token_id=eos)
if state.full_output:
print("PROMPT:", prompt)
print("GENERATING:", end=" ", flush=True)
else:
print("GENERATING:", end=" ", flush=True)
count = 0
t0 = time.time()
for _tok_id, tok_text in gen:
count += 1
print(tok_text, end="", flush=True)
print()
print(f"(generated {count} tokens in {time.time()-t0:.2f}s)")
except KeyboardInterrupt:
print("\n[interrupted] Generation aborted by user.")
except Exception as e:
print("Generation error:", e)
def repl(model, tokenizer, device):
state = State()
help_text = (
"Commands:\n"
" :quit\n"
" :help\n"
" :show\n"
" :set <param> <value> # params: max_new_tokens, temperature, top_p, full_output, stream\n"
" (blank line repeats last prompt)\n"
)
print("Tiny Mistral REPL — device:", device)
print(help_text)
last = ""
while True:
try:
raw = input(PROMPT).strip()
except (EOFError, KeyboardInterrupt):
print("\nExiting.")
break
if not raw:
raw = last
if not raw:
continue
if raw.startswith(":"):
toks = shlex.split(raw)
cmd = toks[0].lower()
if cmd == ":quit":
print("bye.")
break
if cmd == ":help":
print(help_text); continue
if cmd == ":show":
print(f"max_new_tokens={state.max_new_tokens}, temperature={state.temperature}, top_p={state.top_p}, full_output={state.full_output}, stream={state.stream}")
continue
if cmd == ":set":
if len(toks) < 3:
print("usage: :set <param> <value>"); continue
k, v = toks[1], toks[2]
try:
if k == "max_new_tokens":
state.max_new_tokens = int(v)
elif k == "temperature":
state.temperature = float(v)
elif k == "top_p":
state.top_p = float(v)
elif k in ("full_output", "full"):
state.full_output = v.lower() in ("1", "true", "yes", "y")
elif k == "stream":
state.stream = v.lower() in ("1", "true", "yes", "y")
else:
print("unknown param:", k)
continue
print("OK.")
except Exception as e:
print("set error:", e)
continue
print("unknown command")
continue
last = raw
if state.stream:
handle_generation(model, tokenizer, raw, device, state)
else:
# non-streaming generate
try:
inputs = tokenizer(raw, return_tensors="pt", truncation=True, add_special_tokens=False)
inputs = {k: v.to(device) for k, v in inputs.items() if isinstance(v, torch.Tensor)}
inputs.pop("token_type_ids", None)
out = model.generate(**inputs,
max_new_tokens=state.max_new_tokens,
do_sample=True,
temperature=state.temperature,
top_p=state.top_p,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id)
seq = out[0]
input_len = inputs["input_ids"].shape[1] if "input_ids" in inputs else 0
text = tokenizer.decode(seq if state.full_output else seq[input_len:], skip_special_tokens=True)
print("\nOUTPUT\n", text)
except Exception as e:
print("Generation failed:", e)
def main():
device = DEVICE
tokenizer = load_tokenizer(TOKENIZER_DIR)
model = load_model(MODEL_DIR, device)
repl(model, tokenizer, device)
if __name__ == "__main__":
main()
- Downloads last month
- 37
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support