Update README.md

README.md

@@ -97,9 +97,17 @@ None of the benchmarks above directly test whether the model's writing holds tog
 
 **Prompt:** "Once upon a time, there was a"
 
-> Once upon a time, there was a man named Peter who lived in a beautiful country called Ireland. His father was a wise old man who was known for his knowledge of geometry. One day, a young girl named Alice came to visit Peter in Ireland. As they walked together, Alice met a boy named Leo, who had big shoes and a big hat. Alice asked him, "What are you doing with that, dear?" Peter told Alice about a time when he was a boy. He was still trying to understand how things worked. One day, a clever fox came to Peter and asked him how he could have fun. Peter smiled and said, "That's because he learned to ride an imaginary plane!" Alice watched as Peter explained how he learned how to climb trees and follow a river. "You've come to know that, Peter," she said, and Emmanuel smiled. They sat down together and talked about their favorite things, like building a bridge to reach a friend and helping a neighbor.
+> Once upon a time, there was a wise old turtle named Timmy who lived on the coast of South America. In this magical place, no matter how big or large, people could look up at the sea and talk to each other.
 
-There is an actual through line here. Peter gets introduced with a name and a place, Alice arrives and has a real interaction with him, a fox shows up mid story and gets a response, and the ending calls back to the "learning" theme set up in the first sentence. It is not perfectly logical throughout, but it tracks entities and builds toward something across sentences, which is exactly what v1 could not do.
+Timmy asked, "What do you mean, Timmy?"
+
+The turtle replied, "Well, I think you might see people talking about ocean creatures. They are like little waves that carry their voices. Sometimes they say they're too big or small to hear."
+
+After a few moments, Timmy had an idea. "Can we go on a boat-boat tour? I can't believe all the kids in the village are doing that!"
+
+As they sailed further, they saw many beautiful islands and vibrant colors. Each island had its unique culture and traditions. When they reached the top, they saw a group of kids playing and splashing around.
+
+"Wow, lookgies!" said Timmy. "They live in a big ocean full of colorful fish and
 
 ---
 
@@ -148,6 +156,71 @@ Custom byte-level BPE tokenizer trained from scratch on a sample of the pretrain
 
 ---
 
+## Inference 
+
+Here's a basic inference code you can run to immediately start using İvme-Conversate-v2-Base
+
+```python
+import sys
+import torch
+from tokenizers import Tokenizer
+from huggingface_hub import hf_hub_download, snapshot_download
+
+repo_id = "IvmeLabs/Ivme-Conversate-v2-Base"
+
+# Download just the model/ folder (architecture code) into the HF cache,
+# then add it to sys.path so `from model import ...` works without the
+# user needing to manually copy any files.
+repo_local_dir = snapshot_download(repo_id, allow_patterns=["model/*"])
+sys.path.append(repo_local_dir)
+
+from model import IvmeConfig, IvmeConversateV2
+
+ckpt_path = hf_hub_download(repo_id, "ckpt_final.pt")
+tokenizer_path = hf_hub_download(repo_id, "tokenizer.json")
+
+tokenizer = Tokenizer.from_file(tokenizer_path)
+
+# IvmeConfig is a plain dataclass saved into the checkpoint. Trust this only
+# because it's our own checkpoint, produced by our own training code.
+torch.serialization.add_safe_globals([IvmeConfig])
+ckpt = torch.load(ckpt_path, map_location="cuda")
+cfg = ckpt["config"]
+
+model = IvmeConversateV2(cfg)
+
+# Use EMA weights (smoothed), not the raw training weights, for inference.
+# Strip torch.compile's "_orig_mod." prefix if the checkpoint was compiled.
+state_dict = ckpt["ema_state_dict"]
+state_dict = {k.removeprefix("_orig_mod."): v for k, v in state_dict.items()}
+model.load_state_dict(state_dict)
+
+model.cuda().eval()
+
+prompt = "Once upon a time, there was a"
+ids = tokenizer.encode(prompt).ids
+idx = torch.tensor([ids], dtype=torch.long, device="cuda")
+
+eot_id = tokenizer.token_to_id("<|endoftext|>")
+with torch.no_grad():
+    for _ in range(200):
+        idx_cond = idx if idx.size(1) <= cfg.context_len else idx[:, -cfg.context_len:]
+        logits, _ = model(idx_cond)
+        logits = logits[:, -1, :] / 0.8  # temperature
+
+        v, _ = torch.topk(logits, 50)
+        logits[logits < v[:, [-1]]] = -float("inf")
+
+        probs = torch.softmax(logits, dim=-1)
+        next_id = torch.multinomial(probs, num_samples=1)
+        idx = torch.cat([idx, next_id], dim=1)
+
+        if next_id.item() == eot_id:
+            break
+
+print(tokenizer.decode(idx[0].tolist()))
+```
+
 ## Limitations
 
 - Base model only, not instruction tuned, will not follow instructions or answer questions