Erdenel.1

👉 Click here to chat with it

A 4,843,596-parameter character-level GPT, trained from scratch for about 10 hours on a CPU by Neural Labs using one million generated short conversations.

It can chat, remember your name for a few messages, do simple sums, count, spell and reverse words, and it knows capitals, opposites, animal facts and jokes. It scores 99.6% on 1,500 held-back test questions. Outside those skills it gives odd answers.

• 6 transformer layers, 8 attention heads, width 256, context of 256 characters
• model.safetensors holds the weights
• config.json holds the character list and the model settings
• model.py holds the model code

Use

import json
import torch
from safetensors.torch import load_file
from model import TinyGPT

cfg = json.load(open("config.json"))
chars = cfg.pop("chars")
model = TinyGPT(len(chars), **cfg)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

prompt = "User: what is the capital of Greece\nBot:"
idx = torch.tensor([[chars.index(c) for c in prompt]])
reply = ""
for i in model.generate(idx, max_new_tokens=80, temperature=0):
    if chars[i] == "\n":
        break
    reply += chars[i]
print(reply.strip())