thegadgetglare/lighthouse-100k-chatbot

Lighthouse 100k Chatbot

This is a 180,968-parameter model trained on TinyStories using Lighthouse Attention.

Model Details

• Parameters: 180,968
• Context Length: 256
• Vocab Size: 1000
• Architecture: GPT with Lighthouse Attention
• Training: Two-stage (95% Lighthouse, 5% Dense Recovery)
• Dataset: TinyStories
• Validation Perplexity: 66.38

Usage

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

# Download model
model_path = hf_hub_download(
    repo_id="thegadgetglare/lighthouse-100k-chatbot",
    filename="model.safetensors"
)

# Download tokenizer
tokenizer_path = hf_hub_download(
    repo_id="thegadgetglare/lighthouse-100k-chatbot",
    filename="tokenizer.json"
)

tokenizer = Tokenizer.from_file(tokenizer_path)
state_dict = load_file(model_path)

Try It

Check out the Space: https://huggingface.co/spaces/thegadgetglare/lighthouse-100k-chatbot