custom_llm_project/README.md

Custom LLM Model

This is a custom-built language model trained on a small dataset of example sentences about AI and machine learning.

Model Description

• Model Type: Transformer-based language model
• Vocabulary Size: 40 characters
• Hidden Size: 256
• Number of Layers: 4
• Number of Attention Heads: 8
• Feedforward Size: 1024
• Max Sequence Length: 64
• Training Epochs: 10
• Parameters: ~3.2M

Training Data

The model was trained on a small custom dataset containing 10 example sentences about:

• Greetings and small talk
• Weather descriptions
• Machine learning concepts
• Deep learning and transformers
• Natural language processing
• Model publishing and sharing

Usage

import torch
from train_model import TransformerLM, CharacterTokenizer

# Load the saved model
checkpoint = torch.load('custom_llm_model.pth', map_location='cpu')
model_config = checkpoint['model_config']
tokenizer = checkpoint['tokenizer']

# Initialize model
model = TransformerLM(**model_config)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Generate text
def generate_text(prompt, max_length=50, temperature=0.8):
    # Tokenize prompt
    input_ids = tokenizer.encode(prompt, max_length=32, padding=False, return_tensors='pt')
    generated = input_ids.clone()
    
    with torch.no_grad():
        for _ in range(max_length):
            logits = model(generated)
            next_token_logits = logits[0, -1, :] / temperature
            probs = torch.softmax(next_token_logits, dim=-1)
            next_token = torch.multinomial(probs, num_samples=1)
            generated = torch.cat([generated, next_token.unsqueeze(0)], dim=1)
            
            # Stop on period or max length
            if next_token.item() == tokenizer.char_to_idx.get('.', tokenizer.unk_token_id):
                break
    
    return tokenizer.decode(generated[0])

# Example usage
print(generate_text("Hello"))
print(generate_text("The weather"))
print(generate_text("Deep learning"))

Limitations

This is a small demonstration model trained on very limited data. For serious applications, consider:

• Using larger datasets
• Training for more epochs
• Using larger model architectures
• Implementing proper tokenization (BPE, WordPiece, etc.)

License

This model is released under the MIT License.