File size: 2,368 Bytes
64728f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | # Custom LLM Model
This is a custom-built language model trained on a small dataset of example sentences about AI and machine learning.
## Model Description
- **Model Type**: Transformer-based language model
- **Vocabulary Size**: 40 characters
- **Hidden Size**: 256
- **Number of Layers**: 4
- **Number of Attention Heads**: 8
- **Feedforward Size**: 1024
- **Max Sequence Length**: 64
- **Training Epochs**: 10
- **Parameters**: ~3.2M
## Training Data
The model was trained on a small custom dataset containing 10 example sentences about:
- Greetings and small talk
- Weather descriptions
- Machine learning concepts
- Deep learning and transformers
- Natural language processing
- Model publishing and sharing
## Usage
```python
import torch
from train_model import TransformerLM, CharacterTokenizer
# Load the saved model
checkpoint = torch.load('custom_llm_model.pth', map_location='cpu')
model_config = checkpoint['model_config']
tokenizer = checkpoint['tokenizer']
# Initialize model
model = TransformerLM(**model_config)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
# Generate text
def generate_text(prompt, max_length=50, temperature=0.8):
# Tokenize prompt
input_ids = tokenizer.encode(prompt, max_length=32, padding=False, return_tensors='pt')
generated = input_ids.clone()
with torch.no_grad():
for _ in range(max_length):
logits = model(generated)
next_token_logits = logits[0, -1, :] / temperature
probs = torch.softmax(next_token_logits, dim=-1)
next_token = torch.multinomial(probs, num_samples=1)
generated = torch.cat([generated, next_token.unsqueeze(0)], dim=1)
# Stop on period or max length
if next_token.item() == tokenizer.char_to_idx.get('.', tokenizer.unk_token_id):
break
return tokenizer.decode(generated[0])
# Example usage
print(generate_text("Hello"))
print(generate_text("The weather"))
print(generate_text("Deep learning"))
```
## Limitations
This is a small demonstration model trained on very limited data. For serious applications, consider:
- Using larger datasets
- Training for more epochs
- Using larger model architectures
- Implementing proper tokenization (BPE, WordPiece, etc.)
## License
This model is released under the MIT License. |