|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- mehta/CooperLM-354M |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
tags: |
|
|
- toy-llm |
|
|
- gpt2 |
|
|
- 4bit |
|
|
- quantized |
|
|
- casual-lm |
|
|
- transformers |
|
|
- small-llm |
|
|
--- |
|
|
|
|
|
# 🧠 CooperLM-354M (4-bit Quantized) |
|
|
|
|
|
This is a 4-bit quantized version of [CooperLM-354M](https://huggingface.co/mehta/CooperLM-354M), a 354M parameter GPT-2 style language model trained from scratch on a subset of Wikipedia, BookCorpus, and OpenWebText. |
|
|
|
|
|
The quantized model is intended for faster inference and smaller memory footprint, especially useful for CPU or limited-GPU setups. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📌 Model Details |
|
|
|
|
|
- **Base Model**: [mehta/CooperLM-354M](https://huggingface.co/mehta/CooperLM-354M) |
|
|
- **Architecture**: GPT-2 (24 layers, 16 heads, 1024 hidden size) |
|
|
- **Quantization**: 4-bit integer weights via `AutoGPTQ` (safetensors) |
|
|
- **Precision**: INT4 |
|
|
|
|
|
--- |
|
|
|
|
|
## 🛠️ How to Use |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
import torch |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("mehta/CooperLM-354M-4bit") |
|
|
model = AutoModelForCausalLM.from_pretrained("mehta/CooperLM-354M-4bit") |
|
|
|
|
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
|
model.to(device) |
|
|
|
|
|
prompt = "In the distant future," |
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(device) |
|
|
|
|
|
outputs = model.generate( |
|
|
**inputs, |
|
|
max_length=100, |
|
|
temperature=0.8, |
|
|
top_p=0.95, |
|
|
do_sample=True |
|
|
) |
|
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
