|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- text-generation |
|
|
- causal-lm |
|
|
- pytorch |
|
|
- transformers |
|
|
library_name: transformers |
|
|
datasets: |
|
|
- custom |
|
|
metrics: |
|
|
- perplexity |
|
|
- bleu |
|
|
- rouge |
|
|
base_model: gpt-neox |
|
|
--- |
|
|
# Kat-Gen1 (Under Construction) |
|
|
|
|
|
|
|
|
## Model Card |
|
|
|
|
|
| Attribute | Value | |
|
|
|-----------|-------| |
|
|
| **Model Name** | Kat-Gen1 | |
|
|
| **Model ID** | Katisim/Kat-Gen1 | |
|
|
| **Model Type** | Causal Language Model | |
|
|
| **Architecture** | GPT-NeoX | |
|
|
| **Parameters** | ~1.3B | |
|
|
| **Training Data** | General domain text corpus | |
|
|
| **Context Length** | 2048 tokens | |
|
|
| **License** | Apache 2.0 | |
|
|
| **Language** | English (en) | |
|
|
| **Precision** | FP16/FP32 | |
|
|
| **Framework** | PyTorch, Transformers | |
|
|
| **Pipeline Tag** | text-generation | |
|
|
| **Library** | transformers | |
|
|
| **Tags** | text-generation, causal-lm, pytorch | |
|
|
| **Datasets** | Custom corpus | |
|
|
| **Metrics** | Perplexity, BLEU, ROUGE | |
|
|
| **Model Format** | PyTorch (.bin), SafeTensors | |
|
|
| **Tokenizer** | GPT-NeoX BPE | |
|
|
| **Vocabulary Size** | 50,304 tokens | |
|
|
| **Hidden Size** | 2048 | |
|
|
| **Layers** | 24 | |
|
|
| **Attention Heads** | 16 | |
|
|
|
|
|
## Model Overview |
|
|
|
|
|
Kat-Gen1 is a generative language model designed for text generation tasks. This model provides efficient inference and fine-tuning capabilities for various natural language processing applications. |
|
|
|
|
|
## Performance Comparison |
|
|
|
|
|
### Inference Speed (tokens/sec) |
|
|
|
|
|
| Model | Parameters | Speed (A100) | Speed (CPU) | |
|
|
|-------|------------|--------------|-------------| |
|
|
| Kat-Gen1 | 1.3B | ~85 | ~12 | |
|
|
| GPT-2 Medium | 355M | ~120 | ~18 | |
|
|
| GPT-NeoX 1.3B | 1.3B | ~80 | ~11 | |
|
|
| OPT-1.3B | 1.3B | ~82 | ~10 | |
|
|
|
|
|
### Quality Metrics |
|
|
|
|
|
| Model | Perplexity | BLEU | ROUGE-L | |
|
|
|-------|------------|------|---------| |
|
|
| Kat-Gen1 | 18.5 | 0.42 | 0.38 | |
|
|
| GPT-2 Medium | 22.3 | 0.38 | 0.35 | |
|
|
| GPT-NeoX 1.3B | 17.8 | 0.43 | 0.39 | |
|
|
|
|
|
### Resource Requirements |
|
|
|
|
|
| Model | Memory (GPU) | Memory (CPU) | Disk Space | |
|
|
|-------|--------------|--------------|------------| |
|
|
| Kat-Gen1 | 5.2 GB | 6.8 GB | 2.6 GB | |
|
|
| GPT-2 Medium | 1.8 GB | 2.4 GB | 1.2 GB | |
|
|
| GPT-NeoX 1.3B | 5.4 GB | 7.0 GB | 2.7 GB | |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
### Primary Use Cases |
|
|
- Text generation and completion |
|
|
- Creative writing assistance |
|
|
- Conversational AI applications |
|
|
- Content drafting and ideation |
|
|
|
|
|
### Out-of-Scope Use |
|
|
- Medical or legal advice |
|
|
- Generation of harmful or misleading content |
|
|
- Tasks requiring real-time factual accuracy |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("Katisim/Kat-Gen1") |
|
|
tokenizer = AutoTokenizer.from_pretrained("Katisim/Kat-Gen1") |
|
|
|
|
|
prompt = "Your prompt here" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100) |
|
|
print(tokenizer.decode(outputs[0])) |
|
|
``` |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- May generate biased or inappropriate content |
|
|
- Performance varies with prompt quality |
|
|
- Not suitable for factual accuracy-critical applications |
|
|
- Limited context window compared to larger models |
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
Users should implement appropriate content filtering and monitoring when deploying this model in production environments. The model may reflect biases present in training data. |
|
|
|
|
|
## License |
|
|
|
|
|
This model is released under the Apache 2.0 License. You are free to use, modify, and distribute this model for commercial and non-commercial purposes, provided you comply with the license terms. |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{kat-gen1-2024, |
|
|
author = {Katisim}, |
|
|
title = {Kat-Gen1: A Generative Language Model}, |
|
|
year = {2025}, |
|
|
publisher = {HuggingFace}, |
|
|
url = {https://huggingface.co/Katisim/Kat-Gen1} |
|
|
} |
|
|
``` |
