README.md

---
language:
- en
license: apache-2.0
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- pretrained
- chytrej
- base
- tiny
library_name: transformers
datasets:
- HuggingFaceFW/fineweb-edu
---

# Chytrej2-Mini

A fully custom pretrained language model built from scratch on the LLaMA architecture trained on 2B tokens of the [FineWeb Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset.

Built by [PingVortex Labs](https://github.com/PingVortexLabs).

[![Discord](https://img.shields.io/badge/Discord-5865F2?logo=discord&logoColor=white)](https://discord.gg/5SzkjVJBs2)

---

## Model Details

+ **Parameters:** 20M
+ **Context length:** 1024 tokens
+ **Language:** English only
+ **Format:** Base model
+ **Architecture:** LLaMA
+ **License:** Apache 2.0

---

## Benchmark
+ The model achieves score of 35.77% on ARC-Easy benchmark.

---

## Usage

```python
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-Mini")
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-Mini")

prompt = "Neural Networks are"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
print(tokenizer.decode(outputs[0]))
```

---

*Made by [PingVortex](https://pingvortex.com).*