metadata
language:
- en
license: apache-2.0
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- pretrained
- chytrej
- base
- tiny
library_name: transformers
datasets:
- HuggingFaceFW/fineweb-edu
Chytrej2-Mini
A fully custom pretrained language model built from scratch on the LLaMA architecture trained on 2B tokens of the FineWeb Edu dataset.
Built by PingVortex Labs.
Model Details
- Parameters: 20M
- Context length: 1024 tokens
- Language: English only
- Format: Base model
- Architecture: LLaMA
- License: Apache 2.0
Benchmark
- The model achieves score of 35.77% on ARC-Easy benchmark.
Usage
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-Mini")
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-Mini")
prompt = "Neural Networks are"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
print(tokenizer.decode(outputs[0]))
Made by PingVortex.