metadata
license: apache-2.0
language:
- pt
library_name: transformers
tags:
- portuguese
- brazilian
- llama
- causal-lm
- text-generation
datasets:
- uonlp/CulturaX
- HuggingFaceFW/fineweb-2
- eduagarcia/cc_news_pt_v2
pipeline_tag: text-generation
NBR-1B: Brazilian Portuguese Language Model
NBR-1B is a 1.13 billion parameter language model trained from scratch for Brazilian Portuguese.
Model Details
| Attribute | Value |
|---|---|
| Parameters | 1.13B |
| Architecture | LLaMA-style (GQA, RMSNorm, SwiGLU, RoPE) |
| Hidden Size | 2048 |
| Layers | 24 |
| Attention Heads | 16 |
| KV Heads | 4 |
| Vocabulary | 32,000 (BPE) |
| Context Length | 2048 |
| Training Tokens | 3.12B |
| Final Loss | ~2.8 |
Training Data
- CulturaX PT (40%)
- FineWeb-2 PT (52%)
- mC4 PT (5%)
- CC-News PT v2 (2%)
- Books PT (1%)
Usage
This is a base model for text completion. Use with transformers library.
License
Apache 2.0