|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- pt |
|
|
pipeline_tag: text-generation |
|
|
parameters: 10k |
|
|
--- |
|
|
|
|
|
# MiniText-v1.0 |
|
|
|
|
|
## Model Summary |
|
|
|
|
|
MiniText-v1.0 is a tiny **character-level language model** trained from scratch |
|
|
to learn basic Portuguese text patterns. |
|
|
|
|
|
The goal of this project is to explore the **minimum viable neural architecture** |
|
|
capable of producing structured natural language, without pretraining, |
|
|
instruction tuning, or external corpora. |
|
|
|
|
|
This model is intended for **research, education, and experimentation**. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Architecture:** Custom MiniText (character-level) |
|
|
- **Training Objective:** Next-character prediction |
|
|
- **Vocabulary:** Byte-level (0–255) |
|
|
- **Language:** Portuguese (basic) |
|
|
- **Initialization:** Random (no pretrained weights) |
|
|
- **Training:** Single-stream autoregressive training |
|
|
- **Parameters:** ~10K |
|
|
|
|
|
This is a **base model**, not a chat model. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Data |
|
|
|
|
|
The model was trained on a **synthetic Portuguese dataset** designed to emphasize: |
|
|
|
|
|
- Simple sentence structure |
|
|
- Common verbs and nouns |
|
|
- Basic grammar patterns |
|
|
- Repetition and reinforcement |
|
|
|
|
|
The dataset intentionally avoids: |
|
|
- Instruction-following |
|
|
- Dialog formatting |
|
|
- Reasoning traces |
|
|
|
|
|
This design allows clear observation of **language emergence** in small models. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Procedure |
|
|
|
|
|
- Optimizer: Adam |
|
|
- Learning rate: 3e-4 |
|
|
- Sequence length: 64 |
|
|
- Epochs: 12000 |
|
|
- Loss function: Cross-Entropy Loss |
|
|
- CPU - AMD Ryzen 5 5600G 32GB (0.72 TFLOPS) |
|
|
|
|
|
Training includes checkpointing and continuation support. |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
### Supported Use Cases |
|
|
|
|
|
- Educational experiments |
|
|
- Language modeling research |
|
|
- Studying emergent structure in small neural networks |
|
|
- Baseline comparisons for future MiniText versions |
|
|
|
|
|
### Out-of-Scope Use Cases |
|
|
|
|
|
- Conversational agents |
|
|
- Instruction-following systems |
|
|
- Reasoning or math tasks |
|
|
- Production deployment |
|
|
|
|
|
--- |
|
|
|
|
|
## Example Output |
|
|
|
|
|
Prompt: |
|
|
o gato é |
|
|
|
|
|
Sample generation: |
|
|
o gato é um animal |
|
|
|
|
|
Note: Output quality varies due to the minimal size of the model. |
|
|
|
|
|
--- |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Limited vocabulary and coherence |
|
|
- No reasoning or factual understanding |
|
|
- Susceptible to repetition and noise |
|
|
- Not aligned or safety-tuned |
|
|
|
|
|
These limitations are **expected and intentional**. |
|
|
|
|
|
--- |
|
|
|
|
|
## Ethical Considerations |
|
|
|
|
|
This model does not include safety filtering or alignment mechanisms. |
|
|
It should not be used in applications involving sensitive or high-risk domains. |
|
|
|
|
|
--- |
|
|
|
|
|
## Future Work |
|
|
|
|
|
Planned extensions of the MiniText family include: |
|
|
|
|
|
- MiniText-v1.1-Lang (improved Portuguese fluency) |
|
|
- MiniText-Math (symbolic pattern learning) |
|
|
- MiniText-Chat (conversation fine-tuning) |
|
|
- MiniText-Reasoning (structured token experiments) |
|
|
|
|
|
Each version will remain linked to this base model. |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use MiniText-v1.0 in research or educational material, please cite the project repository. |
|
|
|
|
|
--- |
|
|
|
|
|
## License |
|
|
|
|
|
MIT License |
|
|
|
|
|
|
|
|
Made by: Arthur Samuel(loboGOAT) |
|
|
|
