MiniText Family 🧠
Collection
A family of 10k-10M params LM's (This family will grow up with time)
•
2 items
•
Updated
MiniText-v1.0
Model Summary
MiniText-v1.0 is a tiny character-level language model trained from scratch to learn basic Portuguese text patterns.
The goal of this project is to explore the minimum viable neural architecture capable of producing structured natural language, without pretraining, instruction tuning, or external corpora.
This model is intended for research, education, and experimentation.
Model Details
- Architecture: Custom MiniText (character-level)
- Training Objective: Next-character prediction
- Vocabulary: Byte-level (0–255)
- Language: Portuguese (basic)
- Initialization: Random (no pretrained weights)
- Training: Single-stream autoregressive training
- Parameters: ~10K
This is a base model, not a chat model.
Training Data
The model was trained on a synthetic Portuguese dataset designed to emphasize:
- Simple sentence structure
- Common verbs and nouns
- Basic grammar patterns
- Repetition and reinforcement
The dataset intentionally avoids:
- Instruction-following
- Dialog formatting
- Reasoning traces
This design allows clear observation of language emergence in small models.
Training Procedure
- Optimizer: Adam
- Learning rate: 3e-4
- Sequence length: 64
- Epochs: 12000
- Loss function: Cross-Entropy Loss
- CPU - AMD Ryzen 5 5600G 32GB (0.72 TFLOPS)
Training includes checkpointing and continuation support.
Intended Use
Supported Use Cases
- Educational experiments
- Language modeling research
- Studying emergent structure in small neural networks
- Baseline comparisons for future MiniText versions
Out-of-Scope Use Cases
- Conversational agents
- Instruction-following systems
- Reasoning or math tasks
- Production deployment
Example Output
Prompt: o gato é
Sample generation: o gato é um animal
Note: Output quality varies due to the minimal size of the model.
Limitations
- Limited vocabulary and coherence
- No reasoning or factual understanding
- Susceptible to repetition and noise
- Not aligned or safety-tuned
These limitations are expected and intentional.
Ethical Considerations
This model does not include safety filtering or alignment mechanisms. It should not be used in applications involving sensitive or high-risk domains.
Future Work
Planned extensions of the MiniText family include:
- MiniText-v1.1-Lang (improved Portuguese fluency)
- MiniText-Math (symbolic pattern learning)
- MiniText-Chat (conversation fine-tuning)
- MiniText-Reasoning (structured token experiments)
Each version will remain linked to this base model.
Citation
If you use MiniText-v1.0 in research or educational material, please cite the project repository.
License
MIT License
Made by: Arthur Samuel(loboGOAT)