| | --- |
| | language: |
| | - en |
| | tags: |
| | - text generation |
| | - pytorch |
| | - causal-lm |
| | license: mit |
| | datasets: |
| | - allenai/c4 |
| | - HuggingFaceFW/fineweb-edu |
| | - togethercomputer/RedPajama-Data-V2 |
| | - Muennighoff/natural-instructions |
| | - databricks/databricks-dolly-15k |
| | - HuggingFaceTB/smollm-corpus |
| | - open-phi/textbooks |
| | - roneneldan/TinyStories |
| | --- |
| | |
| | # Mixtress 135M |
| |
|
| | ## Model Description |
| |
|
| | Mixtress 135M is a transformer model based upon the [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral) architecture. It is the culmination of approximately 20 weeks of [Kaggle](https://kaggle.com) free hours, and 67 twelve-hour training runs. |
| |
|
| | ## Training data |
| |
|
| | Mixtress was trained on a curated sampling of data from the following datasets: |
| |
|
| | - allenai/c4 |
| | - HuggingFaceFW/fineweb-edu |
| | - togethercomputer/RedPajama-Data-V2 |
| | - Muennighoff/natural-instructions |
| | - databricks/databricks-dolly-15k |
| | - HuggingFaceTB/smollm-corpus |
| | - open-phi/textbooks |
| | - roneneldan/TinyStories |
| |
|
| | ## Training procedure |
| |
|
| | This model was trained for 2.15 billion tokens over 20,000 optimizer steps. It was trained as a masked autoregressive language model, using cross-entropy loss. |
| |
|
| | The final train loss was 1.941, validation loss was 2.206, and perplexity was 9.136. |
| |
|
| | Mixtress was pre-trained and fine-tuned simultaneously. Full reproduction code may be found [at this URL](https://www.kaggle.com/code/luciferianink/pretraining-a-mixtral), or in the Jupyter notebook [in this repository](./pretraining-a-mixtral.ipynb). |
| |
|
| | ## Intended Use and Limitations |
| |
|
| | The model is best at what it was pretrained for, which is generating conversational text and answering questions from a prompt. |
| |
|
| | ### How to use |
| |
|
| | You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run: |
| |
|
| | ```py |
| | >>> from transformers import pipeline |
| | >>> generator = pipeline('text-generation', model='UNSAFE/Mixtress-135M') |
| | >>> generator("In a shocking finding, ", do_sample=True, temperature=0.7, min_length=50) |
| | |
| | [{'generated_text': 'In a shocking finding, 20 years ago, U.S. President Donald Trump'}] |
| | ``` |
| |
|
| | ## Eval results |
| |
|
| | All evaluations were done using the [Pythia evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness). |
| |
|
| | ### Scores |
| |
|
| | | Model and Size | ARC-easy | ARC-challenge | HellaSwag | PiQA | TinyMMLU | TriviaQA | Winogrande | |
| | | ------------------------- | ---------- | ------------- | ---------- | ---------- | ---------- | -------- | ---------- | |
| | | EleutherAI/gpt-neo-125m | 22.95% | N/A | 30.26% | N/A | N/A | N/A | N/A | |
| | | HuggingFaceTB/SmolLM-135M | 43.99% | N/A | 42.30% | 69.60% | 30.23% | 4.11% | 52.70% | |
| | | OpenAI/GPT2-137M | 31.09% | N/A | 29.76% | 62.51% | 26.29% | 0.49% | 49.72% | |
| | | **UNSAFE/Mixtress-135M** | **29.21%** | **24.57%** | **26.99%** | **52.67%** | **31.71%** | **N/A** | **50.91%** | |
| |
|
| | ## Join Us |
| |
|
| | If you would like to chat with us, please join the [Discord](https://discord.gg/8ZmHP8CqUX) server! |
| |
|
