| --- |
| tags: |
| - text-generation-inference |
| - transformers |
| - unsloth |
| - olmo2 |
| license: apache-2.0 |
| language: |
| - en |
| datasets: |
| - Pinkstack/roblox-luau-corpus-text |
| - Roblox/luau_corpus |
| - boatbomber/roblox-info-dump |
| - wikimedia/wikipedia |
| pipeline_tag: text-generation |
| base_model: |
| - allenai/OLMo-2-0425-1B |
| --- |
| |
|  |
|
|
| # print("Before we start") |
| We are not related to Roblox in any way, any mention of Roblox is purely to help people understand what the model is about. |
| As per the [Roblox website](https://create.roblox.com/docs/assistant/guide), they use Meta's Llama 3 (we assume 70B) for their AI assistant. This model, while powerful, cannot come close to the performance of a 70B model. |
|
|
| # print("Stages of pre-training") |
|
|
| This model was continually pre-trained in 3 stages. |
|
|
| - Stage 1: Pre-training on the Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus on 4096 tokens (the maximum olmo 2 can usually reach) |
| |
| - Stage 2: Pre-training on the boatbomber/roblox-info-dump with rope scaling set to 4, so stage 2 was for expanding the context of the model to **16384**. |
| |
| |
| !stage 3 and onwards were with added layers. the model started with 16 layers, then we merged another 20 to make the model bigger and deeper! |
| - Stage 3: Training on a mix of Pinkstack/roblox-luau-corpus-text & Roblox/luau_corpus + wikimedia/wikipedia with rope scaling set to 8, aka **32768** tokens of context. We mixed the wikimedia/wikipedia to hopefully improve the general text and knowledge of the model. |
|
|
| In total, the model was continually pre-trained on up to 1.3B tokens, final loss of **1.916400**. |
|
|
| # print("Use cases") |
| As this is a base model, there isn't much to do with it currently. But, you can fine-tune it on your own datasets to turn it into an instruct - chat type model. |
|
|
| # print("Notice") |
|
|
| This stage-3 base model did not undergo saftey alignment by us, thus it can generate unethical content. Any outputs generated by the LLM are your responsibility. |
|
|
| # print("Additional information") |
| This repo contains the stage 3 pre-trained/base model. |
|
|
| unsloth was used for training (https://unsloth.ai/) |
