| ---
|
| license: mit
|
| language:
|
| - bs
|
| - hr
|
| - sr
|
| - en
|
| base_model:
|
| - absltnull/predBor-v0.5
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| ## predBor-v1 bs/sr/cr Small Language Model
|
|
|
| 
|
|
|
| *The new and shiny version, with better coherence and more brain power, in the same size as the preview.*
|
|
|
| ### Overview
|
| If you've seen the [preview](https://huggingface.co/absltnull/predBor-v0.5), you'll know exactly what this is. If not...
|
|
|
| predBor is the **first of its kind:** a small HBS language model **built from the ground up** to natively understand Bosnian, Croatian and Serbian, with English support. Unlike competitors such as `jerteh/gpt2-orao` or `gordicaleksa/YugoGPT`, **predBor is not based on any existing language model** like Mistral or GPT-2; it is its own original base model.
|
|
|
| ### Model Details
|
| - **Architecture:** LLaMA,
|
| - **Context Length:** 4096,
|
| - **Hidden Size:** 1536,
|
| - **Num. Hidden Layers:** 24,
|
| - **Num. Attention Heads:** 16,
|
| - **Intermediate Size:** 4096,
|
| - **Training Optimizer:** Lion,
|
| - **Vocabulary Size:** 65k
|
|
|
| ### Training Data
|
| predBor was trained on approximately **22 billion tokens of diverse, fully uncensored, cleaned, processed and deduplicated HBS and English data**, making `Bor-CORPUS-22B` one of the largest collections of HBS data as of writing this.
|
|
|
| The dataset spans multiple sources and fields, such as songs, news articles, Wikipedia articles, entire books, *manually* scraped blogs, forums, erotica, conversations from multiple social media platforms (DMs, group chats, comment sections, forum discussions, etc.), cleaned CommonCrawl crawl and FineWeb text, and so on.
|
|
|
| Here are some sources from the dataset, as seen in the logs of my "cleaner and packer" script:
|
| | Category | Total Count | Files |
|
| |:---|---:|:---|
|
| | **Web/Crawl** | 28,290,391 | cc-bosnian-huge.txt, dataset_done.txt, CC_2026.txt, fineweb-small.txt |
|
| | **General/Language** | 14,817,407 | english.txt |
|
| | **Data Split** | 3,997,195 | split.txt |
|
| | **News/Blogs** | 1,089,621 | news_done.txt, oldnewspaper.txt, blogger_done.txt, news_out.txt, titlovi_blogs_output.txt |
|
| | **Reference** | 257,932 | wikipedia_done.txt, wikiahh.txt, islamska_pitanja.txt, books_done.txt |
|
| | **Social** | 92,396 | klixforums.txt, reddit.txt, insta.txt, discord.txt, viber.txt |
|
| | **Other** | 32,375 | lrsum.txt, num_output.txt, sex_done.txt, mobi_output.txt, epub_output.txt |
|
| | **Multimedia** | 14,677 | all_subtitles_combined.txt, lyrics.txt |
|
|
|
| ### Model Evaluation
|
| I've tested both versions of predBor against GPT-2 Orao, and the results were rather satisfying in some benchmarks. Take a look for yourself:
|
|
|
| #### Serbian-LLM-Eval by Gordić Aleksa
|
| 
|
|
|
| Out of the 9 Serbian-translated benchmarks, predBor has crushed GPT-2 Orao on 7 of them.
|
|
|
| #### English Eval (Same Benchmarks)
|
| 
|
|
|
| As you can see, since GPT-2 Orao is based on GPT-2 (who would have guessed), it scores slightly higher than predBor in BoolQ, PiQA and WinoGrande, but gets absolutely demolished in ARC Easy, OpenBookQA, NQ Open and TriviaQA.
|
|
|
| ### What's next?
|
| **More data pretraining + instruct tuning.**
|
|
|
| **predBor-v2 will be a continued pretrain of this checkpoint**, trained on a massive uncensored, cleaned and language-labeled dataset of HBS and English data.
|
|
|
| Then, we get to instruct tuning. That's how we get the final version: **Bor**.
|
|
|
| **Pretraining Dataset Focus:** Human-written, clean, broad, uncensored text data.
|
| **Instruct Dataset Focus:** Synthetic instruct-following data, trained only on one persona, no refusals baked in.
|
|
|
| Stay tuned. absltnull out. |
