| --- |
| language: |
| - mt |
| base_model: utter-project/EuroLLM-1.7B |
| pipeline_tag: text-generation |
| tags: |
| - maltese |
| - malti |
| - continued-pretraining |
| datasets: |
| - MLRS/korpus_malti |
| - saillab/alpaca-maltese-cleaned |
| - CohereForAI/aya_collection |
| --- |
| |
| # Fredu 1.7B Instruct |
|
|
| A Maltese-specialised model built from `utter-project/EuroLLM-1.7B` in two stages |
| on a single consumer GPU (RTX 5060 Ti, 16 GB). |
|
|
| It is fluent in Maltese but hallucinates and does not always reply with anything related to the prompt. |
| It's able to translate english sentences into a related sentence in Maltese. It's quite bad at it but the Maltese translation is related somehow. |
|
|
| ## How to run |
|
|
| ### Ollama |
|
|
| ```sh |
| ollama run hf.co/Cabbache/Fredu-1.7B-Instruct |
| ``` |
|
|
| The prompt template and sampling parameters are stored in the repo, so nothing |
| else is needed. Pick a build with a tag: |
|
|
| | tag | size | notes | |
| |---|---|---| |
| | *(none)* / `:Q4_K_M` | 1.0 GB | 4-bit, the default. | |
| | `:Q8_0` | 1.8 GB | 8-bit, close to full quality | |
| | `:F16` | 3.3 GB | no quantization | |
|
|
|
|
| ### transformers |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("Cabbache/Fredu-1.7B-Instruct") |
| model = AutoModelForCausalLM.from_pretrained("Cabbache/Fredu-1.7B-Instruct", |
| dtype="bfloat16", device_map="auto") |
| |
| q = "X'inhu l-Kunsill Lokali f'Malta?" |
| ids = tok(f"Mistoqsija: {q}\nTweġiba:", return_tensors="pt").to(model.device) |
| out = model.generate(**ids, max_new_tokens=200, temperature=0.3, do_sample=True) |
| print(tok.decode(out[0], skip_special_tokens=True)) |
| ``` |
|
|
| There is no chat template. Use the `Mistoqsija:/Tweġiba:` framing above, which |
| is what the model was trained on. A low temperature helps: what factual |
| knowledge it has is held with very little margin and does not survive sampling |
| at 0.7. |
|
|
| ## What was done |
|
|
| | stage | data | result | |
| |---|---|---| |
| | Continued pretraining | 336M tokens, 125,900 documents from Korpus Malti | perplexity 8.29 → 5.98 | |
| | Supervised fine-tuning | 43,988 Maltese question/answer pairs | answer-loss 1.70 → 1.40 | |
|
|
| Both stages were full fine-tunes (all 1.657B parameters), bf16 weights and |
| gradients with 8-bit AdamW and gradient checkpointing, ~14 GB VRAM. |
|
|
| ## Prompt format |
|
|
| The model was trained on this exact framing, with loss computed **only** on the |
| answer: |
|
|
| ``` |
| Mistoqsija: {question} |
| Tweġiba: {answer} |
| ``` |
|
|
| ## It reads informal Maltese and replies in correct Maltese |
|
|
| 35% of training questions had their diacritics stripped (`ħ→h, ġ→g, ċ→c, ż→z`, |
| so `għ→gh`) while every answer was left correct. Because loss lands only on the |
| answer, the model is shown informal spelling and never rewarded for producing it. |
|
|
| ``` |
| Q: X'inhu n-numru medju ta' sighat ta' rqad li persuna ghandha tiehu? |
| A: In-numru medju ta' sigħat ta' rqad li persuna għandha tieħu huwa madwar |
| 7.5 sigħat kuljum. |
| ``` |
|
|
| Note `sighat → sigħat`, `ghandha → għandha`, `tiehu → tieħu`. |
|
|
| ## Evaluation |
|
|
| Perplexity on held-out documents, by register, against the untrained base: |
|
|
| | source | base | this model | change | |
| |---|---|---|---| |
| | government gazette | 7.84 | 2.97 | −62.1% | |
| | parliament | 6.89 | 3.19 | −53.6% | |
| | press | 7.72 | 6.01 | −22.1% | |
| | academic | 12.61 | 10.38 | −17.7% | |
| | blogs | 10.43 | 8.93 | −14.4% | |
| | wikipedia | 6.51 | 6.06 | −6.9% | |
|
|
| A 57-item Malta factual eval (greedy decoding, an upper bound on knowledge): |
|
|
| | | this model | base | |
| |---|---|---| |
| | overall | **27/57** | 22/57 | |
| | history | **12/13** | 8/13 | |
| | geography | 4/17 | 6/17 | |
|
|
| ## Licence |
|
|
| Do whatever you want with it |
