README.md

---
language:
- mt
base_model: utter-project/EuroLLM-1.7B
pipeline_tag: text-generation
tags:
- maltese
- malti
- continued-pretraining
datasets:
- MLRS/korpus_malti
- saillab/alpaca-maltese-cleaned
- CohereForAI/aya_collection
---

# Fredu 1.7B Instruct

A Maltese-specialised model built from `utter-project/EuroLLM-1.7B` in two stages
on a single consumer GPU (RTX 5060 Ti, 16 GB).

It is fluent in Maltese but hallucinates and does not always reply with anything related to the prompt.
It's able to translate english sentences into a related sentence in Maltese. It's quite bad at it but the Maltese translation is related somehow.

## How to run

### Ollama

```sh
ollama run hf.co/Cabbache/Fredu-1.7B-Instruct
```

The prompt template and sampling parameters are stored in the repo, so nothing
else is needed. Pick a build with a tag:

| tag | size | notes |
|---|---|---|
| *(none)* / `:Q4_K_M` | 1.0 GB | 4-bit, the default. |
| `:Q8_0` | 1.8 GB | 8-bit, close to full quality |
| `:F16` | 3.3 GB | no quantization |


### transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Cabbache/Fredu-1.7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Cabbache/Fredu-1.7B-Instruct",
                                             dtype="bfloat16", device_map="auto")

q = "X'inhu l-Kunsill Lokali f'Malta?"
ids = tok(f"Mistoqsija: {q}\nTweġiba:", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=200, temperature=0.3, do_sample=True)
print(tok.decode(out[0], skip_special_tokens=True))
```

There is no chat template. Use the `Mistoqsija:/Tweġiba:` framing above, which
is what the model was trained on. A low temperature helps: what factual
knowledge it has is held with very little margin and does not survive sampling
at 0.7.

## What was done

| stage | data | result |
|---|---|---|
| Continued pretraining | 336M tokens, 125,900 documents from Korpus Malti | perplexity 8.29 → 5.98 |
| Supervised fine-tuning | 43,988 Maltese question/answer pairs | answer-loss 1.70 → 1.40 |

Both stages were full fine-tunes (all 1.657B parameters), bf16 weights and
gradients with 8-bit AdamW and gradient checkpointing, ~14 GB VRAM.

## Prompt format

The model was trained on this exact framing, with loss computed **only** on the
answer:

```
Mistoqsija: {question}
Tweġiba: {answer}
```

## It reads informal Maltese and replies in correct Maltese

35% of training questions had their diacritics stripped (`ħ→h, ġ→g, ċ→c, ż→z`,
so `għ→gh`) while every answer was left correct. Because loss lands only on the
answer, the model is shown informal spelling and never rewarded for producing it.

```
Q: X'inhu n-numru medju ta' sighat ta' rqad li persuna ghandha tiehu?
A: In-numru medju ta' sigħat ta' rqad li persuna għandha tieħu huwa madwar
   7.5 sigħat kuljum.
```

Note `sighat → sigħat`, `ghandha → għandha`, `tiehu → tieħu`.

## Evaluation

Perplexity on held-out documents, by register, against the untrained base:

| source | base | this model | change |
|---|---|---|---|
| government gazette | 7.84 | 2.97 | −62.1% |
| parliament | 6.89 | 3.19 | −53.6% |
| press | 7.72 | 6.01 | −22.1% |
| academic | 12.61 | 10.38 | −17.7% |
| blogs | 10.43 | 8.93 | −14.4% |
| wikipedia | 6.51 | 6.06 | −6.9% |

A 57-item Malta factual eval (greedy decoding, an upper bound on knowledge):

| | this model | base |
|---|---|---|
| overall | **27/57** | 22/57 |
| history | **12/13** | 8/13 |
| geography | 4/17 | 6/17 |

## Licence

Do whatever you want with it