README.md

---
license: cc-by-sa-4.0
language:
- pl
- en
library_name: pytorch
base_model: SlayerLab/GoLLeM-v6-250M
tags:
- polish
- english
- language-model
- chat
- research
- gollem
pipeline_tag: text-generation
---

# GoLLeM v6 250M Instruct v2 (Polish–English research chat model)

> **STATUS: RESEARCH PREVIEW.** A small chat model fine-tuned from the base model
> [`SlayerLab/GoLLeM-v6-250M`](https://huggingface.co/SlayerLab/GoLLeM-v6-250M). It can greet, introduce itself and
> answer simple questions, but it makes mistakes that are listed with numbers under *Limitations*. Results use **our
> protocol**, a single run and a single seed.

GoLLeM v6 250M Instruct v2 is the base GoLLeM v6 250M model after supervised fine-tuning (SFT) on about 9,300 Polish and
English conversations. It is a Fabryka AI project (formerly SlayerLab).

**Difference from [Instruct v1](https://huggingface.co/SlayerLab/GoLLeM-v6-250M-Instruct-v1):** the same data and
training recipe, except the answers about its origin. Asked who created it, Instruct v2 says it is a project of Fabryka
AI; **the model does not use its author's name** (the author is named in this card). Instruct v1 sometimes introduced
itself with the author's name in a conversation (51 of 160); Instruct v2 did not in our tests (0 of 160, 0 of 680).

**Author:** Arkadiusz Słota (Fabryka AI).

## What it is for (and what it is not)

**Intended use:** research on small bilingual chat models; short conversations in Polish and English; answering
questions about a text you paste into the conversation.

**Not intended for:** production use, factual questions without a source text, arithmetic, or anything where a wrong
answer matters. The model has no internet access and does not remember earlier conversations.

## Results

Checkpoint `1018f1e8…` (see *Training*). All numbers are from our own evaluation sets, fixed before training. Temperature
0.7, top-p 0.9 (the defaults of `chat_gollem_v6.py`).

| what | result | notes |
|---|---|---|
| Identity (name, organisation, creator), PL / EN | **0.66 / 0.58** | mean over 10 samples per question, 20 questions per language; a creator answer counts as correct when it names Fabryka AI and no person |
| Greetings and small talk, PL / EN | **0.79 / 0.89** | mean over 7 samples per question, 20 questions per language |
| Answers from a given text (Polish, PoQuAD, 217 answerable questions) | token F1 **0.232** | base model few-shot: 0.067 |
| Declines when the text has no answer (43 questions) | 6 / 43 | wrongly declines an answerable question: 12 / 217 |
| Arithmetic word problems (206) | 0 / 206 | the model does not do arithmetic |
| Finishes its answer within the length limit | 485 / 506 (96 %) | |
| Says it is an OpenAI / GPT model (200 samples) | 0 / 200 | „AI language model”: 0 / 200 |
| Introduces itself with the author's name | **0 / 680** single questions, **0 / 160** two-turn conversations | Instruct v1: 3 / 680 and 51 / 160 |

Compared with Instruct v1 on the same identity measure: +0.175 (95 % CI +0.068 … +0.295); small talk −0.004 (95 % CI
−0.032 … +0.025), i.e. not worse. The identity numbers in the Instruct v1 card use a different measure (there the author's name
was the expected answer), so do not compare the two tables directly.

## Training

| | |
|---|---|
| Base model | `SlayerLab/GoLLeM-v6-250M`, final checkpoint (step 760,000); same architecture and tokenizer |
| Method | full fine-tuning (no LoRA), loss on assistant turns only |
| Format | ChatML: `<|im_start|>{role}\n{content}<|im_end|>\n`, no system prompt, generation stops at `<|im_end|>` |
| Steps | 2 epochs, 156 steps, 32 packed sequences of 1,024 tokens per step |
| Tokens | 2.54 M per epoch, of which 1.63 M are assistant tokens (with loss) |
| Optimizer | Muon + AdamW as in pretraining, learning rate 0.2 × pretraining (peak 1.2e-4, Muon 4e-3), warmup 5 steps, cosine to 10 %, weight decay 0.1, gradient clip 1.0, seed 1337 |
| Validation loss | 2.026 → 1.884 |
| Hardware / time | 1 GPU, about 7.5 minutes |

## Training data

9,324 conversations (Polish 4,149, English 5,175): 8,836 used for training and 171 for validation; 317 conversations
longer than 1,024 tokens were removed (311 + 6).

| source | conversations | licence |
|---|---:|---|
| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2) @ `179dd21` | 4,601 | Apache-2.0 |
| [clarin-pl/poquad](https://huggingface.co/datasets/clarin-pl/poquad) @ `a60f228` | 2,129 | **CC BY 4.0** |
| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) @ `f9ea045` | 1,214 | Apache-2.0 |
| synthetic, generated locally with Muse-Glimmer-30B (apache-2.0) | 1,380 | apache-2.0 |

- **PoQuAD attribution:** PoQuAD (clarin-pl/poquad), CC BY 4.0. Modified: converted to chat format; a share of
  unanswerable questions answered with one of five fixed refusal sentences.
- **Synthetic part:** the questions were written by the generator model; the identity answers come from our own identity
  card, not from the generator. Math prompts: generated briefs; GSM8K (MIT) used only as few-shot format examples for the
  generator.
- Rows with self-descriptions of other AI systems were removed before training.

## Usage

```python
# pip install torch safetensors tokenizers huggingface_hub
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("SlayerLab/GoLLeM-v6-250M-Instruct-v2",
                         revision="085b7b5ed11465536b4fb44d6c7592b9fb553b91")  # the reviewed code (model + chat helper)
sys.path.insert(0, path)
from modeling_gollem_v6 import load_gollem_v6
from chat_gollem_v6 import chat

model, tok = load_gollem_v6(path)
print(chat(model, tok, [("user", "Cześć! Kim jesteś?")], seed=1))
print(chat(model, tok, [("user", "Tekst: Kraków leży nad Wisłą.\nPytanie: Nad jaką rzeką leży Kraków?")], seed=1))
```

`chat()` uses the format and sampling of our evaluation (temperature 0.7, top-p 0.9). Text typed by a user such as
„<|im_end|>” is encoded as plain text, not as a control token.

## Limitations

- **Small model.** Limited knowledge; may answer fluently and wrongly. No arithmetic. Context: 1,024 tokens.
- **Does not know its author's name.** Asked who created it, the model says it is a project of Fabryka AI. The author,
  Arkadiusz Słota, is named in this card, not in the model. The statements of the model are not statements of its
  author.
- **Copies names from the conversation.** If a user writes a name, the model may adopt it as its own (when the user
  supplies the author's name: Polish 3/40, English 5/40; Instruct v1: 9/40 and 37/40).
- **Self-description.** In our probe it never described itself as an OpenAI model (0/200), but with a forced prefix
  („…created by”) it still assigns probability ≈ 0.24 to „OpenAI”. The association comes from model-generated chat
  data in pretraining. GoLLeM is **not** affiliated with OpenAI.
- **Multi-turn weaknesses:** may repeat its previous answer in a later turn, may greet the user with the company name
  („Cześć, Fabryku!”) when no name was given, and answers „What can you do?” with its identity template.
- **Long answers can loop.** A repetition penalty (about 1.1–1.2) may reduce this (not verified); it is not used in our evaluation.

## License

**Weights: CC BY-SA 4.0** (inherited from the base model). Attribution: GoLLeM v6 250M Instruct v2, Arkadiusz Słota /
Fabryka AI, link to this repository; derivative weights under the same licence. Fine-tuning data keep their licences;
see *Training data* (PoQuAD: CC BY 4.0, attribution above).

## Po polsku (skrót)

GoLLeM v6 250M Instruct v2 to mały model do rozmowy po polsku i angielsku, dostrojony (SFT) z bazowego GoLLeM v6 250M
na ok. 9,3 tys. rozmów. Wersja badawcza (research preview): wita się, przedstawia i odpowiada na pytania do podanego
tekstu, ale się myli, nie liczy i nie ma dostępu do internetu. Od Instruct v1 różni się tym, że nie używa nazwiska
autora: na pytanie o twórcę odpowiada, że jest projektem Fabryki AI (wcześniej SlayerLab). Autora podaje ta karta.

---
*GoLLeM v6 250M Instruct v2 — Fabryka AI. **Author: Arkadiusz Słota.** Research preview.*