noterat/forseti-31b-it

Forseti 31B IT

Forseti 31B IT is the instruction-tuned variant of Forseti 31B, a Swedish government domain language model based on Gemma 4 31B.

Model Description

Forseti 31B IT was fine-tuned on 2,000 synthetic Swedish legal instruction examples generated from the training corpus. The examples cover legal reasoning, regulatory interpretation, and citizen-facing government Q&A in Swedish.

Evaluation (EuroEval Swedish, canonical Modal A100)

Benchmark	CPT	IT	Delta
swerec (MCC)	78.4	79.8	+1.4
suc3 (micro-F1)	77.2	77.6	+0.4
scala-sv (MCC)	74.3	74.4	+0.1
multi-wiki-qa-sv (F1)	78.5	77.2	-1.3
swedn (chrF3++)	33.7	33.8	+0.1
mmlu-sv (MCC)	72.8	72.4	-0.4
hellaswag-sv (acc)	66.5	71.4	+4.9
Average	68.8	69.5	+0.7

Training Details

Parameter	Value
Base model	Forseti 31B (CPT of Gemma 4 31B)
IT examples	2,000 synthetic Swedish legal
Method	LoRA (r=16, alpha=32), merged back to full weights
Epochs	5
Hardware	LUMI supercomputer (8 AMD MI250X GCDs)
Framework	Axolotl + PyTorch FSDP2

Key Finding

Instruction tuning primarily improves common-sense reasoning (hellaswag +4.9 pp) while maintaining domain knowledge. The overall EuroEval average improves by +0.7 pp over the CPT base.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("noterat/forseti-31b-it", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("noterat/forseti-31b-it")

prompt = "Forklara skillnaden mellan forvaltningslagen och kommunallagen."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

GGUF Quantizations

For local deployment, see Forseti 31B IT GGUF (Q4_K_M and Q8_0).

License

Apache 2.0 (inherited from Gemma 4).

Citation

@misc{forseti2026,
  title={Forseti: A Swedish Government Domain Language Model},
  author={Martini, Alexandro},
  year={2026},
  url={https://huggingface.co/noterat/forseti-31b}
}