cs-552-2026-momy/general_knowledge_model

General Knowledge Model — CS-552 (MOMY)

A reasoning-focused model for general-knowledge question answering, post-trained from Qwen/Qwen3-1.7B using GRPO (Group Relative Policy Optimization) with verifiable rewards (RLVR). Developed for the EPFL CS-552 Modern NLP course project (Spring 2026).

The model reasons step-by-step inside <think>...</think> tags before producing a final answer enclosed in a \boxed{} environment, supporting automated answer extraction and verification.

Model Details

• Base model: Qwen3-1.7B
• Training method: GRPO (RLVR) with LoRA (r=16, α=32) adapters merged into base weights
• Domain: General knowledge — science, history, geography, world affairs
• Output format: <think> reasoning chain followed by \boxed{ANSWER}
• Team: MOMY

Intended Use

The model answers both multiple-choice and short open-ended factual questions:

• Multiple-choice: outputs the correct option letter, e.g. \boxed{B}
• Open-ended: outputs a short factual answer, e.g. \boxed{Paris}

Training Data

Source	Type	Share
MMLU-Pro	Graduate-level multiple-choice (up to 10 options)	~7k
TriviaQA	Open-domain factual QA	~2k

A total of 8,100 training and 900 validation examples (90/10 split), each formatted with a chain-of-thought target terminating in Therefore, the final answer is \boxed{·}.

Training Configuration

Hyperparameter	Value
Method	GRPO (RLVR)
Learning rate	1e-4
Effective batch size	16
Rollouts per prompt	4
Training steps	300
Temperature (rollout)	0.9
KL coefficient (β)	0.04
LoRA rank / alpha	16 / 32
Hardware	1× NVIDIA A100-40G

Reward function: +1.0 for a correct boxed answer, +0.1 for a valid <think> block, -0.1 for a missing/malformed box, -0.1 for exceeding the token budget.

Evaluation

Benchmark	pass@1
Course CI (knowledge)	0.44
Qwen3-1.7B base (CI)	0.25
Local held-out (n=900)	0.373 (pass@8: 0.492)

GRPO substantially improves over the base model on the held-out CI knowledge benchmark.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "cs-552-2026-momy/general_knowledge_model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "What is the capital of Australia?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The chat template hardcodes enable_thinking=true, so the model always reasons before answering. A generation_config.json with the recommended inference parameters is included in the repo.

Limitations

• As a 1.7B-parameter model, factual coverage is limited; performance on long-tail or highly specialized knowledge is unreliable.
• The dominant failure mode is option-matching miscalibration: the model may derive a correct value in its reasoning but select a mismatched option when the computed answer is not listed among the choices.
• Knowledge is skewed toward English-language and Western-centric sources (MMLU-Pro, TriviaQA).

Citation

If referencing this model, please cite the underlying methods:

• GRPO: Shao et al., DeepSeekMath (2024)
• MMLU-Pro: Wang et al. (2024)
• TriviaQA: Joshi et al. (2017)
• Base model: Qwen3-1.7B (Qwen Team, 2025)