| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-1.7B |
| tags: |
| - qwen3 |
| - grpo |
| - rlvr |
| - lora |
| - general-knowledge |
| - reasoning |
| - cs-552 |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: transformers |
| --- |
| |
| # General Knowledge Model — CS-552 (MOMY) |
|
|
| A reasoning-focused model for **general-knowledge question answering**, post-trained from |
| [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) using **GRPO** (Group Relative |
| Policy Optimization) with verifiable rewards (RLVR). Developed for the EPFL CS-552 *Modern NLP* |
| course project (Spring 2026). |
|
|
| The model reasons step-by-step inside `<think>...</think>` tags before producing a final answer |
| enclosed in a `\boxed{}` environment, supporting automated answer extraction and verification. |
|
|
| ## Model Details |
|
|
| - **Base model:** Qwen3-1.7B |
| - **Training method:** GRPO (RLVR) with LoRA (r=16, α=32) adapters merged into base weights |
| - **Domain:** General knowledge — science, history, geography, world affairs |
| - **Output format:** `<think>` reasoning chain followed by `\boxed{ANSWER}` |
| - **Team:** MOMY |
|
|
| ## Intended Use |
|
|
| The model answers both multiple-choice and short open-ended factual questions: |
|
|
| - **Multiple-choice:** outputs the correct option letter, e.g. `\boxed{B}` |
| - **Open-ended:** outputs a short factual answer, e.g. `\boxed{Paris}` |
|
|
| ## Training Data |
|
|
| | Source | Type | Share | |
| |---|---|---| |
| | [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) | Graduate-level multiple-choice (up to 10 options) | ~7k | |
| | [TriviaQA](https://huggingface.co/datasets/mandarjoshi/trivia_qa) | Open-domain factual QA | ~2k | |
|
|
| A total of 8,100 training and 900 validation examples (90/10 split), each formatted with a |
| chain-of-thought target terminating in `Therefore, the final answer is \boxed{·}`. |
|
|
| ## Training Configuration |
|
|
| | Hyperparameter | Value | |
| |---|---| |
| | Method | GRPO (RLVR) | |
| | Learning rate | 1e-4 | |
| | Effective batch size | 16 | |
| | Rollouts per prompt | 4 | |
| | Training steps | 300 | |
| | Temperature (rollout) | 0.9 | |
| | KL coefficient (β) | 0.04 | |
| | LoRA rank / alpha | 16 / 32 | |
| | Hardware | 1× NVIDIA A100-40G | |
|
|
| **Reward function:** `+1.0` for a correct boxed answer, `+0.1` for a valid `<think>` block, |
| `-0.1` for a missing/malformed box, `-0.1` for exceeding the token budget. |
|
|
| ## Evaluation |
|
|
| | Benchmark | pass@1 | |
| |---|---| |
| | Course CI (knowledge) | **0.44** | |
| | Qwen3-1.7B base (CI) | 0.25 | |
| | Local held-out (n=900) | 0.373 (pass@8: 0.492) | |
|
|
| GRPO substantially improves over the base model on the held-out CI knowledge benchmark. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "cs-552-2026-momy/general_knowledge_model" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto") |
| |
| messages = [{"role": "user", "content": "What is the capital of Australia?"}] |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20) |
| print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| The chat template hardcodes `enable_thinking=true`, so the model always reasons before answering. |
| A `generation_config.json` with the recommended inference parameters is included in the repo. |
|
|
| ## Limitations |
|
|
| - As a 1.7B-parameter model, factual coverage is limited; performance on long-tail or highly |
| specialized knowledge is unreliable. |
| - The dominant failure mode is **option-matching miscalibration**: the model may derive a correct |
| value in its reasoning but select a mismatched option when the computed answer is not listed |
| among the choices. |
| - Knowledge is skewed toward English-language and Western-centric sources (MMLU-Pro, TriviaQA). |
|
|
| ## Citation |
|
|
| If referencing this model, please cite the underlying methods: |
|
|
| - **GRPO:** Shao et al., *DeepSeekMath* (2024) |
| - **MMLU-Pro:** Wang et al. (2024) |
| - **TriviaQA:** Joshi et al. (2017) |
| - **Base model:** Qwen3-1.7B (Qwen Team, 2025) |
