---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B
tags:
- qwen3
- grpo
- rlvr
- lora
- general-knowledge
- reasoning
- cs-552
language:
- en
pipeline_tag: text-generation
library_name: transformers
---

# General Knowledge Model — CS-552 (MOMY)

A reasoning-focused model for **general-knowledge question answering**, post-trained from
[`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) using **GRPO** (Group Relative
Policy Optimization) with verifiable rewards (RLVR). Developed for the EPFL CS-552 *Modern NLP*
course project (Spring 2026).

The model reasons step-by-step inside `<think>...</think>` tags before producing a final answer
enclosed in a `\boxed{}` environment, supporting automated answer extraction and verification.

## Model Details

- **Base model:** Qwen3-1.7B
- **Training method:** GRPO (RLVR) with LoRA (r=16, α=32) adapters merged into base weights
- **Domain:** General knowledge — science, history, geography, world affairs
- **Output format:** `<think>` reasoning chain followed by `\boxed{ANSWER}`
- **Team:** MOMY

## Intended Use

The model answers both multiple-choice and short open-ended factual questions:

- **Multiple-choice:** outputs the correct option letter, e.g. `\boxed{B}`
- **Open-ended:** outputs a short factual answer, e.g. `\boxed{Paris}`

## Training Data

| Source | Type | Share |
|---|---|---|
| [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) | Graduate-level multiple-choice (up to 10 options) | ~7k |
| [TriviaQA](https://huggingface.co/datasets/mandarjoshi/trivia_qa) | Open-domain factual QA | ~2k |

A total of 8,100 training and 900 validation examples (90/10 split), each formatted with a
chain-of-thought target terminating in `Therefore, the final answer is \boxed{·}`.

## Training Configuration

| Hyperparameter | Value |
|---|---|
| Method | GRPO (RLVR) |
| Learning rate | 1e-4 |
| Effective batch size | 16 |
| Rollouts per prompt | 4 |
| Training steps | 300 |
| Temperature (rollout) | 0.9 |
| KL coefficient (β) | 0.04 |
| LoRA rank / alpha | 16 / 32 |
| Hardware | 1× NVIDIA A100-40G |

**Reward function:** `+1.0` for a correct boxed answer, `+0.1` for a valid `<think>` block,
`-0.1` for a missing/malformed box, `-0.1` for exceeding the token budget.

## Evaluation

| Benchmark | pass@1 |
|---|---|
| Course CI (knowledge) | **0.44** |
| Qwen3-1.7B base (CI) | 0.25 |
| Local held-out (n=900) | 0.373 (pass@8: 0.492) |

GRPO substantially improves over the base model on the held-out CI knowledge benchmark.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "cs-552-2026-momy/general_knowledge_model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [{"role": "user", "content": "What is the capital of Australia?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

The chat template hardcodes `enable_thinking=true`, so the model always reasons before answering.
A `generation_config.json` with the recommended inference parameters is included in the repo.

## Limitations

- As a 1.7B-parameter model, factual coverage is limited; performance on long-tail or highly
  specialized knowledge is unreliable.
- The dominant failure mode is **option-matching miscalibration**: the model may derive a correct
  value in its reasoning but select a mismatched option when the computed answer is not listed
  among the choices.
- Knowledge is skewed toward English-language and Western-centric sources (MMLU-Pro, TriviaQA).

## Citation

If referencing this model, please cite the underlying methods:

- **GRPO:** Shao et al., *DeepSeekMath* (2024)
- **MMLU-Pro:** Wang et al. (2024)
- **TriviaQA:** Joshi et al. (2017)
- **Base model:** Qwen3-1.7B (Qwen Team, 2025)