| # ModernBERT Reward Model (CoT SQL/NL Alignment) | |
| Finetuned `answerdotai/ModernBERT-base` to score how well a generated natural-language description (NL) and chain-of-thought reasoning align with a SQL query. The model is trained as a regression head (sigmoid output in `[0, 1]`) to predict `similarity_with_penalty` scores derived from human preference data plus corruption heuristics. | |
| ## Usage | |
| ```python | |
| import torch | |
| from transformers import AutoTokenizer | |
| from modeling_reward import BERTRewardModel | |
| model_name = "DarianNLP/modernbert-nl-sql" | |
| tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) | |
| model = BERTRewardModel(model_name=model_name) | |
| state_dict = torch.load("model.safetensors") # or use safetensors.torch.load_file | |
| model.load_state_dict(state_dict) | |
| model.eval() | |
| sql = "SELECT COUNT(*) FROM orders WHERE status = 'complete';" | |
| reasoning = "think: Count rows in orders filtered by status 'complete'." | |
| nl = "How many completed orders exist?" | |
| text = f"SQL: {sql}\nReasoning: {reasoning}\nNL: {nl}" | |
| inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=2048) | |
| score = model(**inputs)["scores"].item() | |
| print(f"Reward: {score:.3f}") | |
| ``` | |
| For convenience, `modeling_reward.py` exposes `load_finetuned_model(model_dir)` which handles loading `model.safetensors` or `pytorch_model.bin` and moves the module to GPU if available (falling back to CPU on OOM). | |
| ## Notes | |
| - The reward target is bounded `[0, 1]` and already penalizes copied NL or incorrect reasoning. | |
| - The model uses mean pooling instead of CLS to better leverage long ModernBERT contexts. | |
| - Tokenizer files are saved from the finetuned run; no extra special tokens were introduced. | |