from transformers import AutoModelForCausalLM, AutoTokenizer
model_path = "datumo/E-Star-12B-v2-Base"
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_path)
# ββ System Prompt ββ
system_prompt = """You are a rubric evaluator.Your task is to evaluate a response strictly and only according to the provided pass criteria and scoring rubric. In your output, return the final evaluation (the three output tags: <feedback>, <highlight>, and <decision>).# Evaluation Procedure (must follow all steps):1. First, carefully read the Data to Evaluate, the pass criteria, and the scoring rubric to fully understand the requirements.2. Evaluate the response only against the given criteria: do not introduce external standards, do not reward style unless the rubric explicitly allows it, and judge by absolute rubric definitions rather than relative comparisons.3. Re-check fine-grained details in the response and the rubric, ensuring any tags (if present) are correctly mapped to the pass criteria and that small deviations are not overlooked.4. Write criterion-focused feedback that explicitly references the rubric, quoting exact words or phrases from the response when they are decisive, and clearly stating which criteria are satisfied and which are violated.5. Finally, extract the key verbatim spans that most influenced your judgment and assign the final score according to the scoring rubric."""# ββ User Prompt μμ (Reasoning / Problem Solving) ββ
user_prompt = """You MUST write ALL output (<feedback>, <highlight>, <decision>) in the SAME language as the input question and response being evaluated. If the input is in Korean, your entire output MUST be in Korean.# Output Format:<feedback>Write detailed feedback (reasons) that strictly evaluates the quality of the response using only the given scoring rubric. Do not explicitly state the score in a sentence (e.g., "Therefore, the score is β¦").</feedback><highlight>List of words or phrases that you believe are the most important in determining the score.</highlight><decision>Provide the final integer score assigned based on the scoring rubric.</decision># Data to Evaluate### Problemν 곡μ₯μμ ν루μ 120κ°μ μ νμ μμ°νλ€. λΆλλ₯ μ΄ 5%μΌ λ, μΌμ£ΌμΌ(7μΌ) λμ μμ°λλ μ μ μ νμ μλ?### Model Responseν루 μμ°λ: 120κ°λΆλλ₯ : 5% β λΆλν: 120 Γ 0.05 = 6κ°ν루 μ μ μ ν: 120 - 6 = 114κ°μΌμ£ΌμΌ μ μ μ ν: 114 Γ 7 = 798κ°### Optional Ground Truth798κ°# RubricEvaluate whether the model correctly solves the problem and provides reasoning that is logically consistent with the final answer. Prioritize correctness of the conclusion, then soundness of the reasoning.Score 1: The final answer is wrong and the reasoning is invalid, irrelevant, or missing.Score 2: The response shows limited progress but contains major reasoning flaws leading to an incorrect or unreliable answer.Score 3: The response demonstrates partial reasoning ability but is incomplete, contains mistakes, or reaches an uncertain result.Score 4: The response is mostly correct with generally sound reasoning, though minor errors or gaps may remain.Score 5: The response reaches the correct answer through clear, consistent, and logically valid reasoning appropriate to the problem."""# ββ Inference ββ
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(
input_ids,
max_new_tokens=2048,
temperature=0.0,
do_sample=False,
)
response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)
