NebulaPixel/SummOrchestra-3B-GRPO-BRL-CSDS

SummOrchestra: Coordinating Multi-role Dialogue Understanding and Summarization

📊 Performance Comparison of SummOrchestra on the CSDS Dataset

Model	Training Method	ROUGE-1	ROUGE-2	ROUGE-L	BERTScore
gpt-4.1	-	42.91	14.08	32.04	77.37
gpt-4.1-mini	-	42.78	14.11	32.31	77.40
gpt-4o	-	48.52	19.59	36.55	79.51
gpt-4o-mini	-	45.87	16.93	33.82	78.28
gpt-5	-	39.72	11.47	28.98	75.63
gpt-5-mini	-	41.54	12.95	30.15	76.68
qwen2.5-3B	SFT	56.87	31.35	47.40	82.58
qwen2.5-3B	GRPO(B+R+L)	58.91	33.47	49.69	83.36
qwen2.5-7B	SFT	58.11	33.19	49.05	83.29
qwen2.5-7B	GRPO(B+R+L)	59.32	34.53	50.40	83.64

Note:

• Results are evaluated on the CSDS (Chinese Summarization Dialogues Dataset).
• ROUGE scores are computed using rouge-chinese with F1-based evaluation.
• BERTScore is calculated using the google-bert/bert-base-chinese model.
• GRPO (B+R+L) indicates that the reward function combines BERTScore (B), ROUGE (R), and Length (L) rewards.

🧩 Example: Using Your Fine-Tuned Summarization Model

This is a minimal example showing how to load and run our model.

📦 Install dependencies

pip install transformers torch

Run and infere Results

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "dada122/SummOrchestra-7B-GRPO-BRL-CSDS"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Example input (same structure as your dataset)
messages = [
    {"role": "system", "content": "You are a summarization assistant."},
    {
        "role": "user",
        "content": (
            "Summarize the following conversation:\n"
            "agent: 有什么问题我可以帮您处理或解决呢?\n"
            "user: 你好\n"
            "user: 以前的手机号码销号了,密码也忘了\n"
            "agent: ..."
        )
    }
]

# Convert messages into a single text string
input_text = "\n".join(f"{m['role']}: {m['content']}" for m in messages)

# Generate the summary
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))