Upload README.md with huggingface_hub

README.md

@@ -22,7 +22,7 @@ datasets:
 - klue
 ---
 
-# Propositionizer-mT5-Small (Multilingual)
+# Propositionizer-mT5-Small v2 (Multilingual)
 
 A multilingual atomic fact decomposition model that converts unstructured text into a list of self-contained atomic propositions.
 
@@ -33,7 +33,7 @@ A multilingual atomic fact decomposition model that converts unstructured text i
 | Base Model | [google/mt5-small](https://huggingface.co/google/mt5-small) (300M params) |
 | Training Method | Claude → mT5-small distillation |
 | Languages | English, Korean, Japanese, Chinese |
-| Training Data | ~9,700 examples |
+| Training Data | v1: ~9,700 + v2: ~5,900 Korean examples |
 | Format | ONNX (int8 quantized) |
 | License | Apache 2.0 |
 
@@ -53,7 +53,7 @@ const decomposer = await pipeline(
 
 const result = await decomposer(
   'Title: Meeting. Section: . Content: The deadline is Friday and the rate was reduced.',
-  { max_new_tokens: 256 }
+  { max_new_tokens: 256, repetition_penalty: 2.0 }
 );
 console.log(JSON.parse(result[0].generated_text));
 // ["The deadline is Friday.", "The rate was reduced."]
@@ -69,7 +69,7 @@ model = AutoModelForSeq2SeqLM.from_pretrained("liliplanet/propositionizer-mt5-sm
 
 input_text = "Title: 회의. Section: . Content: 김 대리가 시급을 낮추고 마감은 금요일이다."
 inputs = tokenizer(input_text, return_tensors="pt")
-outputs = model.generate(**inputs, max_length=256)
+outputs = model.generate(**inputs, max_new_tokens=256, repetition_penalty=2.0, no_repeat_ngram_size=3, num_beams=4)
 print(tokenizer.decode(outputs[0], skip_special_tokens=True))
 ```
 
@@ -83,25 +83,23 @@ Title: {title}. Section: {section}. Content: {content}
 
 ## Training
 
+### v1
 - **Source texts**: CNN/DailyMail, XSum, Wikipedia (EN/KO/JA/ZH), KLUE, XLSum
-- **Labeling**: Claude (Haiku 4.5) atomic fact decomposition
-- **Data distribution**: EN 4,879 / KO 2,860 / JA 983 / ZH 975
-- **Training**: 5 epochs, Adafactor optimizer, lr=1e-3, batch_size=16, bf16
-- **Hardware**: NVIDIA RTX A6000
-
-## Evaluation
-
-Evaluated on held-out validation set (970 examples):
-
-| Language | Fuzzy F1 | Fact Count Accuracy |
-|----------|----------|-------------------|
-| English | 0.518 | 0.313 |
-| Korean | 0.219 | 0.108 |
-| Chinese | 0.104 | 0.193 |
-| Japanese | 0.075 | 0.222 |
-| **Overall** | **0.350** | **0.238** |
-
-Note: String-matching metrics understate actual quality. The model produces semantically correct decompositions that differ in surface form from references.
+- **Labeling**: Claude Haiku 4.5 atomic fact decomposition
+- **Data**: EN 4,879 / KO 2,860 / JA 983 / ZH 975 (~9,700 total)
+- **Training**: 5 epochs, Adafactor, lr=1e-3, batch_size=16
+
+### v2 (current)
+- **Focus**: Korean quality improvement
+- **Additional data**: ~5,900 Korean complex sentences (XLSum KO, KLUE NLI/RE/STS, KorQuAD, NSMC)
+- **Improved prompt**: language drift prevention, proper noun preservation
+- **Training**: continued training from v1, 3 epochs, lr=5e-4
+- **Generation config**: repetition_penalty=2.0, no_repeat_ngram_size=3
+
+### v2 improvements over v1
+- Korean language drift (한국어→영어 전환) resolved
+- Repetition loop eliminated
+- Proper noun preservation improved (CEO, Q2 등)
 
 ## Comparison with Original Propositionizer
 
@@ -112,12 +110,12 @@ Note: String-matching metrics understate actual quality. The model produces sema
 | Teacher | GPT-4 | Claude |
 | Training Data | English only | Multilingual |
 
-## Limitations
+## Known Limitations
 
 - Small model (300M) has limited capacity for complex decompositions
-- Korean performance needs improvement (priority for future work)
-- May hallucinate facts not present in the source text
+- May hallucinate facts not present in the source text, especially with uncommon proper nouns
 - Best suited for short-to-medium length paragraphs (< 500 chars)
+- Korean complex sentences with many IT terms may produce errors
 
 ## Citation
 
@@ -132,4 +130,4 @@ Note: String-matching metrics understate actual quality. The model produces sema
 
 ## Part of MemRosetta
 
-This model is a component of the [MemRosetta](https://github.com/memrosetta) project for multilingual memory and knowledge extraction.
+This model is a component of the [MemRosetta](https://github.com/obst2580/memrosetta) project for multilingual memory and knowledge extraction.