Initial Commit

.gitignore

@@ -0,0 +1 @@
+*.bak

README.md

@@ -1,50 +1,24 @@
 ---
-title: ChineseMETEOR
-datasets:
--  
+library_name: evaluate
 tags:
-- evaluate
-- metric
-description: "TODO: add a description here"
-sdk: gradio
-sdk_version: 3.19.1
-app_file: app.py
-pinned: false
+  - nlp
+  - translation
+  - chinese
+  - meteor
+  - jieba
+license: apache-2.0
 ---
 
-# Metric Card for ChineseMETEOR
+# METEOR (Chinese) with Jieba
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
+Classic METEOR score, but **pre-segmented with Jieba** so it works on raw Chinese text.
 
-## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
-
-## How to Use
-*Give general statement of how to use the metric*
-
-*Provide simplest possible example for using the metric*
-
-### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
-
-### Output Values
-
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
-
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
-
-#### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
-
-### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
-
-## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
-
-## Citation
-*Cite the source where this metric was introduced.*
-
-## Further References
-*Add any useful further references.*
+```python
+import evaluate
+meteor = evaluate.load("raptorkwok/chinese_meteor")
+results = meteor.compute(
+    predictions=["我在這裡吃飯"],
+    references=["我在這裡吃飯"]
+)
+print(results)
+# {'meteor': 1.0}

chinesemeteor.py

@@ -1,95 +1,196 @@
-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""TODO: Add a description here."""
-
-import evaluate
+# -*- coding: utf-8 -*-
+"""
+METEOR (Chinese) — with Jieba pre-segmentation + Real CwnGraph Chinese WordNet
+HuggingFace evaluate metric template
+"""
+import jieba_fast as jieba
 import datasets
+from typing import List, Dict
+import numpy as np
+from nltk.translate import meteor_score
+from nltk import word_tokenize
+#import nltk
+import evaluate
+import re
 
+# Download once
+#nltk.download("wordnet", quiet=True)
+#nltk.download("omw-1.4", quiet=True)
+#nltk.download("punkt", quiet=True)
 
-# TODO: Add BibTeX citation
-_CITATION = """\
-@InProceedings{huggingface:module,
-title = {A great new module},
-authors={huggingface, Inc.},
-year={2020}
-}
-"""
+# ------------------------------------------------------------------- #
+#  REAL Chinese WordNet (CwnGraph) Integration
+# ------------------------------------------------------------------- #
+_cwn = None
+def _load_cwn():
+    global _cwn
+    if _cwn is None:
+        try:
+            from CwnGraph import CwnImage
+            print("Loading Chinese WordNet (CwnGraph, first time only)...")
+            _cwn = CwnImage.latest()
+        except ImportError:
+            raise ImportError("CwnGraph failed to load. Run: pip install CwnGraph")
+    return _cwn
+
+# Helper to get lemma name (with fallback for API versions)
+def _get_lemma_name(lemma):
+    try:
+        return lemma.name
+    except AttributeError:
+        return str(lemma).split(': ')[1].split('_')[0]
+
+# Custom Lemma & Synset for NLTK compatibility
+class _CwnLemma:
+    def __init__(self, name): self._name = name
+    def name(self): return self._name
+
+class _CwnSynset:
+    def __init__(self, lemmas, synset_id): 
+        self._lemmas = lemmas
+        self._id = synset_id
+    def lemmas(self):
+        return [_CwnLemma(name) for name in self._lemmas]
+
+# ------------------------------------------------------------------- #
+#  HuggingFace Evaluation Metric
+# ------------------------------------------------------------------- #
 
-# TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
+This evaluation metric is tailor-made to evaluate the translation quality of Chinese translation.
 """
 
-
-# TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
 Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
+    predictions (str): translation sentence to score.
+    references (str): reference sentence for each translation.
 Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
+    meteor: the average METEOR score
+    scores: the METEOR score for each sentence pairs
+    
 Examples:
     Examples should be written in doctest format, and should illustrate how
     to use the function.
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
+    >>> cmeteor = evaluate.load("raptorkwok/chinesemeteor")
+    >>> results = cmeteor.compute(references=["Reference Sentence in Chinese"], predictions=["Predicted Sentence in Chinese"])
     >>> print(results)
-    {'accuracy': 1.0}
+    {'meteor': 0.5111111111111111, 'scores': [0.5111111111111111]}
 """
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
-
-
-@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+# ------------------------------------------------------------------- #
+#  HuggingFace evaluate template
+# ------------------------------------------------------------------- #
 class ChineseMETEOR(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
-
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
         return evaluate.MetricInfo(
-            # This is the description that will appear on the modules page.
             module_type="metric",
             description=_DESCRIPTION,
-            citation=_CITATION,
+            citation="""@inproceedings{denkowski-lavie-2014-meteor,
+                title = "Meteor Universal: Language Specific Translation Evaluation for Any Target Language",
+                author = "Denkowski, Michael  and  Lavie, Alon",
+                booktitle = "Proceedings of the Ninth Workshop on Statistical Machine Translation",
+                year = "2014"
+            }""",
             inputs_description=_KWARGS_DESCRIPTION,
-            # This defines the format of each prediction and reference
-            features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
-            }),
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Value("string"),
+                    "references": datasets.Value("string"),
+                }
+            ),
             # Homepage of the module for documentation
-            homepage="http://module.homepage",
+            homepage="https://yourappapp.com",
             # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"]
+            codebase_urls=["https://github.com/nltk/nltk"],
+            reference_urls=["https://www.cs.cmu.edu/~alavie/METEOR/"],
         )
 
-    def _download_and_prepare(self, dl_manager):
+    def _download_and_prepare(self, dl_manager) -> None:
         """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
+        # CwnGraph auto-downloads on first use
+        import nltk
+        nltk.download("wordnet", quiet=True)
+        nltk.download("omw-1.4", quiet=True)
+        nltk.download("punkt", quiet=True)
         pass
 
-    def _compute(self, predictions, references):
-        """Returns the scores"""
-        # TODO: Compute the different scores of the module
-        accuracy = sum(i == j for i, j in zip(predictions, references)) / len(predictions)
+    def _compute(self, predictions: List[str], references: List[str]) -> Dict[str, float]:
+        pred_seg = [" ".join(jieba.cut(p.strip())) for p in predictions]
+        ref_seg  = [" ".join(jieba.cut(r.strip())) for r in references]
+
+        # --- FORCE Real CWN INTO METEOR ---
+        def _cwn_synsets(self, word, pos=None):  # Matches NLTK method call
+            if not isinstance(word, str) or not word.strip():
+                print(f"DEBUG: Skipping non-string input: {type(word)}")
+                return []
+            cwn = _load_cwn()
+            try:
+                # Use escaped regex for exact match (CwnGraph expects string pattern)
+                pattern = f"^{re.escape(word)}$"
+                lemmas = cwn.find_lemma(pattern)
+            except Exception as e:
+                print(f"DEBUG: Error querying CWN for '{word}': {e}")
+                return []
+            # FIXED: Use _get_lemma_name for comparison (handles missing .name)
+            exact_lemmas = [l for l in lemmas if _get_lemma_name(l) == word]
+            if not exact_lemmas:
+                print(f"DEBUG: No exact lemma found for '{word}'")
+                return []
+            synsets_list = []
+            seen_synset_ids = set()
+            for lemma in exact_lemmas:
+                for sense in lemma.senses:
+                    synset = sense.synset
+                    if synset:
+                        try:
+                            synset_id = synset.id
+                        except AttributeError:
+                            synset_id = str(synset)
+                        if synset_id not in seen_synset_ids:
+                            seen_synset_ids.add(synset_id)
+                            try:
+                                synset_lemmas = synset.lemmas
+                                syn_lemma_names = [_get_lemma_name(l) for l in synset_lemmas]
+                            except AttributeError:
+                                synset_lemmas = []
+                                for s in synset.senses:
+                                    try:
+                                        # Access the single lemma via lemmas[0]
+                                        lemma = s.lemmas[0]
+                                        synset_lemmas.append(lemma)
+                                    except (AttributeError, IndexError, TypeError):
+                                        try:
+                                            lemma = s.lemma
+                                            synset_lemmas.append(lemma)
+                                        except AttributeError:
+                                            print(f"DEBUG: Could not extract lemma from sense {s}")
+                                            continue
+                                syn_lemma_names = [_get_lemma_name(l) for l in synset_lemmas]
+                            syn_lemmas_set = set(syn_lemma_names)
+                            if syn_lemmas_set:
+                                synsets_list.append(_CwnSynset(list(syn_lemmas_set), synset_id))
+            print(f"DEBUG: Found {len(synsets_list)} synsets for '{word}': {synsets_list[0]._lemmas if synsets_list else []}")
+            return synsets_list
+
+        # Use class for proper method binding
+        class ChineseWordNet:
+            def synsets(self, word, pos=None):
+                return _cwn_synsets(self, word, pos)
+
+        chinese_wn = ChineseWordNet()
+
+        scores = [
+            meteor_score.single_meteor_score(
+                word_tokenize(ref),
+                word_tokenize(hyp),
+                wordnet=chinese_wn
+            )
+            for ref, hyp in zip(ref_seg, pred_seg)
+        ]
+
         return {
-            "accuracy": accuracy,
-        }
+            "meteor": float(np.mean(scores)),
+            "scores": scores,
+        }

requirements.txt

@@ -1 +1,5 @@
-git+https://github.com/huggingface/evaluate@main
+evaluate>=0.4.1
+jieba_fast
+CwnGraph>=0.3.0
+nltk>=3.8
+numpy