Update Space (evaluate main: 83129c0f)

README.md

@@ -1,12 +1,77 @@
 ---
-title: Charcut Mt
-emoji: 🌖
-colorFrom: gray
-colorTo: green
+title: CharCut
+emoji: 🔤
+colorFrom: blue
+colorTo: red
 sdk: gradio
-sdk_version: 3.12.0
+sdk_version: 3.0.2
 app_file: app.py
 pinned: false
+tags:
+- evaluate
+- metric
+- machine-translation
+description: >-
+  CharCut is a character-based machine translation evaluation metric.
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Metric Card for CharacTER
+
+## Metric Description
+CharCut compares outputs of MT systems with reference translations. The matching algorithm is based on an iterative
+search for longest common substrings, combined with a length-based threshold that limits short and noisy character
+matches. As a similarity metric this is not new, but to the best of our knowledge it was never applied to highlighting
+and scoring of MT outputs. It has the neat effect of keeping character-based differences readable by humans.
+
+## Intended Uses
+CharCut was developed for machine translation evaluation.
+
+## How to Use
+
+```python
+import evaluate
+charcut = evaluate.load("charcut")
+preds = ["this week the saudis denied information published in the new york times",
+                "this is in fact an estimate"]
+refs = ["saudi arabia denied this week information published in the american new york times",
+                "this is actually an estimate"]
+results = charcut.compute(references=refs, predictions=preds)
+print(results)
+# {'charcut_mt': 0.1971153846153846}
+
+```
+### Inputs
+- **predictions**: a single prediction or a list of predictions to score. Each prediction should be a string with
+     tokens separated by spaces.
+- **references**: a single reference or a list of reference for each prediction. Each reference should be a string with
+     tokens separated by spaces.
+
+
+### Output Values
+- **charcut_mt**: the CharCut evaluation score (lower is better)
+
+### Output Example
+```python
+{'charcut_mt': 0.1971153846153846}
+```
+
+## Citation
+```bibtex
+@inproceedings{lardilleux-lepage-2017-charcut,
+    title = "{CHARCUT}: Human-Targeted Character-Based {MT} Evaluation with Loose Differences",
+    author = "Lardilleux, Adrien  and
+      Lepage, Yves",
+    booktitle = "Proceedings of the 14th International Conference on Spoken Language Translation",
+    month = dec # " 14-15",
+    year = "2017",
+    address = "Tokyo, Japan",
+    publisher = "International Workshop on Spoken Language Translation",
+    url = "https://aclanthology.org/2017.iwslt-1.20",
+    pages = "146--153",
+    abstract = "We present CHARCUT, a character-based machine translation evaluation metric derived from a human-targeted segment difference visualisation algorithm. It combines an iterative search for longest common substrings between the candidate and the reference translation with a simple length-based threshold, enabling loose differences that limit noisy character matches. Its main advantage is to produce scores that directly reflect human-readable string differences, making it a useful support tool for the manual analysis of MT output and its display to end users. Experiments on WMT16 metrics task data show that it is on par with the best {``}un-trained{''} metrics in terms of correlation with human judgement, well above BLEU and TER baselines, on both system and segment tasks.",
+}
+```
+
+## Further References
+- Repackaged version that is used in this HF implementation: [https://github.com/BramVanroy/CharCut](https://github.com/BramVanroy/CharCut)
+- Original version: [https://github.com/alardill/CharCut](https://github.com/alardill/CharCut)

app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("charcut_mt")
+launch_gradio_widget(module)

charcut_mt.py

@@ -0,0 +1,89 @@
+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An implementation for calculating CharCut, a character-based machine translation evaluation metric."""
+from typing import Iterable, Union
+
+import datasets
+from charcut import calculate_charcut
+from datasets import Sequence, Value
+
+import evaluate
+
+
+_CITATION = """\
+@inproceedings{lardilleux-lepage-2017-charcut,
+    title = "{CHARCUT}: Human-Targeted Character-Based {MT} Evaluation with Loose Differences",
+    author = "Lardilleux, Adrien  and
+      Lepage, Yves",
+    booktitle = "Proceedings of the 14th International Conference on Spoken Language Translation",
+    month = dec # " 14-15",
+    year = "2017",
+    address = "Tokyo, Japan",
+    publisher = "International Workshop on Spoken Language Translation",
+    url = "https://aclanthology.org/2017.iwslt-1.20",
+    pages = "146--153"
+}
+"""
+
+_DESCRIPTION = """\
+CharCut compares outputs of MT systems with reference translations. The matching algorithm is based on an iterative
+search for longest common substrings, combined with a length-based threshold that limits short and noisy character
+matches. As a similarity metric this is not new, but to the best of our knowledge it was never applied to highlighting
+and scoring of MT outputs. It has the neat effect of keeping character-based differences readable by humans."""
+
+_KWARGS_DESCRIPTION = """
+Calculates how good predictions are given some references.
+Args:
+    predictions: a list of predictions to score. Each prediction should be a string with
+     tokens separated by spaces.
+    references: a list of reference for each prediction. Each reference should be a string with
+     tokens separated by spaces.
+Returns:
+    charcut_mt: the CharCut score
+Examples:
+    >>> charcut_mt = evaluate.load("charcut_mt")
+    >>> preds = ["this week the saudis denied information published in the new york times",
+    ...          "this is in fact an estimate"]
+    >>> refs = ["saudi arabia denied this week information published in the american new york times",
+    ...         "this is actually an estimate"]
+    >>> charcut_mt.compute(references=refs, predictions=preds)
+    {'charcut_mt': 0.1971153846153846}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Charcut(evaluate.Metric):
+    """Character-based MT evaluation."""
+
+    def _info(self):
+        return evaluate.MetricInfo(
+            # This is the description that will appear on the modules page.
+            module_type="metric",
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            # This defines the format of each prediction and reference
+            features=[
+                datasets.Features(
+                    {"predictions": Value("string", id="prediction"), "references": Value("string", id="reference")}
+                ),
+            ],
+            # Homepage of the module for documentation
+            homepage="https://github.com/BramVanroy/CharCut",
+            # Additional links to the codebase or references
+            codebase_urls=["https://github.com/BramVanroy/CharCut", "https://github.com/alardill/CharCut"],
+        )
+
+    def _compute(self, predictions: Iterable[str], references: Iterable[str]):
+        return {"charcut_mt": calculate_charcut(predictions, references)[0]}

requirements.txt

@@ -0,0 +1,2 @@
+git+https://github.com/huggingface/evaluate@83129c0ff9053422f1031aa12d0c837ec6ff9b56
+charcut>=1.1.1