add description

README.md

@@ -1,5 +1,5 @@
 ---
-title: horizonmetrics
+title: horizon-metrics
 tags:
   - evaluate
   - metric
@@ -10,48 +10,87 @@ app_file: app.py
 pinned: false
 ---
 
-# Metric Card for horizonmetrics
+# Metric Card for horizon--metrics
 
 **_Module Card Instructions:_** _Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples._
 
-## Metric Description
+## SEA-AI/horizon-metrics
 
-_Give a brief overview of this metric, including what task(s) it is usually used for, if any._
+This huggingface metric uses `seametrics.horizon.HorizonMetrics` under the hood to calculate the slope and midpoint errors.
 
 ## How to Use
 
-_Give general statement of how to use the metric_
+To utilize horizon-metrics effectively, start by installing the necessary dependencies using the provided pip command. Once installed, import the evaluate library into your Python environment. Then, use the SEA-AI/horizon-metrics metric to evaluate your horizon prediction models. Ensure that both ground truth and prediction points are correctly formatted before computing the result. Finally, analyze the computed result to gain insights into the performance of your prediction models.
 
-_Provide simplest possible example for using the metric_
+### Getting Started
 
-### Inputs
+To get started with horizon-metrics, make sure you have the necessary dependencies installed. This metric relies on the `evaluate` and `seametrics` libraries.
 
-_List all input arguments in the format below_
+### Installation
 
-- **input_field** _(type): Definition of input, with explanation if necessary. State any default value(s)._
+```bash
+  pip install evaluate git+https://github.com/SEA-AI/seametrics@develop
+```
 
-### Output Values
+### Basic Usage
+
+This is how you can quickly evaluate your horizon prediction models using SEA-AI/horizon-metrics:
+
+```python
+  import evaluate
 
-_Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}_
+  ground_truth_points = [[[0.0, 0.5384765625], [1.0, 0.4931640625]],
+                        [[0.0, 0.53796875], [1.0, 0.4928515625]],
+                        [[0.0, 0.5374609375], [1.0, 0.4925390625]],
+                        [[0.0, 0.536953125], [1.0, 0.4922265625]],
+                        [[0.0, 0.5364453125], [1.0, 0.4919140625]]]
 
-_State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."_
+  prediction_points = [[[0.0, 0.5428930956049597], [1.0, 0.4642497615378973]],
+                      [[0.0, 0.5428930956049597], [1.0, 0.4642497615378973]],
+                      [[0.0, 0.523573113510805], [1.0, 0.47642688648919496]],
+                      [[0.0, 0.5200016849393765], [1.0, 0.4728554579177664]],
+                      [[0.0, 0.523573113510805], [1.0, 0.47642688648919496]]]
 
-#### Values from Popular Papers
 
-_Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported._
+  module = evaluate.load("SEA-AI/horizon-metrics")
+  module.add(predictions=ground_truth_points, references=prediction_points)
+  result = module.compute()
 
-### Examples
+  print(result)
+```
 
-_Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed._
+This is output the evalutaion metrics for your horizon prediciton model:
 
-## Limitations and Bias
+```python
+  {
+    'average_slope_error': 0.014823194839790999,
+    'average_midpoint_error': 0.014285714285714301,
+    'stddev_slope_error': 0.01519178791378349,
+    'stddev_midpoint_error': 0.0022661781575342445,
+    'max_slope_error': 0.033526146567062376,
+    'max_midpoint_error': 0.018161272321428612,
+    'num_slope_error_jumps': 1,
+    'num_midpoint_error_jumps': 1
+  }
+```
 
-_Note any known limitations or biases that the metric has, with links and references if possible._
+### Output Values
 
-## Citation
+SEA-AI/horizon-metrics provides the following performance metrics for horizon prediction:
 
-_Cite the source where this metric was introduced._
+- **average_slope_error**: Measures the average difference in slope between the predicted and ground truth horizon.
+- **average_midpoint_error**: Calculates the average difference in midpoint position between the predicted and ground truth horizon.
+- **stddev_slope_error**: Indicates the variability of errors in slope between the predicted and ground truth horizon.
+- **stddev_midpoint_error**: Quantifies the variability of errors in midpoint position between the predicted and ground truth horizon.
+- **max_slope_error**: Represents the maximum difference in slope between the predicted and ground truth horizon.
+- **max_midpoint_error**: Indicates the maximum difference in midpoint position between the predicted and ground truth horizon.
+- **num_slope_error_jumps**: Calculates the differences between errors in successive frames for the slope. It then counts the number of jumps in these errors by comparing the absolute differences to a specified threshold.
+- **num_midpoint_error_jumps**: Calculates the differences between errors in successive frames for the midpoint. It then counts the number of jumps in these errors by comparing the absolute differences to a specified threshold.
 
 ## Further References
 
-_Add any useful further references._
+Explore the [seametrics GitHub repository](https://github.com/SEA-AI/seametrics/tree/main) for more details on the underlying library.
+
+## Contribution
+
+Your contributions are welcome! If you'd like to improve SEA-AI/horizon-metrics or add new features, please feel free to fork the repository, make your changes, and submit a pull request.

horizon-metrics.py

@@ -11,65 +11,109 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""TODO: Add a description here."""
 
 import evaluate
 import datasets
-import numpy as np
 
 from seametrics.horizon.utils import *
 
-# TODO: Add BibTeX citation
 _CITATION = """\
 @InProceedings{huggingface:module,
-title = {A great new module},
+title = {Horizon Metrics},
 authors={huggingface, Inc.},
-year={2020}
+year={2024}
 }
 """
 
 # TODO: Add description of the module here
 _DESCRIPTION = """\
-This new module is designed to solve this great ML task and is crafted with a lot of care.
-"""
+TThis metric is intended to calculate horizon prediction metrics."""
 
 # TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
 Calculates how good are predictions given some references, using certain scores
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string with tokens separated by spaces.
-    references: list of reference for each prediction. Each
-        reference should be a string with tokens separated by spaces.
-Returns:
-    accuracy: description of the first score,
-    another_score: description of the second score,
-Examples:
-    Examples should be written in doctest format, and should illustrate how
-    to use the function.
+    predictions: list of predictions for each image. Each prediction
+        should be a nested array like this:
+        - [[x1, y1], [x2, y2]]
 
-    >>> my_new_module = evaluate.load("my_new_module")
-    >>> results = my_new_module.compute(references=[0, 1], predictions=[0, 1])
-    >>> print(results)
-    {'accuracy': 1.0}
-"""
+    references: list of references for each image. Each reference
+        should be a nested array like this:
+        - [[x1, y1], [x2, y2]]
+Returns:
+    dict containing following metrics:
+    'average_slope_error': Measures the average difference in slope between the predicted and ground truth horizon.
+    'average_midpoint_error': Calculates the average difference in midpoint position between the predicted and ground truth horizon.
+    'stddev_slope_error': Indicates the variability of errors in slope between the predicted and ground truth horizon.
+    'stddev_midpoint_error': Quantifies the variability of errors in midpoint position between the predicted and ground truth horizon.
+    'max_slope_error': Represents the maximum difference in slope between the predicted and ground truth horizon.
+    'max_midpoint_error': Indicates the maximum difference in midpoint position between the predicted and ground truth horizon.
+    'num_slope_error_jumps': Calculates the differences between errors in successive frames for the slope. It then counts the number of jumps in these errors by comparing the absolute differences to a specified threshold.
+    'num_midpoint_error_jumps': Calculates the differences between errors in successive frames for the midpoint. It then counts the number of jumps in these errors by comparing the absolute differences to a specified threshold.
 
-# TODO: Define external resources urls if needed
-BAD_WORDS_URL = "http://url/to/external/resource/bad_words.txt"
+Examples:
+    >>> ground_truth_points = [[[0.0, 0.5384765625], [1.0, 0.4931640625]],
+                       [[0.0, 0.53796875], [1.0, 0.4928515625]],
+                       [[0.0, 0.5374609375], [1.0, 0.4925390625]],
+                       [[0.0, 0.536953125], [1.0, 0.4922265625]],
+                       [[0.0, 0.5364453125], [1.0, 0.4919140625]]]
+
+    >>> prediction_points = [[[0.0, 0.5428930956049597], [1.0, 0.4642497615378973]],
+                     [[0.0, 0.5428930956049597], [1.0, 0.4642497615378973]],
+                     [[0.0, 0.523573113510805], [1.0, 0.47642688648919496]],
+                     [[0.0, 0.5200016849393765], [1.0, 0.4728554579177664]],
+                     [[0.0, 0.523573113510805], [1.0, 0.47642688648919496]]]
+
+
+    >>> module = evaluate.load("SEA-AI/horizon-metrics")
+    >>> module.add(predictions=ground_truth_points, references=prediction_points)
+    >>> module.compute()
+    >>> {'average_slope_error': 0.014823194839790999,
+         'average_midpoint_error': 0.014285714285714301,
+         'stddev_slope_error': 0.01519178791378349,
+         'stddev_midpoint_error': 0.0022661781575342445,
+         'max_slope_error': 0.033526146567062376,
+         'max_midpoint_error': 0.018161272321428612,
+         'num_slope_error_jumps': 1,
+         'num_midpoint_error_jumps': 1}
+    """
 
 
 @evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION,
                                                 _KWARGS_DESCRIPTION)
 class HorizonMetrics(evaluate.Metric):
-    """TODO: Short description of my evaluation module."""
+    """
+    HorizonMetrics is a metric class that calculates horizon prediction metrics.
+
+    Args:
+        roll_threshold (float, optional): Threshold for roll angle. Defaults to 0.5.
+        pitch_threshold (float, optional): Threshold for pitch angle. Defaults to 0.1.
+        vertical_fov_degrees (float, optional): Vertical field of view in degrees. Defaults to 25.6.
+        **kwargs: Additional keyword arguments.
+
+    Attributes:
+
+        slope_threshold (float): Threshold for slope calculated from roll threshold.
+        midpoint_threshold (float): Threshold for midpoint calculated from pitch threshold.
+        predictions (list): List of predicted horizons.
+        ground_truth_det (list): List of ground truth horizons.
+        slope_error_list (list): List of slope errors.
+        midpoint_error_list (list): List of midpoint errors.
+
+    Methods:
+
+        _info(): Returns the metric information.
+        add(predictions, references, **kwargs): Updates the predictions and ground truth detections.
+        _compute(predictions, references, **kwargs): Computes the horizon error across the sequence.
+    """
 
     def __init__(self,
                  roll_threshold=0.5,
                  pitch_threshold=0.1,
                  vertical_fov_degrees=25.6,
                  **kwargs):
-        super().__init__(**kwargs)
 
+        super().__init__(**kwargs)
         self.slope_threshold = roll_to_slope(roll_threshold)
         self.midpoint_threshold = pitch_to_midpoint(pitch_threshold,
                                                     vertical_fov_degrees)
@@ -79,7 +123,12 @@ class HorizonMetrics(evaluate.Metric):
         self.midpoint_error_list = None
 
     def _info(self):
-        # TODO: Specifies the evaluate.EvaluationModuleInfo object
+        """
+        Returns the metric information.
+
+        Returns:
+            MetricInfo: The metric information.
+        """
         return evaluate.MetricInfo(
             # This is the description that will appear on the modules page.
             module_type="metric",
@@ -88,30 +137,25 @@ class HorizonMetrics(evaluate.Metric):
             inputs_description=_KWARGS_DESCRIPTION,
             # This defines the format of each prediction and reference
             features=datasets.Features({
-                'predictions': datasets.Value('int64'),
-                'references': datasets.Value('int64'),
+                'predictions':
+                datasets.Sequence(datasets.Value("float")),
+                'references':
+                datasets.Sequence(datasets.Value("float")),
             }),
-            # Homepage of the module for documentation
-            homepage="http://module.homepage",
-            # Additional links to the codebase or references
-            codebase_urls=["http://github.com/path/to/codebase/of/new_module"],
-            reference_urls=["http://path.to.reference.url/new_module"])
+            codebase_urls=["http://github.com/path/to/codebase/of/new_module"])
 
     def add(self, *, predictions, references, **kwargs):
         """
-        Update the predictions and ground truth detections.
-
-        Parameters
-        ----------
-        predictions : list
-            List of predicted horizons.
-        ground_truth_det : list
-            List of ground truth horizons.
+        Updates the predictions and ground truth detections.
 
+        Parameters:
+            predictions (list): List of predicted horizons.
+            references (list): List of ground truth horizons.
+            **kwargs: Additional keyword arguments.
         """
-
-        # does not impact the metric, but is required for the interface x_x
-        super(evaluate.Metric, self).add(prediction=0, references=0, **kwargs)
+        super(evaluate.Metric, self).add(prediction=predictions,
+                                         references=references,
+                                         **kwargs)
 
         self.predictions = predictions
         self.ground_truth_det = references
@@ -127,19 +171,11 @@ class HorizonMetrics(evaluate.Metric):
 
     def _compute(self, *, predictions, references, **kwargs):
         """
-        Compute the horizon error across the sequence.
-
-        Returns
-        -------
-        float
-            The computed horizon error.
+        Computes the horizon error across the sequence.
 
+        Returns:
+            float: The computed horizon error.
         """
         return calculate_horizon_error_across_sequence(
             self.slope_error_list, self.midpoint_error_list,
             self.slope_threshold, self.midpoint_threshold)
-
-    def _download_and_prepare(self, dl_manager):
-        """Optional: download external resources useful to compute the scores"""
-        # TODO: Download external resources if needed
-        pass