backwards compatibility with 'add_batch' approach but with deprecation warning

README.md

@@ -1,11 +1,10 @@
 ---
-title: Detection Metrics
+title: det-metrics
 tags:
 - evaluate
 - metric
 description: >-
-  Compute multiple object detection metrics at different bounding box area
-  levels.
+  Modified cocoevals.py which is wrapped into torchmetrics' mAP metric with numpy instead of torch dependency.
 sdk: gradio
 sdk_version: 3.19.1
 app_file: app.py
@@ -13,12 +12,75 @@ pinned: false
 emoji: 🕵️
 ---
 
-# Detection Metrics
+# SEA-AI/det-metrics
 
-## Description
-This metric can be used to calculate object detection metrics. It has an option to calculate the metrics at different levels of bounding box sizes, so that more insight is provided into the performance for different objects. It is adapted from the base of pycocotools metrics.
+This hugging face metric uses `seametrics.detection.PrecisionRecallF1Support` under the hood to compute coco-like metrics for object detection tasks. It is a [modified cocoeval.py](https://github.com/SEA-AI/seametrics/blob/develop/seametrics/detection/cocoeval.py) wrapped inside [torchmetrics' mAP metric](https://lightning.ai/docs/torchmetrics/stable/detection/mean_average_precision.html) but with numpy arrays instead of torch tensors.
+
+## Getting Started
+
+To get started with det-metrics, make sure you have the necessary dependencies installed. This metric relies on the `evaluate` and `seametrics` libraries for metric calculation and integration with FiftyOne datasets.
+
+### Installation
+
+First, ensure you have Python 3.8 or later installed. Then, install det-metrics using pip:
+
+```sh
+pip install evaluate git+https://github.com/SEA-AI/seametrics@develop
+```
+
+### Basic Usage
+
+Here's how to quickly evaluate your object detection models using SEA-AI/det-metrics:
+
+```python
+import evaluate
+
+# Define your predictions and references (dict values can also by numpy arrays)
+predictions = [
+    {
+        "boxes": [[449.3, 197.75390625, 6.25, 7.03125], [334.3, 181.58203125, 11.5625, 6.85546875]],
+        "labels": [0, 0],
+        "scores": [0.153076171875, 0.72314453125],
+    }
+]
+
+references = [
+    {
+        "boxes": [[449.3, 197.75390625, 6.25, 7.03125], [334.3, 181.58203125, 11.5625, 6.85546875]],
+        "labels": [0, 0],
+        "area": [132.2, 83.8],
+    }
+]
+
+# Load SEA-AI/det-metrics and evaluate
+module = evaluate.load("SEA-AI/det-metrics")
+module.add(prediction=predictions, reference=references)
+results = module.compute()
+
+print(results)
+```
+
+This will output the evaluation metrics for your detection model.
+```
+{'all': {'range': [0, 10000000000.0],
+  'iouThr': '0.00',
+  'maxDets': 100,
+  'tp': 2,
+  'fp': 0,
+  'fn': 0,
+  'duplicates': 0,
+  'precision': 1.0,
+  'recall': 1.0,
+  'f1': 1.0,
+  'support': 2,
+  'fpi': 0,
+  'nImgs': 1}
+```
+
+## FiftyOne Integration
+
+Integrate SEA-AI/det-metrics with FiftyOne datasets for enhanced analysis and visualization:
 
-## How to Use
 ```python
 import evaluate
 import logging
@@ -26,6 +88,7 @@ from seametrics.payload import PayloadProcessor
 
 logging.basicConfig(level=logging.WARNING)
 
+# Configure your dataset and model details
 processor = PayloadProcessor(
     dataset_name="SAILING_DATASET_QA",
     gt_field="ground_truth_det",
@@ -34,9 +97,12 @@ processor = PayloadProcessor(
     data_type="rgb",
 )
 
+# Evaluate using SEA-AI/det-metrics
 module = evaluate.load("SEA-AI/det-metrics")
-module.add_from_payload(processor.payload)
-module.compute()
+module.add_payload(processor.payload)
+results = module.compute()
+
+print(results)
 ```
 
 ```console
@@ -55,205 +121,16 @@ module.compute()
   'nImgs': 22}}
 ```
 
-### Metric Settings
-When loading module: `module = evaluate.load("SEA-AI/det-metrics", **params)`, multiple parameters can be specified.
-- **area_ranges_tuples** *List[Tuple[str, List[int]]]*: Different levels of area ranges at which metrics should be calculated. It is a list that contains tuples, where the first element of each tuple should specify the name of the area range and the second element is list specifying the lower and upper limit of the area range. Defaults to `[("all", [0, 1e5.pow(2)])]`.
-- **bbox_format** *Literal["xyxy", "xywh", "cxcywh"]*: Bounding box format of predictions and ground truth. Defaults to `"xywh"`.
-- **iou_threshold** *Optional[float]*: at which IOU-treshold the metrics should be calculated. IOU-threshold defines the minimal overlap between a ground truth and predicted bounding box so that it is considered a correct prediction. Defaults to `1e-10`.
-- **class_agnostic** *bool*. Defaults to `True`. Non-class-agnostic metrics are currently not supported.
-
-
-### Input Values
-Add predictions and ground truths to the metric with the function `module.add_batches(payload)`.
-The format of payload should be as returned by function `fo_to_payload()` defined in seametrics library. 
-An example of how a payload might look like is:
-
-``` 
-{
-    'dataset': 'SAILING_DATASET_QA',
-    'models': ['yolov5n6_RGB_D2304-v1_9C'
-    ],
-    'gt_field_name': 'ground_truth_det',
-    'sequences': {
-        # sequence 1,
-        1 frame with 1 pred and 1 gt
-        'Trip_14_Seq_1': {
-            'resolution': (720,
-            1280),
-            'yolov5n6_RGB_D2304-v1_9C': [
-                [fo.Detection(
-                    label='FAR_AWAY_OBJECT',
-                    bounding_box=[
-                        0.35107421875,
-                        0.274658203125,
-                        0.0048828125,
-                        0.009765625
-                    ], # tp nr1
-                    confidence=0.153076171875
-                )
-                ]
-            ],
-                'ground_truth_det': [
-                [fo.Detection(
-                    label='FAR_AWAY_OBJECT',
-                    bounding_box=[
-                        0.35107421875,
-                        0.274658203125,
-                        0.0048828125,
-                        0.009765625
-                    ]
-                )
-                ]
-            ]
-        },
-        # sequence 2,
-        2 frames with frame 1: 2 pred,
-        1 gt; frame 2: 1 pred 1 gt
-        'Trip_14_Seq_2': {
-            'resolution': (720,
-            1280),
-            'yolov5n6_RGB_D2304-v1_9C': [
-                [
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.389404296875,
-                        0.306640625,
-                        0.005126953125,
-                        0.0146484375
-                    ], # tp nr 2
-                        confidence=0.153076171875
-                        ),
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.50390625,
-                        0.357666015625,
-                        0.0048828125,
-                        0.00976562
-                    ], # fp nr 1
-                        confidence=0.153076171875
-                        ),
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.455078125,
-                        0.31494140625,
-                        0.00390625,
-                        0.0087890625
-                    ], # fp nr 2
-                        confidence=0.153076171875
-                    )
-                ],
-                [
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.455078125,
-                        0.31494140625,
-                        0.00390625,
-                        0.0087890625
-                    ], # tp nr 3
-                        confidence=0.153076171875
-                        )
-                ],
-                [
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.455078125,
-                        0.31494140625,
-                        0.00390625,
-                        0.0087890625
-                    ], # fp nr 3
-                        confidence=0.153076171875
-                        )
-                ]
-            ],
-            'ground_truth_det': [
-                # frame nr 1
-                [
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.389404296875,
-                        0.306640625,
-                        0.005126953125,
-                        0.0146484375
-                    ],
-                        )
-                ],
-                # frame nr 2
-                [
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.455078125,
-                        0.31494140625,
-                        0.00390625,
-                        0.0087890625
-                    ],
-                        confidence=0.153076171875
-                    ),
-                    fo.Detection(
-                        label='FAR_AWAY_OBJECT',
-                        bounding_box=[
-                        0.35107421875,
-                        0.274658203125,
-                        0.0048828125,
-                        0.009765625
-                    ], # missed nr 1
-                        confidence=0.153076171875
-                    )
-                ],
-                # frame nr3
-                [],
-            ]
-        }
-    },
-    "sequence_list": [
-        "Trip_14_Seq_1", 'Trip_14_Seq_2'
-    ]
-}
-```
+## Metric Settings
 
-Optionally, you can pass the model as string that should be evaluated, via `model=model_str`. By default, it will evaluate the first model, i.e. `model = payload["models"][0]`.
-
-### Output Values
-The metric outputs a dictionary that contains sub-dictionaries for each name of the specified area ranges.
-Each sub-dictionary holds performance metrics at the specific area range level:
-- **range**: corresponding area range
-- **iouThr**: IOU-threshold used in calculating the metric
-- **maxDets**: maximum number of detections in calculating the metrics
-- **tp**: number of true positive predictions
-- **fp**: number of false positive predictions
-- **fn**: number of false negative predictions
-- **duplicates**: number of duplicated bounding box predictions
-- **precision**: ratio between true positive predictions and positive predictions (tp/(tp+fp))
-- **recall**: ratio between true positive predictions and actual ground truths (tp/(tp+fn))
-- **f1**: trades-off precision and recall (2*(precision*recall)/(precision+recall))
-- **support**: number of ground truth bounding boxes that are considered in the metric
-- **fpi**: number of images with predictions but no ground truths
-- **nImgs**: number of total images considered in calculating the metric
-
-
-### Examples
-We can specify different area range levels, at which we would like to compute the metrics.
-```python
-import evaluate
-import logging
-from seametrics.payload import PayloadProcessor
+Customize your evaluation by specifying various parameters when loading SEA-AI/det-metrics:
 
-logging.basicConfig(level=logging.WARNING)
-
-processor = PayloadProcessor(
-    dataset_name="SAILING_DATASET_QA",
-    gt_field="ground_truth_det",
-    models=["yolov5n6_RGB_D2304-v1_9C"],
-    sequence_list=["Trip_14_Seq_1"],
-    data_type="rgb",
-)
+- **area_ranges_tuples**: Define different area ranges for metrics calculation.
+- **bbox_format**: Set the bounding box format (e.g., `"xywh"`).
+- **iou_threshold**: Choose the IOU threshold for determining correct detections.
+- **class_agnostic**: Specify whether to calculate metrics disregarding class labels.
 
+```python
 area_ranges_tuples = [
     ("all", [0, 1e5**2]),
     ("small", [0**2, 6**2]),
@@ -266,68 +143,28 @@ module = evaluate.load(
     iou_thresholds=[0.00001],
     area_ranges_tuples=area_ranges_tuples,
 )
-module.add_from_payload(processor.payload)
-module.compute()
 ```
 
-```
-{'all': {'range': [0, 10000000000.0],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 89,
-  'fp': 13,
-  'fn': 15,
-  'duplicates': 1,
-  'precision': 0.8725490196078431,
-  'recall': 0.8557692307692307,
-  'f1': 0.8640776699029126,
-  'support': 104,
-  'fpi': 0,
-  'nImgs': 22},
- 'small': {'range': [0, 36],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 12,
-  'fp': 3,
-  'fn': 8,
-  'duplicates': 0,
-  'precision': 0.8,
-  'recall': 0.6,
-  'f1': 0.6857142857142857,
-  'support': 20,
-  'fpi': 0,
-  'nImgs': 22},
- 'medium': {'range': [36, 144],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 50,
-  'fp': 10,
-  'fn': 7,
-  'duplicates': 1,
-  'precision': 0.8333333333333334,
-  'recall': 0.8771929824561403,
-  'f1': 0.8547008547008548,
-  'support': 57,
-  'fpi': 0,
-  'nImgs': 22},
- 'large': {'range': [144, 10000000000.0],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 27,
-  'fp': 0,
-  'fn': 0,
-  'duplicates': 0,
-  'precision': 1.0,
-  'recall': 1.0,
-  'f1': 1.0,
-  'support': 27,
-  'fpi': 0,
-  'nImgs': 22}}
-```
+## Output Values
+
+SEA-AI/det-metrics provides a detailed breakdown of performance metrics for each specified area range:
+
+- **range**: The area range considered.
+- **iouThr**: The IOU threshold applied.
+- **maxDets**: The maximum number of detections evaluated.
+- **tp/fp/fn**: Counts of true positives, false positives, and false negatives.
+- **duplicates**: Number of duplicate detections.
+- **precision/recall/f1**: Calculated precision, recall, and F1 score.
+- **support**: Number of ground truth boxes considered.
+- **fpi**: Number of images with predictions but no ground truths.
+- **nImgs**: Total number of images evaluated.
 
 ## Further References
-*seametrics* library: https://github.com/SEA-AI/seametrics/tree/main
 
-Calculating metrics is based on pycoco tools: https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools
+- **seametrics Library**: Explore the [seametrics GitHub repository](https://github.com/SEA-AI/seametrics/tree/main) for more details on the underlying library.
+- **Pycoco Tools**: SEA-AI/det-metrics calculations are based on [pycoco tools](https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools), a widely used library for COCO dataset evaluation.
+- **Understanding Metrics**: For a deeper understanding of precision, recall, and other metrics, read [this comprehensive guide](https://www.analyticsvidhya.com/blog/2020/09/precision-recall-machine-learning/).
+
+## Contribution
 
-Further info about metrics: https://www.analyticsvidhya.com/blog/2020/09/precision-recall-machine-learning/
+Your contributions are welcome! If you'd like to improve SEA-AI/det-metrics or add new features, please feel free to fork the repository, make your changes, and submit a pull request.

det-metrics.py

@@ -14,14 +14,13 @@
 """TODO: Add a description here."""
 
 from typing import List, Tuple, Literal
+from deprecated import deprecated
 
 import evaluate
 import datasets
 import numpy as np
 
 from seametrics.detection import PrecisionRecallF1Support
-from seametrics.detection.utils import payload_to_det_metric
-from seametrics.payload import Payload
 
 _CITATION = """\
 @InProceedings{coco:2020,
@@ -125,16 +124,12 @@ class DetectionMetric(evaluate.Metric):
         **kwargs
     ):
         super().__init__(**kwargs)
-        area_ranges = [v for _, v in area_ranges_tuples]
-        area_ranges_labels = [k for k, _ in area_ranges_tuples]
-        iou_threshold = (
-            [iou_threshold] if not isinstance(iou_threshold, list) else iou_threshold
-        )
-
         self.coco_metric = PrecisionRecallF1Support(
-            iou_thresholds=iou_threshold,
-            area_ranges=area_ranges,
-            area_ranges_labels=area_ranges_labels,
+            iou_thresholds=(
+                iou_threshold if isinstance(iou_threshold, list) else [iou_threshold]
+            ),
+            area_ranges=[v for _, v in area_ranges_tuples],
+            area_ranges_labels=[k for k, _ in area_ranges_tuples],
             class_agnostic=class_agnostic,
             iou_type=iou_type,
             box_format=bbox_format,
@@ -183,15 +178,45 @@ class DetectionMetric(evaluate.Metric):
 
     def add(self, *, prediction, reference, **kwargs):
         """Adds a batch of predictions and references to the metric"""
+        # in case the inputs are lists, convert them to numpy arrays
+        prediction = self._preprocess(prediction)
+        reference = self._preprocess(reference)
+
         self.coco_metric.update(prediction, reference)
 
         # does not impact the metric, but is required for the interface x_x
         super(evaluate.Metric, self).add(
-            prediction=[self._np_to_lists(p) for p in prediction],
-            references=[self._np_to_lists(r) for r in reference],
+            prediction=self._postprocess(prediction),
+            references=self._postprocess(reference),
             **kwargs
         )
 
+    @deprecated(reason="Use `module.from_payload` instead")
+    def add_batch(self, payload, model_name: str = None):
+        """Takes as input a payload and adds the batch to the metric"""
+        self.add_payload(payload, model_name)
+
+    def _compute(self, *, predictions, references, **kwargs):
+        """Called within the evaluate.Metric.compute() method"""
+        return self.coco_metric.compute()["metrics"]
+
+    def add_payload(self, payload, model_name: str = None):
+        """Converts the payload to the format expected by the metric"""
+        # import only if needed since fiftyone is not a direct dependency
+        from seametrics.detection.utils import payload_to_det_metric
+
+        predictions, references = payload_to_det_metric(payload, model_name)
+        self.add(prediction=predictions, reference=references)
+        return self
+
+    def _preprocess(self, list_of_dicts):
+        """Converts the lists to numpy arrays for type checking"""
+        return [self._lists_to_np(d) for d in list_of_dicts]
+
+    def _postprocess(self, list_of_dicts):
+        """Converts the numpy arrays to lists for type checking"""
+        return [self._np_to_lists(d) for d in list_of_dicts]
+
     def _np_to_lists(self, d):
         """datasets does not support numpy arrays for type checking"""
         for k, v in d.items():
@@ -201,12 +226,11 @@ class DetectionMetric(evaluate.Metric):
                 d[k] = v.tolist()
         return d
 
-    def _compute(self, *, predictions, references, **kwargs):
-        """Returns the scores"""
-        return self.coco_metric.compute()["metrics"]
-
-    def add_from_payload(self, payload: Payload, model_name: str = None):
-        """Converts the payload to the format expected by the metric"""
-        predictions, references = payload_to_det_metric(payload, model_name)
-        self.add(prediction=predictions, reference=references)
-        return self
+    def _lists_to_np(self, d):
+        """datasets does not support numpy arrays for type checking"""
+        for k, v in d.items():
+            if isinstance(v, dict):
+                self._lists_to_np(v)
+            elif isinstance(v, list):
+                d[k] = np.array(v)
+        return d