Merge branch 'feature/confidence-curve'

README.md

@@ -28,7 +28,7 @@ First, ensure you have Python 3.8 or later installed. Then, install det-metrics
 pip install evaluate git+https://github.com/SEA-AI/seametrics@develop
 ```
 
-### Basic Usage
+## Basic Usage
 
 Here's how to quickly evaluate your object detection models using SEA-AI/det-metrics:
 
@@ -60,22 +60,23 @@ results = module.compute()
 print(results)
 ```
 
-This will output the evaluation metrics for your detection model.
-```
-{'all': {'range': [0, 10000000000.0],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 2,
-  'fp': 0,
-  'fn': 0,
-  'duplicates': 0,
-  'precision': 1.0,
-  'recall': 1.0,
-  'f1': 1.0,
-  'support': 2,
-  'fpi': 0,
-  'nImgs': 1}
+This will output the following dictionary containing metrics for the detection model. The key of the dictionary will be the model name or "custom" if no model names are available like in this case.
+
+```json
+{
+    "custom": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    }
+}
 ```
+- `metrics`: A dictionary containing performance metrics for each area range
+- `eval`: Output of COCOeval.accumulate()
+- `params`: COCOeval parameters object
+
+See [Output Values](#output-values) for more detailed information about the returned results structure, which includes metrics, eval, and params fields for each model passed as input.
+
 
 ## FiftyOne Integration
 
@@ -92,35 +93,40 @@ logging.basicConfig(level=logging.WARNING)
 processor = PayloadProcessor(
     dataset_name="SAILING_DATASET_QA",
     gt_field="ground_truth_det",
-    models=["yolov5n6_RGB_D2304-v1_9C"],
+    models=["yolov5n6_RGB_D2304-v1_9C", "tf1zoo_ssd-mobilenet-v2_agnostic_D2207"],
     sequence_list=["Trip_14_Seq_1"],
     data_type="rgb",
 )
 
 # Evaluate using SEA-AI/det-metrics
-module = evaluate.load("SEA-AI/det-metrics")
-module.add_payload(processor.payload)
+module = evaluate.load("SEA-AI/det-metrics", payload=processor.payload)
 results = module.compute()
 
 print(results)
 ```
-
-```console
-{'all': {'range': [0, 10000000000.0],
-  'iouThr': '0.00',
-  'maxDets': 100,
-  'tp': 89,
-  'fp': 13,
-  'fn': 15,
-  'duplicates': 1,
-  'precision': 0.8725490196078431,
-  'recall': 0.8557692307692307,
-  'f1': 0.8640776699029126,
-  'support': 104,
-  'fpi': 0,
-  'nImgs': 22}}
+This will output the following dictionary containing metrics for the detection model. The key of the dictionary will be the model name. 
+```json
+{
+    "yolov5n6_RGB_D2304-v1_9C": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    },
+    "tf1zoo_ssd-mobilenet-v2_agnostic_D2207": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    }
+}
 ```
 
+- `metrics`: A dictionary containing performance metrics for each area range
+- `eval`: Output of COCOeval.accumulate()
+- `params`: COCOeval parameters object
+
+See [Output Values](#output-values) for more detailed information about the returned results structure, which includes metrics, eval, and params fields for each model passed as input.
+
+
 ## Metric Settings
 
 Customize your evaluation by specifying various parameters when loading SEA-AI/det-metrics:
@@ -146,8 +152,32 @@ module = evaluate.load(
 ```
 
 ## Output Values
+For every model passed as input, the results contain the metrics, eval, and params fields. If no specific model was passed (usage without payload), the default model name “custom” will be used.
+
+```json
+{
+    "model_1": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    },
+    "model_2": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    },
+    "model_3": {
+        "metrics": ...,
+        "eval": ...,
+        "params": ...
+    },
+    ...
+}
+```
 
-SEA-AI/det-metrics provides a detailed breakdown of performance metrics for each specified area range:
+### Metrics
+
+SEA-AI/det-metrics metrics dictionary provides a detailed breakdown of performance metrics for each specified area range:
 
 - **range**: The area range considered.
 - **iouThr**: The IOU threshold applied.
@@ -159,6 +189,106 @@ SEA-AI/det-metrics provides a detailed breakdown of performance metrics for each
 - **fpi**: Number of images with predictions but no ground truths.
 - **nImgs**: Total number of images evaluated.
 
+### Eval
+
+The SEA-AI/det-metrics evaluation dictionary provides details about evaluation metrics and results. Below is a description of each field:
+
+- **params**: Parameters used for evaluation, defining settings and conditions.
+  
+- **counts**: Dimensions of parameters used in evaluation, represented as a list [T, R, K, A, M]:
+  - T: IoU threshold (default: [1e-10])
+  - R: Recall threshold (not used)
+  - K: Class index (class-agnostic, so only 0)
+  - A: Area range (0=all, 1=valid_n, 2=valid_w, 3=tiny, 4=small, 5=medium, 6=large)
+  - M: Max detections (default: [100])
+  
+- **date**: The date when the evaluation was performed.
+  
+- **precision**: A multi-dimensional array [TxRxKxAxM] storing precision values for each evaluation setting.
+  
+- **recall**: A multi-dimensional array [TxKxAxM] storing maximum recall values for each evaluation setting.
+  
+- **scores**: Scores for each detection.
+  
+- **TP**: True Positives - correct detections matching ground truth.
+  
+- **FP**: False Positives - incorrect detections not matching ground truth.
+  
+- **FN**: False Negatives - ground truth objects not detected.
+  
+- **duplicates**: Duplicate detections of the same object.
+  
+- **support**: Number of ground truth objects for each category.
+  
+- **FPI**: False Positives per Image.
+  
+- **TPC**: True Positives per Category.
+  
+- **FPC**: False Positives per Category.
+  
+- **sorted_conf**: Confidence scores of detections sorted in descending order.
+
+> Note:
+> **precision** and **recall** are set to -1 for settings with no ground truth objects.
+
+### Params
+
+The params return value of the COCO evaluation parameters in PyCOCO represents a dictionary with various evaluation settings that can be customized. Here’s a breakdown of what each parameter means:
+
+- **imgIds**: List of image IDs to use for evaluation. By default, it evaluates on all images.
+- **catIds**: List of category IDs to use for evaluation. By default, it evaluates on all categories.
+- **iouThrs**: List of IoU (Intersection over Union) thresholds for evaluation. By default, it uses thresholds from 0.5 to 0.95 with a step of 0.05 (i.e., [0.5, 0.55, …, 0.95]).
+- **recThrs**: List of recall thresholds for evaluation. By default, it uses 101 thresholds from 0 to 1 with a step of 0.01 (i.e., [0, 0.01, …, 1]).
+- **areaRng**: Object area ranges for evaluation. This parameter defines the sizes of objects to evaluate. It is specified as a list of tuples, where each tuple represents a range of area in square pixels.
+- **maxDets**: List of thresholds on maximum detections per image for evaluation. By default, it evaluates with thresholds of 1, 10, and 100 detections per image.
+- **iouType**: Type of IoU calculation used for evaluation. It can be ‘segm’ (segmentation), ‘bbox’ (bounding box), or ‘keypoints’.
+- **useCats**: Boolean flag indicating whether to use category labels for evaluation (default is 1, meaning true).
+
+> Note:
+> If useCats=0 category labels are ignored as in proposal scoring.
+> Multiple areaRngs [Ax2] and maxDets [Mx1] can be specified.
+
+## Confidence Curves
+When you use **module.generate_confidence_curves()**, it creates a graph that shows how metrics like **precision, recall, and f1 score** change as you adjust confidence thresholds. This helps you see the trade-offs between precision (how accurate positive predictions are) and recall (how well the model finds all positive instances) at different confidence levels. As confidence scores go up, models usually have higher precision but may find fewer positive instances, reflecting their certainty in making correct predictions.
+
+#### Confidence Config
+The `confidence_config` dictionary is set as `{"T": 0, "R": 0, "K": 0, "A": 0, "M": 0}`, where:
+- `T = 0`: represents the IoU (Intersection over Union) threshold.
+- `R = 0`: is the recall threshold, although it's currently not used.
+- `K = 0`: indicates a class index for class-agnostic mean Average Precision (mAP), with only one class indexed at 0.
+- `A = 0`: signifies that all object sizes are considered for evaluation. (0=all, 1=small, 2=medium, 3=large, ... depending on area ranges) 
+- `M = 0`: sets the default maximum detections (`maxDets`) to 100 in precision_recall_f1_support calculations.
+
+
+```python
+import evaluate
+import logging
+from seametrics.payload.processor import PayloadProcessor
+
+logging.basicConfig(level=logging.WARNING)
+
+# Configure your dataset and model details
+processor = PayloadProcessor(
+    dataset_name="SAILING_DATASET_QA",
+    gt_field="ground_truth_det",
+    models=["yolov5n6_RGB_D2304-v1_9C"],
+    sequence_list=["Trip_14_Seq_1"],
+    data_type="rgb",
+)
+
+# Evaluate using SEA-AI/det-metrics
+module = evaluate.load("SEA-AI/det-metrics", payload=processor.payload)
+results = module.compute()
+
+# Plot confidence curves
+confidence_config={"T": 0, "R": 0, "K": 0, "A": 0, "M": 0}
+fig = module.generate_confidence_curves(results, confidence_config)
+fig.show()
+```
+
+![Alt text](assets/example_confidence_curves.png)
+
+
 ## Further References
 
 - **seametrics Library**: Explore the [seametrics GitHub repository](https://github.com/SEA-AI/seametrics/tree/main) for more details on the underlying library.

det-metrics.py

@@ -13,14 +13,14 @@
 # limitations under the License.
 """TODO: Add a description here."""
 
-from typing import List, Tuple, Literal
-from deprecated import deprecated
+from typing import List, Literal, Tuple
 
-import evaluate
 import datasets
+import evaluate
 import numpy as np
-
+from deprecated import deprecated
 from seametrics.detection import PrecisionRecallF1Support
+from seametrics.detection.utils import payload_to_det_metric
 from seametrics.payload import Payload
 
 _CITATION = """\
@@ -92,7 +92,7 @@ Examples:
     >>> from seametrics.payload.processor import PayloadProcessor
     >>> payload = PayloadProcessor(...).payload
     >>> module = evaluate.load("SEA-AI/det-metrics", ...)
-    >>> module.add_payload(payload)
+    >>> module._add_payload(payload)
     >>> result = module.compute()
     >>> print(result)
         {'all': {
@@ -123,20 +123,36 @@ class DetectionMetric(evaluate.Metric):
         class_agnostic: bool = True,
         bbox_format: str = "xywh",
         iou_type: Literal["bbox", "segm"] = "bbox",
-        **kwargs
+        payload: Payload = None,
+        **kwargs,
     ):
         super().__init__(**kwargs)
+
+        # save parameters for later
+        self.payload = payload
+        self.model_names = payload.models if payload else ["custom"]
+        self.iou_thresholds = (
+            iou_threshold if isinstance(iou_threshold, list) else [iou_threshold]
+        )
+        self.area_ranges = [v for _, v in area_ranges_tuples]
+        self.area_ranges_labels = [k for k, _ in area_ranges_tuples]
+        self.class_agnostic = class_agnostic
+        self.iou_type = iou_type
+        self.box_format = bbox_format
+
+        # initialize coco_metrics
         self.coco_metric = PrecisionRecallF1Support(
-            iou_thresholds=(
-                iou_threshold if isinstance(iou_threshold, list) else [iou_threshold]
-            ),
-            area_ranges=[v for _, v in area_ranges_tuples],
-            area_ranges_labels=[k for k, _ in area_ranges_tuples],
-            class_agnostic=class_agnostic,
-            iou_type=iou_type,
-            box_format=bbox_format,
+            iou_thresholds=self.iou_thresholds,
+            area_ranges=self.area_ranges,
+            area_ranges_labels=self.area_ranges_labels,
+            class_agnostic=self.class_agnostic,
+            iou_type=self.iou_type,
+            box_format=self.box_format,
         )
 
+        # initialize evaluation metric
+        self._init_evaluation_metric()
+
     def _info(self):
         return evaluate.MetricInfo(
             # This is the description that will appear on the modules page.
@@ -186,29 +202,63 @@ class DetectionMetric(evaluate.Metric):
 
         self.coco_metric.update(prediction, reference)
 
+    def _init_evaluation_metric(self, **kwargs):
+        """
+        Initializes the evaluation metric by generating sample data, preprocessing predictions and references,
+        and then adding the processed data to the metric using the super class method with additional keyword arguments.
+
+        Parameters:
+            **kwargs: Additional keyword arguments for the super class method.
+
+        Returns:
+            None
+        """
+        predictions, references = self._generate_sample_data()
+        predictions = self._preprocess(predictions)
+        references = self._preprocess(references)
+
         # does not impact the metric, but is required for the interface x_x
         super(evaluate.Metric, self).add(
-            prediction=self._postprocess(prediction),
-            references=self._postprocess(reference),
-            **kwargs
+            prediction=self._postprocess(predictions),
+            references=self._postprocess(references),
+            **kwargs,
         )
 
-    @deprecated(reason="Use `module.add_payload` instead")
+    @deprecated(reason="Use `module._add_payload` instead")
     def add_batch(self, payload: Payload, model_name: str = None):
         """Takes as input a payload and adds the batch to the metric"""
-        self.add_payload(payload, model_name)
+        self._add_payload(payload, model_name)
 
     def _compute(self, *, predictions, references, **kwargs):
         """Called within the evaluate.Metric.compute() method"""
-        return self.coco_metric.compute()["metrics"]
 
-    def add_payload(self, payload: Payload, model_name: str = None):
+        results = {}
+        for model_name in self.model_names:
+            print(f"\n##### {model_name} #####")
+            # add payload if available (otherwise predictions and references must be added with add function)
+            if self.payload:
+                self._add_payload(self.payload, model_name)
+
+            results[model_name] = self.coco_metric.compute()
+
+            # reset coco_metrics for next model
+            self.coco_metric = PrecisionRecallF1Support(
+                iou_thresholds=self.iou_thresholds,
+                area_ranges=self.area_ranges,
+                area_ranges_labels=self.area_ranges_labels,
+                class_agnostic=self.class_agnostic,
+                iou_type=self.iou_type,
+                box_format=self.box_format,
+            )
+        return results
+
+    def _add_payload(self, payload: Payload, model_name: str = None):
         """Converts the payload to the format expected by the metric"""
         # import only if needed since fiftyone is not a direct dependency
-        from seametrics.detection.utils import payload_to_det_metric
 
         predictions, references = payload_to_det_metric(payload, model_name)
         self.add(prediction=predictions, reference=references)
+
         return self
 
     def _preprocess(self, list_of_dicts):
@@ -236,3 +286,79 @@ class DetectionMetric(evaluate.Metric):
             elif isinstance(v, list):
                 d[k] = np.array(v)
         return d
+
+    def generate_confidence_curves(
+        self, results, confidence_config={"T": 0, "R": 0, "K": 0, "A": 0, "M": 0}
+    ):
+        """
+        Generate confidence curves based on results and confidence configuration.
+
+        Parameters:
+            results (dict): Results of the evaluation for different models.
+            confidence_config (dict): Configuration for confidence values. Defaults to {"T": 0, "R": 0, "K": 0, "A": 0, "M": 0}.
+                                      T: [1e-10] iou threshold
+                                      R: recall threshold (not used)
+                                      K: class index (class-agnostic mAP, so only 0)
+                                      A: 0=all, 1=small, 2=medium, 3=large, ... (depending on area ranges)
+                                      M: [100] maxDets default in precision_recall_f1_support
+
+        Returns:
+            fig (plotly.graph_objects.Figure): The plotly figure showing the confidence curves.
+        """
+        import plotly.graph_objects as go
+        from seametrics.detection.utils import get_confidence_metric_vals
+
+        # Create traces
+        fig = go.Figure()
+        metrics = ["precision", "recall", "f1"]
+        for model_name in self.model_names:
+            print(f"##### {model_name} #####")
+            plot_data = get_confidence_metric_vals(
+                cocoeval=results[model_name]["eval"],
+                T=confidence_config["T"],
+                R=confidence_config["R"],
+                K=confidence_config["K"],
+                A=confidence_config["A"],
+                M=confidence_config["M"],
+            )
+
+            for metric in metrics:
+                fig.add_trace(
+                    go.Scatter(
+                        x=plot_data["conf"],
+                        y=plot_data[metric],
+                        mode="lines",
+                        name=f"{model_name} {metric}",
+                        line=dict(dash=None if metric == "f1" else "dash"),
+                    )
+                )
+
+        fig.update_layout(
+            title="Metric vs Confidence",
+            hovermode="x unified",
+            xaxis_title="Confidence",
+            yaxis_title="Metric value",
+        )
+        return fig
+
+    def _generate_sample_data(self):
+        """
+        Generates dummy sample data for predictions and references used for initialization.
+
+        Returns:
+            Tuple[List[Dict[str, List[Union[float, int]]]], List[Dict[str, List[Union[float, int]]]]]:
+                - predictions (List[Dict[str, List[Union[float, int]]]]): A list of dictionaries representing the predictions. Each dictionary contains the following keys:
+                    - boxes (List[List[float]]): A list of bounding boxes in the format [x, y, w, h].
+                    - labels (List[int]): A list of labels.
+                    - scores (List[float]): A list of scores.
+                - references (List[Dict[str, List[Union[float, int]]]]): A list of dictionaries representing the references. Each dictionary contains the following keys:
+                    - boxes (List[List[float]]): A list of bounding boxes in the format [x, y, w, h].
+                    - labels (List[int]): A list of labels.
+                    - area (List[float]): A list of areas.
+        """
+        predictions = [
+            {"boxes": [[1.0, 2.0, 3.0, 4.0]], "labels": [0], "scores": [1.0]}
+        ]
+        references = [{"boxes": [[1.0, 2.0, 3.0, 4.0]], "labels": [0], "area": [1.0]}]
+
+        return predictions, references

requirements.txt

@@ -1,3 +1,4 @@
 git+https://github.com/huggingface/evaluate@main
 git+https://github.com/SEA-AI/seametrics@develop
-fiftyone
+fiftyone
+plotly