README.md

---
app_file: app.py
colorFrom: yellow
colorTo: green
description: 'TODO: add a description here'
emoji: 🤑
pinned: false
runme:
  id: 01HPS3ASFJXVQR88985QNSXVN1
  version: v3
sdk: gradio
sdk_version: 4.36.0
tags:
- evaluate
- metric
title: user-friendly-metrics
---

# How to Use

```python {"id":"01HPS3ASFHPCECERTYN7Z4Z7MN"}
import evaluate
from seametrics.payload.processor import PayloadProcessor

payload = PayloadProcessor(
    dataset_name="SENTRY_VIDEOS_DATASET_QA",
    gt_field="ground_truth_det_fused_id",
    models=["ahoy_IR_b2_engine_3_7_0_757_g8765b007_oversea"],
    sequence_list=["Sentry_2023_02_08_PROACT_CELADON_@6m_MOB_2023_02_08_14_41_51"],
    # tags=["GT_ID_FUSION"],
    tracking_mode=True
).payload

module = evaluate.load("SEA-AI/user-friendly-metrics")
res = module._compute(payload, max_iou=0.5, recognition_thresholds=[0.3, 0.5, 0.8])
print(res)
```

```json
{
    "global": {
        "ahoy_IR_b2_engine_3_6_0_49_gd81d3b63_oversea": {
            "all": {
                "f1": 0.15967351103175614,
                "fn": 2923.0,
                "fp": 3666.0,
                "num_gt_ids": 10,
                "precision": 0.14585274930102515,
                "recall": 0.1763877148492533,
                "recognition_0.3": 0.1,
                "recognition_0.5": 0.1,
                "recognition_0.8": 0.1,
                "recognized_0.3": 1,
                "recognized_0.5": 1,
                "recognized_0.8": 1,
                "tp": 626.0
            }
        }
    },
    "per_sequence": {
        "Sentry_2023_02_08_PROACT_CELADON_@6m_MOB_2023_02_08_12_51_49": {
            "ahoy_IR_b2_engine_3_6_0_49_gd81d3b63_oversea": {
                "all": {
                    "f1": 0.15967351103175614,
                    "fn": 2923.0,
                    "fp": 3666.0,
                    "num_gt_ids": 10,
                    "precision": 0.14585274930102515,
                    "recall": 0.1763877148492533,
                    "recognition_0.3": 0.1,
                    "recognition_0.5": 0.1,
                    "recognition_0.8": 0.1,
                    "recognized_0.3": 1,
                    "recognized_0.5": 1,
                    "recognized_0.8": 1,
                    "tp": 626.0
                }
            }
        }
    }
}
```

## Metric Settings

The `max_iou` parameter is used to filter out the bounding boxes with IOU less than the threshold. The default value is 0.5. This means that if a ground truth and a predicted bounding boxes IoU value is less than 0.5, then the predicted bounding box is not considered for association. So, the higher the `max_iou` value, the more the predicted bounding boxes are considered for association.

## Output

The output is a dictionary containing the following metrics:

| Name                 | Description                                                                        |
| :------------------- | :--------------------------------------------------------------------------------- |
| recall               | Number of detections over number of objects.                                       |
| precision            | Number of detected objects over sum of detected and false positives.               |
| f1   | F1 score                                     |
| num_gt_ids       | Number of unique objects on the ground truth                     |
| fn    | Number of false negatives                  |
| fp          | Number of of false postives                        |
| tp  | number of true positives                                    |
| recognized_th           | Total number of unique objects on the ground truth that were seen more then th% of the times |
| recognition_th           | Total number of unique objects on the ground truth that were seen more then th% of the times over the number of unique objects on the ground truth|

## How it Works

We levereage one of the internal variables of motmetrics ```MOTAccumulator``` class, ```events```, which keeps track of the detections hits and misses. These values are then processed via the ```track_ratios``` function which counts the ratio of assigned to total appearance count per unique object id. We then define the ```recognition``` function that counts how many objects have been seen more times then the desired threshold.




## W&B logging
When you use **module.wandb()**, it is possible to log the User Frindly metrics values in Weights and Bias (W&B). The W&B key is stored as a Secret in this repository.

### Params
- **wandb_project** - Name of the W&B project (Default: `'user_freindly_metrics'`)
- **log_plots** (bool, optional): Generates categorized bar charts for global metrics. Defaults to True
- **debug** (bool, optional): Logs everything to the console and w&b Logs page. Defaults to False
  
```python
import evaluate
import logging
from seametrics.payload.processor import PayloadProcessor

logging.basicConfig(level=logging.WARNING)

# Configure your dataset and model details
payload = PayloadProcessor(
    dataset_name="SENTRY_VIDEOS_DATASET_QA",
    gt_field="ground_truth_det_fused_id",
    models=["ahoy_IR_b2_engine_3_7_0_757_g8765b007_oversea"],
    sequence_list=["Sentry_2023_02_08_PROACT_CELADON_@6m_MOB_2023_02_08_14_41_51"],
    tracking_mode=True
).payload


# Evaluate using SEA-AI/user-friendly-metrics
module = evaluate.load("SEA-AI/user-friendly-metrics")
res = module._compute(payload, max_iou=0.5, recognition_thresholds=[0.3, 0.5, 0.8])

module.wandb(res,log_plots=True, debug=True)
```

- If `log_plots` is `True`, the W&B logging function generates four bar plots:  
  - **User_Friendly Metrics (mostly_tracked_score_%)** mainly for non dev users  
  - **User_Friendly Metrics (mostly_tracked_count_%)** for dev
  - **Evaluation Metrics** (F1, precision, recall)  
  - **Prediction Summary** (false negatives, false positives, true positives)
   
- If `debug` is `True`, the function logs the global metrics plus the per-sequence evaluation metrics in descending order of F1 score under the **Logs** section of the run page.

- If both `log_plots` and `debug` are `False`, the function logs the metrics to the **Summary**.  

![image/png](https://cdn-uploads.huggingface.co/production/uploads/65ca2aafdc38a2858aa43f1e/RYEsFwt6K-jP0mp7_RIZv.png)


## Citations

```bibtex {"id":"01HPS3ASFJXVQR88985GKHAQRE"}
@InProceedings{huggingface:module,
title = {A great new module},
authors={huggingface, Inc.},
year={2020}}
```

```bibtex {"id":"01HPS3ASFJXVQR88985KRT478N"}
@article{milan2016mot16,
title={MOT16: A benchmark for multi-object tracking},
author={Milan, Anton and Leal-Taix{\'e}, Laura and Reid, Ian and Roth, Stefan and Schindler, Konrad},
journal={arXiv preprint arXiv:1603.00831},
year={2016}}
```

## Further References

- [Github Repository - py-motmetrics](https://github.com/cheind/py-motmetrics/tree/develop)