Beyond mAP: Towards better evaluation of instance segmentation

Introduction

Correctness of instance segmentation constitutes:

Counting the number of objects.
Localizing all predictions.
Classifying each localized prediction.

We wanted to verify how good mAP is at evaluating these three tasks. We analyse the mAP (and the underlying precision-recall curves) of state-of-the-art models on the COCO dataset.

Problem statement

An initial analysis shows that state-of-the-art models achieve significant gains in AP, but produce qualitatively unsatisfactory results with a lot of duplicates. A deeper dive shows that these predictions are not penalized by AP, and rather, are rewarded by the metric. We call these low-confidence false positives hedged predictions because they are hedged/predicted to improve mAP, but clutter the segmentation result.

AP curves for person class for (a) MaskNMS , (b) SoftNMS, and (c) MatrixNMS shows the tremendous number of false-positives that are introduced (upto 4x compared to MaskNMS), but virtually have the same AP.

Solution

We propose a three-fold contribution:

Demonstrate the shortcoming of AP to quantify low-confidence false positives leading to hedging.
Propose existing (F1, LRP) and new measures (DC, NE) to quantify the amount of both spatial and categorical hedging.
Propose Semantic Sorting and Semantic NMS to mitigate both forms of hedging while preserving mAP.

The underlying model for the left and right images is the same. We utilize Semantic Sorting to re-rank the predictions.
Then, the Semantic NMS ensures that instances are preserved based on occupancy of the Semantic Mask.
This prevents spatial hedging because heavily overlapping instances cannot claim the same pixels in the semantic mask.
This prevents categorical hedging because Semantic Mask at a region does not predict multiple classes.

Please read the paper for more exciting details!

Paper Abstract

Correctness of instance segmentation constitutes counting the number of objects, correctly localizing all predictions and classifying each localized prediction. Average Precision is the de-facto metric used to measure all these constituents of segmentation. However, this metric does not penalize duplicate predictions in the high-recall range, and cannot distinguish instances that are localized correctly but categorized incorrectly. This weakness has inadvertently led to network designs that achieve significant gains in AP but also introduce a large number of false positives. We therefore cannot rely on AP to choose a model that provides an optimal tradeoff between false positives and high recall. To resolve this dilemma, we review alternative metrics in the literature and propose two new measures to explicitly measure the amount of both spatial and categorical duplicate predictions. We also propose a Semantic Sorting and NMS module to remove these duplicates based on a pixel occupancy matching scheme. Experiments show that modern segmentation networks have significant gains in AP, but also contain a considerable amount of duplicates. Our Semantic Sorting and NMS can be added as a plug-and-play module to mitigate hedged predictions and preserve AP.

BibTeX

@article{jena2023beyond,
  author    = {Jena, Rohit and Zhornyak, Lukas and Doiphode, Nehal and Chaudhari, Pratik and Buch, Vivek and Gee, James and Shi, Jianbo},
  title     = {Beyond mAP: Towards better evaluation of instance segmentation},
  journal   = {CVPR},
  year      = {2023},
}