Evaluation Clarification

[tyler-hayes]

Hello, thank you for your really interesting work!

We wanted to clarify the evaluation procedure from the paper. Could you indicate if the following interpretation is correct or incorrect? If it is incorrect, any clarifications would be greatly appreciated! :)

For an IoU threshold:

Stage 1:

1. For each image in the test set, compute RPN boxes and keep only RPN boxes that overlap with ground truth (GT)
2. For each remaining RPN box in an image, compute its novel class ID and mark down the novel class ID and the GT class label
3. Run Hungarian Matching algorithm to compute mapping between novel class IDs and GT class labels

Stage 2:

1. For all RPN boxes in the test set (i.e., not removing RPN boxes without overlap with ground truth), compute novel class ID and assign class label from mapping from Stage 1
2. Compute mAP score on boxes with class labels

[vlfom]

Dear Tyler,

Thank you for the interest in our work!

Your Stage 2 understanding is correct. For Stage 1, however, it's wrong. There, we do not use RPN boxes, but only GT boxes and their class predictions. We then run Hunagian Matching on top of GT boxes with generated class assignments.

While your suggested approach is also sound and would be more resemblant of real-world evaluation, in my opinion, we tried to follow existing works [1], including mimicking NCD setup, e.g. [2], as close as possible.

[tyler-hayes]

Thanks for the clarification!

One additional question: for use of the Hungarian Algorithm (HA) with a different number of clusters than classes, does the RNCDL method use dummy variables to create a square prediction matrix for HA evaluation?

[vlfom]

Yes, we create additional vertices, when necessary, ultimately just keeping the matched pairs that contain GT classes.

…

On Tue, Apr 4, 2023 at 7:48 AM Tyler Hayes ***@***.***> wrote: Thanks for the clarification! One additional question: for use of the Hungarian Algorithm (HA) with a different number of clusters than classes, does the RNCDL method use dummy variables to create a square prediction matrix for HA evaluation? — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGQXRBXO2Z6NZBPG5NGP23W7QYCFANCNFSM6AAAAAAUVKJVXU> . You are receiving this because you commented.Message ID: ***@***.***>

[tyler-hayes]

Hello again, I had another clarification question. Regarding the matching step of GT boxes to novel class IDs, is this step run on the GT test boxes or the GT train boxes?
Thank you in advance!