Parse-evaluator doesn't ignore reference parses with no links

[glicerico]

Currently, the main parse-evaluator doesn't ignore reference sentences with no links when calculating the overall performance metrics (precision, recall, F1).
Given this situation, if a reference parse comes with no links (because the gold standard didn't parse it, or all its links are ignored after using -i option) then this sentence will get a 0 score and be averaged with the rest of the sentences. In my parse-evaluator (-a version), reference sentences with no links left are ignored, instead. I think this is fair because there is no way to get a score above 0 if there are no reference links.

My question is: is this a bug or a feature? I think we should ignore reference sentences with no links.

[glicerico]

If you want to see an example of this, check https://github.com/glicerico/stream-parser/blob/pipeline/data/sample/parses/sample_corpus2.txt.ull
The first sentence comes intentionally with no links, and when the parses are evaluated against itself (scores = 1 expected), the sentence gets a 0 score, which gets averaged with the rest of the corpus (5 sentences) giving a 0.8 score: https://github.com/glicerico/stream-parser/blob/pipeline/data/sample/parses.stat

[akolonin]

I think that:

1. Having sentences with no link in reference corpus makes no sense and points to bugs in reference corpus, it can not be called "gold" at all in such case.
2. Ignoring sentences in reference corpus on basis like missed links appears weird because what if you have a sentence with partial links? Will you exclude it or not? What if you have sentence with 10 words and only one link? Will you keep it in the reference corpus? If you keep it then why do you drop the one which has no links at all?
3. Given the above, dropping the sentences in the reference corpus, if implemented, should be done in case if at least one word is not linked, so the linkage is incomplete. But better use the latest filtering option like implemented by @alexei-gl - like indicate number of words in a reference corpus sentence that may be not linked. Then, if you have the option = 0, then all words in a sentence have to be linked otherwise the sentence have to be dropped.
4. The sooner we eliminate the difference between "with -a option" and "with no -a option", the better, so if you @glicerico could make the code straight for that and keep unittests passing that would be awesome.

[glicerico]

Thanks for the comments @akolonin . My reply:

1. Imagine this type of case, which actually occurs in some of the project's corpora:

Welcome !
0 ###LEFT-WALL### 1 Welcome
0 ###LEFT-WALL### 2 !

When using the -i option, both links will be ignored by the evaluator, leaving a reference parse with no links and thus automatically a 0 score. I think this kind of sentences should be ignored for evaluation, and the parse-evaluator should handle them.

2 + 3) I'm not saying to ignore sentences because of missing links. Only when ALL links in reference sentence are missing or ignored, only because otherwise we get an automatic 0 score for it, even if no actual evaluation happened.

4. I agree, but I also don't think it's high priority now :)

[akolonin]

1-2-3) I agree that we need to ignore such situations but "ALL links" can not be defined because you don't know what "ALL" means. So ideally I would have two options to ignore sentences:

• default - ignore all sentences which have no linked words at all
• strict - ignore all sentences which have at least one word not linked

Still, if the behaviour above can be configured with latest options implemented by @alexei-gl , I would suggest re-use them and do not overcomplicate design. If it can't, then it seems ok to add the extra "ignorance" logic.

[glicerico]

I don't understand why we can't know what "ALL" means. If, while processing a reference parse, we realize the set of links we will evaluate is empty, we skip the sentence. This is what my parse-evaluator does.
Your strict option is talking about a different problem, not the one I raised here.

[alexei-gl]

@akolonin , @glicerico , I think in case reference parse sentence is missing a link (no matter weather it originaly had no links or it is missing a link because of the switched on options such as filtering LEFT-WALL) exception should be raised because without a link it can no longer be a reference and further comparison makes no sense.
On the other hand we do use them in all cases, except for gold standard parses... :-(

[alexei-gl]

@akolonin , @glicerico , strict option -T in current parse-evaluator implementation (no option -a enabled) checks tokenization for exact match. It does not check reference sentences to have all tokens linked.