Support for LIFO / FIFO setting on judgement queue

[sauterl]

In some cases, it would be desirable for evaluation organisers to influence how answers are judged, e.g. whether simply the new submissions are added to the end of the queue or to the front (i.e. judged earlier).
This could be a setting on the task run.

[sauterl]

Hint: We will include this as a parameter for judgement-target typed tasks.

[sauterl]

We have implemented this for the time being in such a way as that the JUDGEMENT or VOTE target options now come with a configuration key order, which accepts LIFO (case insensitive) to produce a LIFO queue of judgements.
However, this is per task and the dequeue endpoints still operate on the first open validator.

An additional desirable outcome could be that always the newest (based on time) should be judged in the UI.
I am not sure on how we would achieve this with the current architecture, as we would have to not only heavily alter the implemented JudgementValidators, but also track across multiple ones, which submission came in last.

@lucaro any thoughts on this one?

[lucaro]

I'd argue that rather than (or in addition to?) changing the insertion order per validator, it is probably more useful to give a priority flag to the validators. This could be any number; the higher, the more important. The mechanism that selects the validator would then not just use the least recent, but the least recent with the highest number.

[sauterl]

Fair enough, I implemnted this with e033a43 by adding the configuration parameter JUDGEMENT.priority which expects an integer. Higher value represents higher priority.

Evaluation administrators have to set the priority on each task template.

[lucaro]

Do they have to set the priority, or can they set the priority? I guess there should be a reasonable default, no?

[sauterl]

Let's adjust this once more:

The default is the index in the task list (as opposed to the 0 currently implemented).

However, I'd suggest that then a run parameter could influence whether higher priority value results in being considered first or last.
Do you agree?