Add priorities to job queue and user quota system

[JulianFP]

Currently we already have an addressable priority queue in place for the jobs that are waiting to be processed (and after the rework this will still be the case but in redis using redis sorted sets datastructure), however we currently set the same priority for all jobs making it effectively a FIFO queue. This needs to change for the following reasons:

1. It doesn't care about who submitted the jobs, its first-come-first-served. If a user submits 100 jobs all at once everybody has to wait for all those 100 jobs to complete before any of their jobs will be processed
2. It doesn't care about how long the jobs are: Very short jobs that would take seconds to finish have to wait for multi-hour jobs to complete just because they were submitted a little bit earlier
3. It doesn't care about how much processing time on the runners a user already used in the past

Ideas of how to tackle this:

• The priority is some static value minus the amount of jobs a user has already sitting in the queue. This would solve problem 1, but doesn't solve problem 2 and 3. Moreover this would create an environment where a user is possibly at an advantage when they create only one job at a time, wait for its completion, and only then submit the next job. I conclude that the priority needs to take previous already completed jobs into account, not just pending jobs
• Jobs already have timestamps attached to them (after the rework), so just use them and set the priority to some static value minus the amount of jobs the user submitted in last 24 hours or so (including both pending and completed jobs). This solves problem 1 and somewhat problem 3, but not problem 2, as this system would treat users who submit only very short jobs and users who submit many hour long audio files the same. This creates an environment where merging many audio files into one job would put users into a big advantage, even though for the service provider multiple smaller jobs would probably be nicer. Also how would we treat aborted and failed jobs?
• Track the computation time that a job needs, multiply that by the runner priority (which is some indicator for the runners hardware capability if used correctly by the administrator) and we get a nice metric for processing resources that a particular user has used. Store that in the database and calculate the priority of jobs by some static value minus this indicator for the past 24 hours or past week or whatever. This solves all three problems plus also allows for another feature: user quotas. The user quota would be a hard limit of processing resources a particular user could use within some time frame, set by the administrator if desired. This solution is the best of the three, there are some remaining problems with it though:
  • Pending jobs should also be included in the priority of jobs, otherwise problem 1 wouldn't really be solved (it would be the reverse of the first solution: Only taking finished but not pending jobs into account). For this we will need to estimate the processing resources a job takes before it's being executed, i.e. be using the size of the audio file.
  • How does the "for the past 24 hours" work? Do we just reset the used-up quota in the database periodically to zero? If yes we would need to run some periodic quota cleanup job

Note: This is definitely after-rework stuff. Just had a lot of thoughts about this and wanted to write it down somewhere.