SPEED: Polling intervals should be reset when job is launched #63
Description
Background
When requesting the value of a futures that is not ready, it will poll for it being resolved using a waiting scheme that waits a certain time before checking again. This waiting interval increases with a small scale factor over time. Thus, earlier on it checks much more frequently than later on. The idea for this strategy is that the number of polls for all futures will be on roughly the same order of magnitude.
Issue
Now, when BatchJobs are submitted to the cluster via a job scheduler they're first queued. If the cluster is very busy, then the future may not start in a long time. However, regardless of the queuing time, our ad hoc waiting scheme will still slowly increase the time interval between polls. This is independent of the actual processing time. When the future then starts, it may very well be that we wait a very long before we check the results - even when the future process completes in seconds.
Suggestion
If we can detect when the BatchJobs transistions from being queued to being processed, the we could reset the waiting scheme back to its original time.