Runner: Proper error handling #93
Description
Currently the runner will just shut down gracefully if it encounters any http status code other than 200 when talking to the backend. We expect the runner to be set up in a way so that it automatically restarts in such a case, however there is still a number of reasons why this needs to be refined:
- The runner should not do this when encountering a status code other than 200 in the 200-299 range. None of these are returned explicitly by our backend server but flask or any other webserver/proxy/mirror/... might return these (e.g. status code 203)
- Some of these errors might not be critical and the runner could be able to continue if the error is properly handled (e.g. when the register call returns 400 because the runner is already registered). This should obviously be preferred over restarting since it doesn't interrupt a currently running job and is more efficient and resilient. Restarting should be a last resort.
- Maybe there are better ways to handle even critical errors like the backend being completely offline. In such a case the runner could just continue processing the job and then wait for the backend to come online again to submit the done job. Currently every restart or any other brief interruption of the backend service would possibly result in the runner to dismiss all progress on its current job.