Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Linkcheck is spawning off a new thread to handle finding links in new/updated content. This thread uses the same database cursor as the main thread, which can result in a deadlock (#208).
One can abuse the boolean
linkcheck.listeners.tests_runningto force this to run synchronously, but this is not feasible when more than a few objects are to be checked.As proposed in that issue, we implement a third path to do this: Giving the task to a Celery worker. This is an entirely separate process that has its own dedicated database cursor and should not suffer from such deadlocks.
It also means that we cannot just pass djangos python proxy object of the database entry but need to reference it by app and model name and ID to submit it through the broker.
Proposed changes in detail:
listeners.pytoworker_tasks.pyinstance_pre_delete()→do_instance_pre_delete()for consistencyLINKCHECK_IN_CELERYcelery.pyaccepting the model and id for the instance to be checkedLINKCHECK_IN_CELERYis truthy, importcelery.pyinlisteners.py. This means that otherwise, nothing from celery is imported, preventing celery from becoming a hard dependency of django-linkcheckLINKCHECK_IN_CELERYis truthy, don't spawn a new thread to run any linkcheck tasks, but submit them to celerySince arguments for celery tasks have to be primitive types (not python objects representing database entries) we have to juggle around a bit and find out the app and model name of the object to send along with the id, and on the other side use it to fetch the object again to hand to the function doing the actual work