Celery is great if you want to distribute your tasks into multi processes/machines.
However if you use Python’s multiprocess with a Celery task, you may probably encounter:
It’s a known issue that Celery doesn’t support multiprocess well in task issue. One solution will be to refactorize your code to avoid using multiprocess, since you already have Celery to do it for you.
However if you are using libries lick scikit-learn, ntlk which use multiprocess underneath, you may try the following to workaround it: