Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Don’t use Celery.

It doesn’t solve a single problem that a set of bare queue consumers can’t resolve.



Like having a python programming interface to seamlessly defer tasks to arbitrary local/remote workers?


I think it's closer to "Celery is a great tool to get started, but as your product matures you'll end up falling back to a simple AMQP consumer stack":

- By default, task IDs are just pickled Python objects - if you want to change the location of a function, it might break; - The whole process appears to be reloaded for each task - any heavy loading ends up being performed on each call; - I couldn't find any "ops" documentation: how does it interact with RabbitMQ? How are deferred tasks implemented? What happens if a node crashes?

Although the API is nice, the product itself seems ill-suited for actual reliable, production use — at that point, it's sometimes easier to just deploy your own minimal API using the serialization format you've chosen to adopt (JSON, Protobuf, ASN.1, … ;)


I’ve found celery to be extremely useful and reliable at handling > 1MM daily async tasks with non deterministic latencies. The programming interface melds beautifully with Flask.


For the record, back in the days, Instagram used Celery at large scale : http://lc0.github.io/blog/2013/05/01/celery-messaging-at-sca...


Anecdotal, but we use Celery in a large scale production environment without too many issues.


Celery has been pretty reliable when i have used it in the past. Admittedly I have newer done anything particularly fancy with it, just running some background tasks and sending out some emails.


So it didn't solve any problems that other simpler libraries would not solve, as I have said above.


You also said that it was unreliable. Not in my experience.


"Don't use a message queue/broker/scaled out system".

"It doesn't solve a single problem that a set of queue instances on a single machine solve. Who needs any resilience?"

Weird take, that is provably incorrect and wrong.

No thanks.


I stated that Celery as a brand is horrible - I am not saying you should not use a task queue product.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: