I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305

I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".

I love Dramatiq but I don’t get your point about task completion. You have to choose between at most and at least once.

The main problem with Celery is that it supports many brokers and many features. This creates many subtle edge cases, specifically for brokers not built with AMQP, for instance the "visibility_timeout" attribute of the redis broker which feels like a hack.

This is why I created yet another task framework based on redis with robustness as a the main goal: https://github.com/NicolasLM/spinach

Fair enough but I always interpreted the Celery docs as saying Redis was kind of mostly unofficially supported as a broker.

Personally, I'd use Arq instead of Celery.

https://github.com/samuelcolvin/arq

I find celery's api kinda painful.

Personally I prefer meesee[1], because setting up Celery or other big worker systems is always a lot of work, and they have more components than I usually need.

Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you.

You just need a redis instance and then you're good to go :)

1: https://github.com/Attumm/meesee

Sounds very similar to RQ. Do you have opinions on the latter and/or their differences?

I haven't heard of RQ before, but on first glance they look pretty similar.

The main difference I see is that meesee workers are always setup as python programs, whereas RQ workers are separate programs. (calling a `rq` program insead of `python3 my_worker.py`) Though looking at the docs tells me that RQ can do the latter as well.

Isn't this basically the core of airflow?

Airflow is painful if you want to use it for anything other than time-bounded batch jobs. See https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I have no financial interest in Prefect, but I have used Airflow and their criticisms ring true.