I love seeing commercial activity around using Postgres as a queue. Last year I wrote a post titled "Choose Postgres queue technology" that spent quite a bit of time on the front page here. I don't think it's likely that my post actually sparked new development in this area, but for the people who were already using Postgres queues in their applications, I hope it made them feel more comfortable talking about it in public. And I've seen a notable increase in public discussions around the idea, and they're not all met with derision. There's long been a dogma around Postgres and relational databases being the wrong tool for the job, and indeed they are not perfect, but neither is adding Redis or RabbitMQ to our software stacks simply to support queue use cases. Kudos to the Hatchet team! I hope you all find success.
Yes, I remember reading the post and the discussion surrounding it being very high quality!
I particularly like the section on escape hatches - though you start to see the issue with this approach when you use something like Celery, where the docs and Github issues contain a number of warnings about using Redis. RabbitMQ also tends to be very feature-rich from an MQ perspective compared to Redis, so it gets more and more difficult to support both over time.
We'd like to build in escape hatches as well - this starts with the application code being the exact same whether you're on cloud or self-hosted - and adding support for things like archiving task result storage to the object store of your choice, or swapping out the pub/sub system.
> neither is adding Redis or RabbitMQ to our software stacks simply to support queue use cases
I disagree that "adding Redis to our software stack" to support a queue is problematic. It's a single process and extremely simple. Instead now with tools like this, you're clobbering up your database with temporal tasks alongside your operational data.
Out of all the processes/infrastructure ive had to manage in my career, Redis has been the simplest, and least hassle out of all of them. Even when you add Redis sentinel to the picture, it just does its job
I mostly agree... a traditional RDBMS can vertically scale a lot on modern hardware. It's usually easier for devs to reason with. And odds are already part of your stack. You can go a long way with just PostgreSQL. It works well for traditional RDBMS cases, works well enough as a Document store and other uses as well. The plugin ecosystem is pretty diverse as well, more than most competing options.
Where I defer is if you already have Redis in the mix, I might be inclined to reach for it first in a lot of scenarios. If you have complex distribution needs then something more like RabbitMQ would be better.
I remember that post and I’ve read it a few times, thank you for it! I was already working on River at the time but it was refreshing to see the case made so strongly by another person who gets it.
