I have an application where once a job succeeds it needs to go through a series of steps, some are CPU intensive (ffmpeg, etc.), others require communicating with an external API that may fail (Discord), etc.
This is not a massive data-engineering project. I just need to ensure the steps get run to completion.
My current plan to do this is to just use Redis lists as queues, and then have steps go from one queue to the next, but I'm wondering if there's a better way.
- https://camel.apache.org/
- https://www.windmill.dev/
- https://github.com/huginn/huginn
Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.