Redid + Sidekiq was a default for a long time in the Rails world as well, but it’s an unnecessary complication (and expense) for most use cases. Just use your existing DB until you need to seriously scale up, and then look at a message queue.
I’ve used Redis for leaderboards and random matchmaking though, stuff which is doable in postgres but is seriously write-heavy and a bit of a faff. Gives you exactly the sort of goodies you need on top of a K/V store without being difficult to set up.
As for caching - it’s nice to use as an engineer for sure, but pretty pricey. It wouldn’t be my default choice any more.
Rails is attempting to solve this with Solid Queue, which was inspired heavily by GoodJob, both of which use Postgresql (and more in the case of Solid Queue). Both seem to be fairly capable of "serious scale", at least being equivalent to Sidekiq.
A place I work at which I can’t name uses GoodJob at FAANG scale. And it works perfectly. The small startups and lower scale places still reach for sidekiq because they seem to think “it’s faster,” but it ends up being a nightmare for many people because when they do start reaching some scale, their queues and infra are so jacked up that they continually have sidekiq “emergencies.” GoodJob (and SolidQueue) for the win.
I like the sidekiq guy and wish him the best, but for me, the ubiquitous Redis dependency on my Rails apps is forever gone. Unless I actually need a KV store, but even for that, I can get away with PG and not know the difference.
Unfortunately there are still some CTOs out there that haven’t updated their knowledge are are still partying like it’s 2015.
I’m curious, are you running GoodJob in the same database as the application? For smaller scale stuff this is super convenient, but I wonder if it will become a problem at higher loads.
I’ve used Redis for leaderboards and random matchmaking though, stuff which is doable in postgres but is seriously write-heavy and a bit of a faff. Gives you exactly the sort of goodies you need on top of a K/V store without being difficult to set up.
As for caching - it’s nice to use as an engineer for sure, but pretty pricey. It wouldn’t be my default choice any more.