A lot of points made here is also is why I believe Phoenix framework is the BEST stack for building mvps in 2024.
1. you get concurrency and pubsub built in with no external dependancies.
2. postgres suppport out of the box. our entire deployment is just phoenix + postgres
3. creating a microservice is easy. create a genserver, add it to your application.ex -> DONE. no need to create a seperate repo. its already able to send and receieve messages from other parts of your monolith using the built in pubsub system
4. Oban, the background worker library uses postgres.
end result is that as a small team, we are able to knock out a lot of features and scale it without any of the complexity most startups have. Its been 4 years and only now we are starting to consider bringing in an external message queue.
I feel like it is significantly harder to learn elixir and phoenix as a newbie. The learning curve is very steep owing to the fact that good learning resources seems to be scarce, maybe not for elixir but definitely for phoenix. I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found and oh, please don't tell me to go read the documentation. It is not meant to be read by beginners. Almost everyone in the community recommends the prag studio course but $200?? I can't afford it XD
>The learning curve is very steep owing to the fact that good learning resources seems to be scarce
thats a fair assessment and I sympathize. I'm at the point where being able to hire devs is a concern of mine. we just hired a new elixir dev to join my team and it was difficult finding people with experience. We had to go though a recruiter who specializes in elixir.
That said, I encourage you to not give up. I became an elixir dev after 9 years of working with javascript and nodejs. Personally I found it pretty easy to learn (was building apps in it as quickly as I could in nodejs after jsut 2 weeks). Its a magnitude order easier to learn than rust (though thats admittedly a low bar).
> I have been trying to grasp it for a month now, but an up-to-date resource is nowhere to be found
This is definitly an issue I admit. once you know elixir, you know the idioms and picking up the newest changes is pretty easy but I remember how it was to be a begginer. would love your input on what would be needed for a learning resource as I have been thinking about writing a book on phoenix for newbies.
As far as getting into elixir, if you have experience with javascript, I reccomend checking out immutable-js. the data structures are basicly elixir's data structures but in javascript. my experience ins scheme and functional style js were a huge help for me when I was learning elixir.
> Personally I found it pretty easy to learn (was building apps in it as quickly as I could in nodejs after jsut 2 weeks)
Makes me feel better! I have also successfully converted one of my java spring-boot app to phoenix though I am not sure how efficient it is. It definitely involves way more magic than spring for sure!
Maybe its just quite hard for me to make the jump from imperative/OOPs way of coding to the functional way.
> would love your input on what would be needed for a learning resource as I have been thinking about writing a book on phoenix for newbies.
I would love to help! I think the harder part of making such a resource is updating and staying on par with the framework devs.
> It definitely involves way more magic than spring for sure!
Elixir and Phoenix really strive to keep the "magic" to a minimum. Bending your mind the functional way instead of the OOP way, and learning how Elixir macros really helps dispel the magic and realize it's less excessive cleverness as Ruby/Rails often indulges in and more just the benefits of FP.
Having on-boarded quite a few jr developers with no previous experience with Elixir (and have purchased the prag studio course for most of them) I've heard that many of them also really enjoyed
From looking around there seems to be a host of free videos on YT as well as many books on elixir/phoenix. Also, I believe prag studio gives away 1/2 of that initial course for free, probably enough to get you started.
hmmm, yes you are correct. I thought they used to offer more for free but maybe my memory is off. Apologies.
You are also right about that the Phoenix book needs an update, they changed A LOT with the way they structure code in the last release. Which, you don't have to use btw, you can still use the older methods. The major differences are here https://elixircasts.io/upgrading-to-phoenix-1.7 which I can honestly understand the pain a new developer feels when the books are out of date.
I'd still consider going through an older book, you'll learn 90-95% of the same thing.
> I'd still consider going through an older book, you'll learn 90-95% of the same thing.
Seconded. I have the book, and it's pretty easy to understand where the differences are. And worth learning the pre 1.7 Phoenix because a lot of the changes thereafter make so much more sense when you know the pain points that motivated them.
As always, "it depends". There are situations where you can use Postgres as a queue perfectly fine. But there is no way Postgres can do as well or better than RabbitMQ in all situations. Same for text search, and so on.
Of course... that said, there are too many times where the additional complexity isn't necessary or worth it. I say this as someone who is usually somewhat early to introduce RabbitMQ or Redis into a new environment.
Most applications, in most environments are not ever going to see more than 10k simultaneous users. You can vertically scale an RDBMS like PostgreSQL a LOT.
Not to mention, there are also a lot of ways to approach similar techniques with PostgreSQL that you might reach for a different DB platform for. JSONB in Postgres is actually easier to reason with IMO than MongoDB as a platform. Depending on your needs, it's a much better fit with hybrid tables with additional details in JSONB.
> Use Postgres for Fulltext Search instead of Elastic.
Yea, that's not quite the same. Im going to make the argument that a well tuned search platform along side your data storage is probably one of the most useful things you can have. It's one of the first places where you will be "let down" by the Postgres solution.
As a document store, a queue, Postgres will do the job up to a point. When and where it breaks down in one of these ancillary roles you're likely to have the resource to support more systems, or a more "focused" tool.
I think that's true of most of the examples. There are times you will want/need to break out. When you do, it should be apparent and compelling to do so. Whether it's Redis, RabbitMQ(or another queue option), or Elastic you should have a clue as to what and why you need to break out.
Many/most applications don't have those needs. Then again, I'm also a fan of using JSONB for details in fewer tables over overly normalized structures. It varies.
I've found this bias to be useful with two main exceptions:
1. It's nice if you can avoid using Postgres to store large files.
2. It's nice if you can avoid using Postgres for an event stream with massive volume (say billions of records per month that must be maintained and queried).
Postgres will work for either of these, but there are real advantages that come if you can keep your database lighter than these allow for.
An event stream / message queue implemented in PG can actually be fine at some billions of events per month and has some major advantages over standard queues. I just wouldn't get too high into the billions.
I thought unlogged was used so crash recovery would truncate / dump the table on a crash with 100% Dataloss but this let you put stuff on a tempfs / ramdisk?
My leading use for redis on a system where I am already using Postgres is to use Celery. I suspect this is just one of many cases where people add an additional datastore because their frameworks and/or libraries make it easier to to that than to use Postgres for that type of data.
All of these points are valid, when given a caveat of "if Postgres covers your actual needs".
Like, I've personally worked on many systems that used Redis for absolutely no reason. As it was not actually measurably improving the latency of any part of the system. It was basically just another place to store data.
Obviously, Postgres and Redis are not the same, and there are situations where keeping data in memory at all times is important for latency. Hence the caveat. But most companies are not in any such situation.
Same story with the rest of these technologies - people often reach for them without really considering whether they are any sort of improvement over just storing the data in a relational database. This is commonly done by engineers who underestimate the capabilities of a relational database.
For me, I tend to reach for Redis when I need/want caching or otherwise data that can expire or otherwise disappear. Once I have Redis, I may use it for other types of things as well as caching. I know that you can use persistence and other features for more use cases, which can work well.
Of course, I'm more inclined today to just use PostgreSQL unless and until something truly needs to break the mold.
1. you get concurrency and pubsub built in with no external dependancies.
2. postgres suppport out of the box. our entire deployment is just phoenix + postgres
3. creating a microservice is easy. create a genserver, add it to your application.ex -> DONE. no need to create a seperate repo. its already able to send and receieve messages from other parts of your monolith using the built in pubsub system
4. Oban, the background worker library uses postgres.
end result is that as a small team, we are able to knock out a lot of features and scale it without any of the complexity most startups have. Its been 4 years and only now we are starting to consider bringing in an external message queue.