> I would double and triple check your apps configuration, it seems there is something really unexpected going on.
More than one thing for sure. It's big and highly heterogenous ecosystem (multiple umbrellas) in distributed environment with high idempotency requirement. Without safeguards decisions were made that today make things very complicated. It's difficult to challenge long used patterns without hard recommendation or concrete evidence (I looked into Perceived Complexity analysis for Elixir but couldn't find anything).
My organization now builds the story of "Elixir codebase is hard to work and unreliable - let's switch to other tools". I don't like that story because I still remember all the fun I had and all the systems I produced that stood years with 0 maintenance. But those were small teams and small projects and today it's an enormous Jenga tower that's risky to breath around.
I would go as far as to say that our codebase is somewhat of a Petri dish for all kinds of issues (especially on dev/test envs, but not only). I've seen code merged to main branch because it wasn't picked up as changed and used stale cache, multiple-Elixir and OTP versions used in compilation, arch spillovers and more.
>I can see popular libraries spread across 3-4 different major versions), This is also something that should not be possible.
We have overrides and I don't see the umbrella test helper so I guess that umbrella-level overrides don't play nice with non-compiled in-app test runs.
> I am glad to say there is work happening towards this area (including soon the ability to search across all of the dependencies of your own project).
Looking forward to it, one thing that I often change too, is changelog across libraries, so it would be nice to always have those up-to-date.
> It should not be using one core, even it if compiles one dependency at a time. My whole point is that you get parallelism from within the dependency/project.
Project or a single-dependency compilation is fine - I felt different after recent updates to our stack and won't complain. In one umbrella I have opened I see ~250 deps packages and deps.tree shows me ~6500 lines of output, some of those are compiled multiple times - I blame the loops.
I have similar CPU - 8 performance cores and 4 efficiency ones. Usually deps.compile takes less than 100% of total CPU with spike to 150%. On partitioned tests I can feel the warmth of 1100% CPU usage (it also makes me smile, because I like big numbers). Right now I'm thinking that maybe I could spawn ~250 containers, make each compile dependency and then merge output into one and see what broke ;-)
> So you are getting the download count of one package, one that has been added to Erlang/OTP (and Elixir itself) and is more than expected to decrease in download count, to estimate the popularity of the whole language and a ecosystem of 20k+ packages?
Not ideal, but the best I could find. I also looked at Ecto which is standard, but figured out that json is more often used in projects than a database. Given quality of software itself I'd expect steady increase on the "core" libraries. But I also hear from prior projects about them being sunsetted. 2 or 3 projects in Elixir less, no big deal. In current organization few of us are actively advocating for Elixir and BEAM. We're minority and newcomers encounter as a first thing difficult stack setup (Erlang and Elixir version) long compilation time, hundreds of odd warnings and LSP that takes 40s to pick up on changes and highlight some errors.
--
I'm not in position of making any demands, it's self inflicted 99% of the time, and it's not a bug that can be fixed upstream it's just a subjective experience, and I wish it could be better.
I am sorry to hear. I understand it may be something out of your control but it seems you have clearly identified some "smells" that would be worth spending some time investigating.
For example, you can't have loops in deps, and therefore ~250 deps should not print a 6500 thousand entries long tree. At least, for this particular problem, you can isolate your project structure, without any code, and try to reproduce it externally. And, while you can override deps, the goal of an umbrella is to share dependencies, so overriding an umbrella sibling dependency is a smell too.
You said it's an enormous Jenga tower that's risky to breath around but it seems at the same time no one wants to invest on an air purifier. If it is of any help, you can look at the Remote case on the Elixir website (https://elixir-lang.org/blog/2025/01/21/remote-elixir-case/), they have a large codebase, around 15k files, 300 engineers (several dozens being Elixir ones), and while their codebase is healthy, you can see they had to invest on some "bottlenecks" that appeared along the way, such as CI times. And the need to invest in the code base itself will be true of any language as time passes. Best of luck!
Thank you. I completely agree and I'm not going to back out (nor switch technology at this). I hope that one day I will be able to open-source some of the work and help others who struggle. In the meantime I'm looking forward to deps compilation time speedup and documentation tooling (not to mention 1.19). My own experiment of parallel isolated builds showed speedup comparable to the one you mentioned, but I dislike hacks so I'd rather wait for peer reviewed version :)
> I would double and triple check your apps configuration, it seems there is something really unexpected going on.
More than one thing for sure. It's big and highly heterogenous ecosystem (multiple umbrellas) in distributed environment with high idempotency requirement. Without safeguards decisions were made that today make things very complicated. It's difficult to challenge long used patterns without hard recommendation or concrete evidence (I looked into Perceived Complexity analysis for Elixir but couldn't find anything).
My organization now builds the story of "Elixir codebase is hard to work and unreliable - let's switch to other tools". I don't like that story because I still remember all the fun I had and all the systems I produced that stood years with 0 maintenance. But those were small teams and small projects and today it's an enormous Jenga tower that's risky to breath around.
I would go as far as to say that our codebase is somewhat of a Petri dish for all kinds of issues (especially on dev/test envs, but not only). I've seen code merged to main branch because it wasn't picked up as changed and used stale cache, multiple-Elixir and OTP versions used in compilation, arch spillovers and more.
>I can see popular libraries spread across 3-4 different major versions), This is also something that should not be possible.
We have overrides and I don't see the umbrella test helper so I guess that umbrella-level overrides don't play nice with non-compiled in-app test runs.
> I am glad to say there is work happening towards this area (including soon the ability to search across all of the dependencies of your own project).
Looking forward to it, one thing that I often change too, is changelog across libraries, so it would be nice to always have those up-to-date.
> It should not be using one core, even it if compiles one dependency at a time. My whole point is that you get parallelism from within the dependency/project.
Project or a single-dependency compilation is fine - I felt different after recent updates to our stack and won't complain. In one umbrella I have opened I see ~250 deps packages and deps.tree shows me ~6500 lines of output, some of those are compiled multiple times - I blame the loops.
I have similar CPU - 8 performance cores and 4 efficiency ones. Usually deps.compile takes less than 100% of total CPU with spike to 150%. On partitioned tests I can feel the warmth of 1100% CPU usage (it also makes me smile, because I like big numbers). Right now I'm thinking that maybe I could spawn ~250 containers, make each compile dependency and then merge output into one and see what broke ;-)
> So you are getting the download count of one package, one that has been added to Erlang/OTP (and Elixir itself) and is more than expected to decrease in download count, to estimate the popularity of the whole language and a ecosystem of 20k+ packages?
Not ideal, but the best I could find. I also looked at Ecto which is standard, but figured out that json is more often used in projects than a database. Given quality of software itself I'd expect steady increase on the "core" libraries. But I also hear from prior projects about them being sunsetted. 2 or 3 projects in Elixir less, no big deal. In current organization few of us are actively advocating for Elixir and BEAM. We're minority and newcomers encounter as a first thing difficult stack setup (Erlang and Elixir version) long compilation time, hundreds of odd warnings and LSP that takes 40s to pick up on changes and highlight some errors.
--
I'm not in position of making any demands, it's self inflicted 99% of the time, and it's not a bug that can be fixed upstream it's just a subjective experience, and I wish it could be better.