Cases will vary, but if you have 2 million active users at a time and can't cover $1500/mo, then I'm not sure what your options will be. Along these lines, I'm really excited about what kinds of creations Phoenix will enable exactly because the kind of scale it gives you for current hardware. I think we'll see disruptive applications/services come out because of what you can get out of a single machine, so I find stressing price as not affordable in this context really bizarre.
I understand, I do like the idea of Phoenix, and it is definitely great to be able to keep 2M connections open at all.
I'm just wondering where is the per-connection overhead going into. Is there some inherent limitation of the WebSocket protocol that forces the server to keep large buffers or something? Not trying to bash on Phoenix, I'm just genuinely interested in what is the lowest possible overhead one could achieve while keeping an open WS connection.
Doing some rough math and my limited understanding of linux network internals, it's about 40KB per connection in this benchmark. I know that cowboy is going to require ~4KB or so per connection. Consulting a local ubuntu install, the default minimum buffer sizes in TCP will be at least 4KB each (2* for read and write), but by default 16KB each, and by default the max goes to 1.5MB or so each. This is required for TCP retransmits and such. If you have clients on shoddy connections or see packet loss, your memory could skyrocket on you. I remember reading of a case where someone had a service die despite memory overhead of 33% when the TCP packet loss rate went up (still under 1%), but it caused their buffer sizes to grow large enough to run out of memory.
So that's 8KB (will be higher with more usage) for TCP buffers in the kernel, 4KB or so for cowboy, and 28KB or so left for various other bits of the system when amortized per connection.
