What do you mean by concurrency here? JS like async or Java/C# like threads? And why would concurrency help in problems where PHP is involved, in the problems I solve with PHP I don't think I ever needed async. For unning more things in parallel like processing 1000 RSS feeds you can spawn 10 scripts that run in parallel and if one crashes the others still continue to work(I had experienced crashes with some RSS XML files crashing the built-in XML parser).
Anything that will block the server. Fetching data from an external API, doing some kind of expensive computation, sending an email etc. You can either handling this Node style with blocking i/o, throw the work onto a background queue like Sidekiq, or use a language with first-class concurrency support like Golang or Kotlin. To me Kotlin feels like a far nicer and more scalable language than Golang.
I like this about PHP, you don't have a callback hell. I don't disagree with you just sharing my experience here, we might work on different kind of problems.
I work on highly concurrent backends that handle 100k requests per second across a cluster, and PHP will never work in those situations. Any time you have to orchestrate multiple data store or service calls in the background, you don't want to block.
Callbacks are not how you do non-blocking IO. You typically have multiple threads and use constructs like futures or promises to make them easy to manage.
PHP bakes you into a very simple request flow. There's so much more you can build if you get out of that manner of thinking.
Yes, I never disagreed with that, best tool for the job. But your problem is not the usual stuff that web projects are about.
Edit: FYI I have experience with other languages too, so if at my work I use PHP does not imply that I am some inexperienced person that has no idea how other languages work. I do not have experience with your domain of high concurrency, low latency stuff but I worked with threads, and thread pools and multi-processes programs(it happened that some thread would segfault and bring the app down so we had to use background processes instead of threads) .
Just curious what problem are you trying to solve, I never had the need for different threads to communicate in PHP, most of the time in my work a user makes a request, and a PHp scripts does something like update the database or does some searching/working and returns a result.
What if that request was a document upload and your system is calling out some external api or processing for long enough such that you’d rather return now and update later when the job finishes. Maybe through Ajax or a redirect.
The shenanigans that WP plugin devs go through to get around the lack of a queue is ridiculous (chunk the file and keep reloading the page until we’ve processed all the chunks or have an Ajax endpoint that processes a chunk at a time...) and the when I asked a marketing friend of mine who works with Wordpress he said that redis is an “advanced requirement” and I should keep things simple.
In Django land I’ve always set up queues even just to send emails because it keeps things snappy.
For that kind of problem we create a database table. then you schedule there your work tasks and background scripts will fetch jobs from the database and run them. This also works great if the third party is busy or fails, the background jobs are set to attempt a few times before giving up.
What solution would you prefer for this kind of jobs? launching a new thread directly? or have a worker project always running and send to it the job directly?
.... As you identified at the end, the correct solution for this problem is a job queue. There is nothing about PHP that prevents the use of a job queue - in fact the shared-nothing, process-per-request model almost encourages the developer to offload as much work as possible to a job queue..
Using Wordpress (or it's plugins) as any kind of reference for what PHP is capable of, is like using McDonalds as the basis for a cook book.
If putting a job/event into a queue, or writing a log entry is taking so long that you need a separate thread just to enqueue/log that entry, something is dreadfully wrong with your application.
Ironically, the root cause that made me think of that example was AWS.
The example was where each SQS call (there could be as many as five) was taking between 50ms-200ms, and logging took another 75ms. The rest of the request processing time was approx 50ms.
There's a bunch of other architecture options we could have looked at, but due to the share-nothing nature of PHP, they weren't an option.
I do. However, the process would have stuck around until it finished. It wouldn't have changed our overall RPS at all.
Arguably the same problem would exist in other languages, but at least there we would have had the option of using threads, and we could have executed multiple calls to SQS in parallel. Probably not something I'd want to do all the time, but it would be nice to have had that option in this case.
