I find this line of thinking to be a holdover from an earlier age. One could replace the word "concurrency" in the above comment with "memory safety" to express the popular sentiment as of the 1980s--but in the decades hence the vast, vast majority of programmers have come to be able to completely ignore concerns related to the careful management of allocating and deallocating memory, and though we can argue that the result is a proliferation of memory-hungry Electron-like apps, on balance it's been a dramatic victory for letting people focus on solving the problem at hand rather than distract them with tiresome pointer-juggling.

It's true that in the 90s it was important to cultivate a healthy fear of concurrency in the same way that parents in Cambodia must sadly teach their children to fear landmines. However, there's nothing inherent in the problem space that dictates that the concerns of one's ancestors must be the concerns of their descendants. One day we hope the landmines in Cambodia will be cleared, just as we hope the landmines in concurrent programming will be, and I'll be thankful when that day comes.

I like to say that threads & locks makes concurrency an exponential problem, but the fearless concurrency solutions make them a polynomial problem. Still a problem, but no longer insane, incomprehensible, and impossible.

Personally I suspect generalized concurrency is always going to be something that "professional" programmers deal with, and non-professionals will only get pre-canned solutions for particular problems, because the general case will always be a bit more complicated than non-specialist programmers are going to want to handle. I think concurrency is worse in terms of what you need to keep in your head than raw pointers are, and those are already too much for a non-specialist.

Good comment. I will note, though, that there were forms of safer concurrency going way back. PL designers and developers mostly just didn't use them. The main exceptions in industry were Ada Ravenscar and Eiffel SCOOP. Concurrent Pascal deserves mention as forerunner.

How do you solve for deadlocks at the compiler level? Even if all your memory access is perfectly safe, you can still deadlock on external resources if you aren't pay attention.

That's what I mean by fear and respect for concurrent programming. That's the problem that hasn't been solved.

Deadlocks is a solved problem. Technically, they can't even exist in any concurrency model that doesn't share anything. What can exist is processes waiting for messages from each other, but that's not a deadlock, but a valid behavior and is only potentially problematic without timeouts. Asynchronous message passing with event-driven/reactive semantics farther enforce impossibility to block on waiting for a specific message. In practice strict event-driven semantics are not necessary for it to never be a problem.

Deadlocks are not restricted to shared memory communication. Two Unix processes talking via a socket pair can trivially deadlock ( for example if they are both blocked waiting for the other side to speak first).

Also asynchronous systems can deadlock as well, it is just much harder to debug as the debugger won't show an obvious system thread blocked on some system call; the deadlocked threads of execution still exist but will have been subject to CPS and hidden from view (just some callback waiting forever on some wait queue).

The term deadlock has been used for message passing issues since the dawn of time. It is literally the same issue.

Using timeouts to paper over issues is just wrong. I accept that timeouts are necessary to deal with network issues (and a timeout should cause the connection to be dropped, so won't solve the deadlock issue), but certainly they are not required for in-application message passing.

Finally if an actor won't send a message untill it has received another one, I fail to see how statically declared handlers will help.

> Finally if an actor won't send a message untill it has received another one, I fail to see how statically declared handlers will help.

Think of it as reacting to messages, not waiting. In that model actors of course can react by sending messages, but can't have a special waiting state for specific messages, making it impossible to block other handlers. I'm not sure why this is hard to understand.

We do have this problem solved in every possible way. But it's so not a big deal with actor model, that there is no point sacrificing any flexibility for it.

Forget about waiting. Think about state machines. Let's say that that there is a rule that, if the machine is on state S1, on reception of message M, send message M and move to state S2. This the only rule for state S1. Now if two actors implementating this state machine and exchanging messages find themselves in state S1 at the same time, they are stuck. This is a bug in the state machine specification of course and I would call it a deadlock. How would you call it? How would the actor model statically prevent you from implementing such a state transition rule?

Edit: BTW, not sure why you got downvoted.

This is why I'm talking about specific model with static handlers per actor, where you can't choose handlers dynamically depending on the state you are in. Whether you are on state S1 or S2, all handlers are still able to receive messages, what they can't do is run at the same time.

It can receive all messages you want, but if the only message that it that would cause a state transition, and send out a message, is M, then it is still stuck.

I mean, I'm no expert, but I guess you could statically analize the state machine and figure out, given a set of communicating actors, which sequence of messages would lead to a global state from which there is no progress. I assume that, because message ordering is not deterministic the analysis is probably non easy to do.

Well, this is the limit all models have. You can abuse memory safety the same way and use indices of bounds checkable arrays as raw pointers for example.

>holdover from an earlier age

People have stopped using Java and Go?

> Fear, and respect, the absurdly difficult challenge that is writing correct concurrent code, even when your compiler is helping you out.

There are plenty of safe and easy models for writing concurrent code. Here's a famous one that's easy to overlook:

    gzip -cd compressed.gz | grep error

On Unix, this doesn't use a temporary file. It creates two concurrent processes. The first decompresses a file and writes it to a pipe, and the second reads from a pipe and searches for a string of text. You could call this "coroutines over a stream," I suppose.

And of course, people have been writing shell pipes for decades without concurrency errors. Unix enforces process isolation, and makes sure all the data flows in one direction.

Now, there's no reason a programming language couldn't enforce similar restrictions. For example, I've spent the last few years at work writing highly concurrent Rust code, and I've never had a single case of a deadlock or memory corruption.

One slightly trickier problem in highly concurrent systems is error reporting. In the Unix example, either "gzip" or "grep" could fail. In the shell example, you could detect this by setting "set -o pipefail". In Erlang, you can use supervision trees. In Rust, sometimes you can use "crossbeam::scope" to automatically fail if any child thread fails. In other Rust code, or in Go, you might resort to reporting errors using a channel. And I've definitely seen channel-based code go subtly wrong—but not necessarily more wrong than single-threaded error-recovery code in C.

With the right abstractions, writing concurrent code doesn't require superhuman vigilance and perfection.

Yes there will still be fear sometimes, but the point of the compiler is to help people to be fearless sometimes.

The halting problem does not need to be solved, since no program in practice runs forever. In some of the environments like C we don't have a way to specify finite things, where cases would be enumerable and safety guaranteed. To be fearless sometimes greatly reduces the mental workload of programmers and people who read other people's code.

A lot of the fear around concurrency comes from hard won experience. Shared memory concurrency is _really_ hard to get right, and the results are often streams of execution that are very perplexing. Message passing concurrency isn't without pitfalls, but the pitfalls are fewer, and the debugging usually more clear -- your stuck processes/threads usually indicate what they're waiting for, and when you find two threads waiting for a message from the other, you know you messed up the ordering. Of course, the message passing infrastructure still has some tricky concurrency problems itself, but that's a smaller, more tractable problem.

> But until the halting problem is solved at a compiler level (at a level that can account for runtime cases)

Idris can already check for totality (i.e. guarantee its programs will halt) without solving the (unsolvable) halting problem. So we're almost there!

And even in Haskell, you can write certain classes of concurrent code without thinking thanks to 1) its great RTS and concurrency libraries and 2) the fact that Haskell programs compose so well: If I have thread safe programs A and B, Haskell gives me a variety of ways to compose their results in such a way that thread safety is always preserved.

I'm not sure I understand the argument, why is the halting problem even important here? Surely a compiler can split loops into ones that halt so they only run a little bit at a time. But you don't necessarily need a compiler for that, OS can help too and, for example, invoke a signal handler from which you can stop currently running actor. Actors can be safely stopped, destroyed, preempted, etc. without breaking other actors.

It is impossible to ever handle concurrency completely at the compiler level. We can handle them at the runtime level (via things like transactional semantics and timewarps), though the overhead can be great.