> I'm excited for a world where a normal software developer doesn't need to know about...
I'm not excited for this. There's a quote I cannot find that I miss greatly, but maybe a Whitehead quote or some such about Civilization being measured by that which it doesn't have to think about, that which it takes for granted. It's always struck me as powerful, but giving ourselves the ability to forget & un-learn does not tempt me in development. We are the builders, and this great rich pool of possibilities is rarely improved with merely forgetting & becoming an exclusively higher-level operator. Depth is deeply rewarding in development.
Let's talk about the cloud some, & this pool of capabilities we are so delightfully placed at the helm of, and the cloud's influence on these capabilities.
> Somewhat ironically, software development is one of a vanishingly small subset of knowledge jobs for which the main work tool hasn't moved to the cloud. We still write code locally, thus we're constrained to things that work in the same way both locally and in the cloud. Thus, adapter tools like Docker.
It's super hard for me to imagine a replacement, not because replacements won't be great, but because replacements will have a hard time becoming core knowledge for the software development world.
(It's problematic because great openness let us roam too freely unboundedly but,) one of the greatest glories of software development is how unboundedly open & downright democratic it is. We use software until it stops serving us well, and then we either roll up our sleeves, dive in & improve it, or start something else entirely. But we can keep drawing from the same pool, from the many many possibilities & ideas which all interrelate & support each other, to shape new ideas & give life to new forms.
The authors premise, to me, feels like a proposition that we will be so well served the cloud that we can just leave where we are behind. This is about not needing the pool of knowledge or capabilities we have, the systems we have, because we'll be working somewhere else.
In many ways though, that to me sounds like declaring that the future is different, therefore we need a new primordial soup to start from. 'Using only ATCG for genetics is a paradigm that must be eclipsed!'
But you know what? Someone builds that new place. And in 98% of cases, the same old fundamentals & tools are still at play, underneath the new tools, underneath the new abstraction. Larry Wall (in Perl is the first post-modern language) would say: the truth is our systems will always be post-modern. Modernity's shining image of itself as brilliant sprung from nothing novelty is rarely true. New ideas more often than not creative re-applications and re-mixed of old materials.
Rather than simply say that a new paradigm is probably not so new, I think there's a deeper challenge, which is: how does a new paradigm ever rise to such a height that it becomes well known? How do we adopt a new paradigm & start teaching it & using it? How does it become the next thing?
We are very well served by the cloud. It's presence as a system of services, as far off, maintained-by-other-people, no-longer-our-problem miracle working wondermachine is all true & very well reported and it is coming for everything and everyone doing work today.
But I'm not at all afraid, because, in 99.99% of cases, these vast neo-mainframes have no way to pass on their genetics. They are alone and isolated and developers cannot get into their bowels. There is no "The Midnight Computer Wiring Society" of the cloud, and there never can be and there never will be because the cloud until the PC and the tools we have here is about control & orchestration & order & rules, and there's no permission in the cloud to go develop your own culture, to become a new wave, to change everything for everyone (including the other devs) because you are just one lone neo-mainframe, just a couple of your own ideas that you're calling your paradigm & building your cloud by, but in a huge number of cases your ability to interact with talk with share with other people also doing cloud or to enable other people to try to cloud like you do is exceedingly small. Clouds are all unique and alone & they have a much harder time spreading socially.
How does a new cloudy paradigm ever get the ball rolling? What are it's central tenants & beliefs that make it a flexible, malleable, swiss-army knife where all developers everywhere have even more power & creativity- not just at building applications atop it but enhancing & growing & exploring the platform as well? I do think eventually we will find new things to make core, to form real communities & new shared basis upon (I think Kubernetes' apiserver+controller paradigm is probably a core construct in the future, for example). There's early early signs we are civilizing what so many hyper & not so scale cloud technologies have frontiersed. (But oh it's so early, and it's at so much more depth where this happens than the shallow 'your workflow will be replaced' message of this article).
I think there's a lot of good peering into the mid-horizon to do, that the attempt to peer forwards are good, & commend this article. It's right to ask (sic):
> Rethinking these abstractions to be native to the new world let's us start over and redefine the abstractions for what we need?
re else of note).
But I highly doubt we will really get release/escape from the past. The article tries to question the past 50 years, and I think perhaps yes we might diminish it, it might not be at the forefront forever. But I have a hard time imagining enough real value or enough real difference- even if we switch to Zircon or KataOS or Zephyr or Genode or the next thing, I tend to think most existing abstractions will largely remain, perhaps mutated some, some more prevalent than others, and that the view will not really end up looking that very different. Platform will continue, yes changing, but also in many ways similar.
The above all speaks to a fairly slow-building evolutionary view of the future. That said, I think we really dropped the ball on trying to bring software development online, have really had a shitty unambitious maxima we've been stuck at. We still write a crap ton of code that's just driving http clients. gRPC is still deeply one process talking to one service, a very convenient way to still manually write individual send(call)/receive(return) calls/streams. Cap'n'Proto dared to dream a little more multi-service, to follow somewhat after E-language, but never materialized 3-plus-way communication.
The long hangover after CORBA and SOAP blew out & got eaten by simple-is-better ReST has turned into a forgetting, to not trying. The idea of finding new abstractions is interesting, actually (contrary to first part of my rant), but programming language design has remained so focused within the language that we don't have the creativity to expose & play earnestly with the abstractions we have. Language design has focused near exclusively on building better processes, and without integrative cross-system rework, without a much higher scope of change desired. We're just shuffling the cards again and again with the same base system; it's all different syntax spins, different ergonomics, maybe a new safety guarantee (and boy are people excited about that!) but all the same underlying patterns, variously cloaked. A dull post-modernism. If there is change, real change, I think it comes from going back & re-trying an E-lang or an Erlang, or more generally working aggressively towards multi-system. And much of that could just be taking what we do inside processes & doing it on the web/net. I ask myself regularly, and alas, I haven't gotten around to fucking around and finding out: what would a EcmaScript/JavaScript Promise look like, but on the web? This kind of primitive material makes total sense to developers, but we don't really express these things in systems ways; there's the inner process world, & outer systems world, and it's not abstractions we need per se: we just need to tear down the veil between these two realms. The process already has the materials to make a great cloud, we just haven't opened the box yet.
> There's a quote I cannot find that I miss greatly ...
This is probably it: "Civilization advances by extending the number of important operations which we can perform without thinking of them." - Alfred North Whitehead
(And I agree that depth is rewarding and civilization allowing us to forget things isn't always tempting!)
A similar quote from A.N. Whitehad: "By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems, and, in effect, increases the mental power of the race."
> Cap'n'Proto dared to dream a little more multi-service, to follow somewhat after E-language, but never materialized 3-plus-way communication.
I still intend to implement this! It just has yet to become the most pressing issue for the projects I'm focused on, since the development model is already supported using proxying which makes 3-party handoff merely an optimization.
Its not at all like I dont see you being extremely extremely visible elsewhere doing tons of stuff. But I am excited to hear you still are excited for 3-party.
I'm not sure what thr designspace of a language that natively supports cap'n'proto would be & whether that'd be signififantly different than really good libraries or macros for a language. Maybe existing languages are flexible enough. But my gut is that we can mind of start having more ambient systems, ambient data, ambient code, more freely if we really built the language to work beyond the process scope from the start, using cap'n'proto as a base. I readily admit though there's probably some very fine ways to make this remote coding very seamless a development experience with what we've got though, if we want, perhaps with very very few rough spots.
I think you're right that the same old fundamentals & tools will still dominate, but I think this 'new world' can arrive by just making those old fundamentals more powerful and useful.
Erik's example of Spotify has within it the story of an ever more powerful and reliable internet network making first music streaming possible, then video streaming.
The (mobile and wired) internet network is getting so fast these days that maybe even a cloud-first, cloud-only software engineering process is ready to be enjoyed.
Spotify is an interesting example to bring up, & somewhat a sore one for me. Because their Spotify Apps API was in effect an attempt to make Spotify a music cloud. It was, in my view, raringly successful, a great system that thousands of people built intensely good and new music experiences on, embedded within the Spotify desktop app. It turned the desktop client into a cloud-run system for rapid music app deployment, all inside the Spotify walled-garden.
So Spotify is now an example of a dead end, a former cloud, a mere product to me. They've taken great works from the past & about, and they've built one product, and they're going to spend decades moving around where the buttons are and trying to tweak how and when they can ring the cash register & deposit some money into Spotify, Inc. They're from the cloud (sort of), but they've wound back 90% of their ambition to be or contribute to clouds.
Spotify (& many others) can build what they build because of a conflux of factors. Simply having gobs more hard drive, cpu, network throughput, and (most important of all) new online consumers are the core hard & fast requirement that enabled Spotify to become Spotify. But that's only semi-related to what I think is really at the heart of this conversation: the cloud. Yes the team was good about scaling out & aggressively deploying new technologies, devops & core tech, and that helped them go. But it's confusing & unhelpful as a case study. Fact is: switches were just getting better, cpus were just getting better, and Spotify could almost certainly have happened in a fairly legacy way, with fairly legacy ideas, & there'd been streaming before, albeit executing & attracting/keeping necessary talent would have had worse odds on legacy ideas.
Yes: from a consumer perspective, Spotify leverages the internet to on-demand deliver content. It's an example of most of the computing happening in a far off neo-mainframe. That's absolutely something we associate with the cloud. I absolutely see that as core to Erik's story here.
But the characteristic seems somewhat uninteresting to me in isolation. I do think more scale out abstractions & ideas have a huge place, a huge future, but also, they keep running into the "then all developers are just consumers of shit they really have no idea of or power over" problem that means there's no real social environment surrounding these advances.
Something that seemed real to me from these threads: the comment griping about never managing firewall rules by hand rings true. And we are developing control planes aka controllers aka operators, are building more intent-based autonomic systems, reasonably well, that do our lifting for us. There's a host of good new "edge" (not edgy edge edge, just like, lots of data centers edge) tech that's also like- yeah- cloud it up more. Think less about computers/resources/clusters, just push code. These are all in the heart of cloud, of making available various grid computing/utility computing notions that have circled around for a long time, of making us think less specifics. And I think that's indeed true & powerful. But it keeps running into the asocial problem above, that there's no social environment, most of the secret-sauce is retained, locked inside the neo-mainframe.
CloudFlare and Deno seemingly are some of the only two who seem to realize the Tim O'Reilly adage that I hear no-where near enough this decade: "Create more value than you capture." Or else your dream is going to some day die as your dream, with yes maybe good marks, but no real lasting success. If cloud computing is to be a future of real note, it has to be a shared one. That's been an exceedingly brutal gauntlet that few technological/cloudogical would-be's have proven their advance through.
I'm not excited for this. There's a quote I cannot find that I miss greatly, but maybe a Whitehead quote or some such about Civilization being measured by that which it doesn't have to think about, that which it takes for granted. It's always struck me as powerful, but giving ourselves the ability to forget & un-learn does not tempt me in development. We are the builders, and this great rich pool of possibilities is rarely improved with merely forgetting & becoming an exclusively higher-level operator. Depth is deeply rewarding in development.
Let's talk about the cloud some, & this pool of capabilities we are so delightfully placed at the helm of, and the cloud's influence on these capabilities.
> Somewhat ironically, software development is one of a vanishingly small subset of knowledge jobs for which the main work tool hasn't moved to the cloud. We still write code locally, thus we're constrained to things that work in the same way both locally and in the cloud. Thus, adapter tools like Docker.
It's super hard for me to imagine a replacement, not because replacements won't be great, but because replacements will have a hard time becoming core knowledge for the software development world.
(It's problematic because great openness let us roam too freely unboundedly but,) one of the greatest glories of software development is how unboundedly open & downright democratic it is. We use software until it stops serving us well, and then we either roll up our sleeves, dive in & improve it, or start something else entirely. But we can keep drawing from the same pool, from the many many possibilities & ideas which all interrelate & support each other, to shape new ideas & give life to new forms.
The authors premise, to me, feels like a proposition that we will be so well served the cloud that we can just leave where we are behind. This is about not needing the pool of knowledge or capabilities we have, the systems we have, because we'll be working somewhere else.
In many ways though, that to me sounds like declaring that the future is different, therefore we need a new primordial soup to start from. 'Using only ATCG for genetics is a paradigm that must be eclipsed!'
But you know what? Someone builds that new place. And in 98% of cases, the same old fundamentals & tools are still at play, underneath the new tools, underneath the new abstraction. Larry Wall (in Perl is the first post-modern language) would say: the truth is our systems will always be post-modern. Modernity's shining image of itself as brilliant sprung from nothing novelty is rarely true. New ideas more often than not creative re-applications and re-mixed of old materials.
Rather than simply say that a new paradigm is probably not so new, I think there's a deeper challenge, which is: how does a new paradigm ever rise to such a height that it becomes well known? How do we adopt a new paradigm & start teaching it & using it? How does it become the next thing?
We are very well served by the cloud. It's presence as a system of services, as far off, maintained-by-other-people, no-longer-our-problem miracle working wondermachine is all true & very well reported and it is coming for everything and everyone doing work today.
But I'm not at all afraid, because, in 99.99% of cases, these vast neo-mainframes have no way to pass on their genetics. They are alone and isolated and developers cannot get into their bowels. There is no "The Midnight Computer Wiring Society" of the cloud, and there never can be and there never will be because the cloud until the PC and the tools we have here is about control & orchestration & order & rules, and there's no permission in the cloud to go develop your own culture, to become a new wave, to change everything for everyone (including the other devs) because you are just one lone neo-mainframe, just a couple of your own ideas that you're calling your paradigm & building your cloud by, but in a huge number of cases your ability to interact with talk with share with other people also doing cloud or to enable other people to try to cloud like you do is exceedingly small. Clouds are all unique and alone & they have a much harder time spreading socially.
How does a new cloudy paradigm ever get the ball rolling? What are it's central tenants & beliefs that make it a flexible, malleable, swiss-army knife where all developers everywhere have even more power & creativity- not just at building applications atop it but enhancing & growing & exploring the platform as well? I do think eventually we will find new things to make core, to form real communities & new shared basis upon (I think Kubernetes' apiserver+controller paradigm is probably a core construct in the future, for example). There's early early signs we are civilizing what so many hyper & not so scale cloud technologies have frontiersed. (But oh it's so early, and it's at so much more depth where this happens than the shallow 'your workflow will be replaced' message of this article).
I think there's a lot of good peering into the mid-horizon to do, that the attempt to peer forwards are good, & commend this article. It's right to ask (sic):
> Rethinking these abstractions to be native to the new world let's us start over and redefine the abstractions for what we need? re else of note).
But I highly doubt we will really get release/escape from the past. The article tries to question the past 50 years, and I think perhaps yes we might diminish it, it might not be at the forefront forever. But I have a hard time imagining enough real value or enough real difference- even if we switch to Zircon or KataOS or Zephyr or Genode or the next thing, I tend to think most existing abstractions will largely remain, perhaps mutated some, some more prevalent than others, and that the view will not really end up looking that very different. Platform will continue, yes changing, but also in many ways similar.
The above all speaks to a fairly slow-building evolutionary view of the future. That said, I think we really dropped the ball on trying to bring software development online, have really had a shitty unambitious maxima we've been stuck at. We still write a crap ton of code that's just driving http clients. gRPC is still deeply one process talking to one service, a very convenient way to still manually write individual send(call)/receive(return) calls/streams. Cap'n'Proto dared to dream a little more multi-service, to follow somewhat after E-language, but never materialized 3-plus-way communication.
The long hangover after CORBA and SOAP blew out & got eaten by simple-is-better ReST has turned into a forgetting, to not trying. The idea of finding new abstractions is interesting, actually (contrary to first part of my rant), but programming language design has remained so focused within the language that we don't have the creativity to expose & play earnestly with the abstractions we have. Language design has focused near exclusively on building better processes, and without integrative cross-system rework, without a much higher scope of change desired. We're just shuffling the cards again and again with the same base system; it's all different syntax spins, different ergonomics, maybe a new safety guarantee (and boy are people excited about that!) but all the same underlying patterns, variously cloaked. A dull post-modernism. If there is change, real change, I think it comes from going back & re-trying an E-lang or an Erlang, or more generally working aggressively towards multi-system. And much of that could just be taking what we do inside processes & doing it on the web/net. I ask myself regularly, and alas, I haven't gotten around to fucking around and finding out: what would a EcmaScript/JavaScript Promise look like, but on the web? This kind of primitive material makes total sense to developers, but we don't really express these things in systems ways; there's the inner process world, & outer systems world, and it's not abstractions we need per se: we just need to tear down the veil between these two realms. The process already has the materials to make a great cloud, we just haven't opened the box yet.