Oh you sweet child, you haven't seen complexity until you have deployed J2EE services generated using WSDL, running in an OSGi container, interfaced with EJBs packaged using Ant and deployed into GlassFish, integrated with IIS running on a Windows Server.

Try finding a starter kit for that.

I did Java EE, Glassfish, JSF/IceFaces for my first job and at the same time I could have ftp'd a PHP/Perl script on a free host and called it a day (had I known better).

I think Devops has a played a huge role in why things have gotten more complicated. They want Kubernetes to be the solution to all problems. Heroku has fallen out of fashion for literally no reason. They won't even consider ElasticBeanstalk or App runner for even the smallest of projects. Companies would rather hire a 200k plus benefits person instead of paying a few hundred dollars more a month on hosted solutions. I've honestly never seen anything like it.

You're oversimplifying the problem and/or not viewing the problem for the organization. Maybe for an individual developer it's easier to create a Dyno or a Beanstalk deployment, but it's not necessarily easier for the organization.

IMO, Kubernetes solves a lot of issues for both camps but most of the Kubernetes detractors I've met are only hostile to it because of ignorance, not any reasonable criticisms. "It's different, it's hard or it's something I don't know" aren't valid excuses.

My main complaint about Kubernetes is that it's basically a https://en.wikipedia.org/wiki/Galactic_algorithm - unless you're FAANG-scale, the extra overhead and complexity it imposes to use it at all massively outweigh its benefits. And it's usually not that it's different things that you have to understand, but rather that it's more things, to accomplish the same final goal. E.g., to set up networking without Kubernetes, you need to understand traditional networking architecture, but to set up networking with Kubernetes, you need to understand traditional networking Architecture and the Kubernetes way of doing networking.

Easier to deploy for the developers is better for the organization. The dev team should own this process since they are the ones that probably have to support it. Objectively, deploying on any of the more managed AWS services, App Runner, ECR, Beanstalk is better for the org. Less risk if someone is leaving, vs the 3 terabytes of Kubernetes scripts required. The team knows how to manage and deploy their stuff so when they're on call they know how to fix it. Why would someone learn Kubernetes when you can spend 5 minutes learning Beanstalk or App Runner?

Things were never a problem when devs just managed this stuff. We used Heroku or Capistrano on plain EC2 and it was fine. Once you hit scale and things become more expensive then Docker on ECR / Fargate is fine. This whole dev ops fad just has to go.

Anecdotally, I was part of an organization that was all in on Beanstalk (hundreds of apps) that did a full migration to Kubernetes and we ended up reducing our spend, deploy times and pipeline complexity by half. We're talking a total yearly cloud spend probably near 9 digits. Moving to Kubernetes vastly simplified our lives and developers were able to learn it pretty easily. Development team was in the low hundreds.

So for every story I hear about "it was better back then", I just remember how much it was a horror for us and find it really hard to believe people voicing these options are actually arguing in good faith.

If that's your experience with it, then it sounds like a success and I'll concede that.

My experience is: devops is introduced to 2 orgs I was at (much smaller dev teams) to save money. Even though hiring devops negates any savings, but that's another story. They introduce Kubernetes (even though existing solutions were fine) as the flavor of the month to build up their resumes. They all leave after 6 months to a year and dev teams are stuck with the cleanup trying support and be in call for things we had no say in. It's harder to see logs, get a rails console, deploy, debug, etc.

Meanwhile they also spent 6 months trying to get PR apps and staging environments at one company. Something that is one click on Heroku.

If we didn't have Heroku at these companies when we started out we would have literally failed. End of story.

Personal rant: the solution to fixing a broken k8s installation made by a previous team in an attempt at some sort "enterprisey" lock-in is, apparently, wiping the clusters and creating a new k8s from scratch.

This isn't the future I wanted, let's solve these problems in some other way. K8s seems to be the second-system effect in all the worst ways.

Sounds like an unfortunate edge case that shouldn't detract from all the good things Kubernetes affords.

I haven't seen any good things brought by Kubernetes.

The frontend answer to complexity had to do with young/new devs fomo’ing into resume driven development and augmenting their growing identity/ego as a developer.

But this Java shit, that has to all come from enterprise solutions right? I can’t imagine there were tons of influencer-like blog posts that JavaScript had. What hell hole did those Java ideas even come from?

Blah. One way or another, people manage to fuck everything up. We just can’t have nice things.

I think some of this is because it's way harder to write a new library for a well-established language & ecosystem and become well-known for it, than it is to do the same in a new ecosystem. This goes for languages and frameworks both—React, for example, generated tons of opportunities like this, to become known for something by basically solving a problem that'd already been solved 1,000 times—but with React.

Like, if you want to build your résumé and get offered some of the cushier programming-adjacent gigs and get to present at prominent conferences and generally become A Name, how likely is that to happen if you try to launch some big web framework, or an auth library, or an ORM—in Ruby? Or Java? Nah, too crowded.

But if you pick some up-and-coming language (that's up and coming in part because everyone involved is doing exactly what you're trying to do) or hook your wagon to some new framework's (or database's, et c.) hype cycle, well, now you're talking. As a bonus, many of these projects require little or no novelty—it's all well-trod territory. Just half-competently do what hundreds of others have already done, succeed at promoting it (tellingly, this is the harder part) and you're a rockstar. And hell, maybe doing all that does demonstrate some kind of remarkable and valuable skill, for all I know!

But the point is, if that's the kind of thing you're trying to do, you need churn, and a bunch of people all trying to accomplish the above drives that very churn. Very little of that activity actually advances the industry, and the churn itself is a huge drag on everything.

15-25 years ago, there were hip-for-the-time tech influencer bloggers who would write all sorts of stuff about whatever new frameworks and tech were available in the Java world.

If you take all the scary Java Enterprise buzzwords in one of the previous posts on this thread, and look them up, you'll find that they all came about, roughly, between 1999 and 2007.

I don't fully remember what all the hip influencer tech blogs were during that era (I recall "Joel on Software" having been important), but I do remember that people who paid more attention to such things were really into "Enterprise Java Beans" at the time.

So the JVM world might suffer from a bit of an analogous problem. But that world's complexity explosion was decades ago.

My feeling is that backend has always been complex, depending on the scale you were trying to operate at (and how much scale your framework and ecosystem forced you to think about whether you wanted to or not). The best you can do to mitigate it is continually ask "Do I really need my backend to handle the scale I'm designing it for?" and "Is all of the complexity in my preferred stack still necessary, or can some of it be hidden/automated?"

Heh. So propaganda (or marketing as we now know it) has always worked.

When are we ever going to get hip to that reality?

I try to remember that this decades cool hip software trends might become the next decade's enterprise Java.

And that's the outcome for the successful stuff!

(Doesn't mean you shouldn't get into it though -- if I'd jumped on enterprise java 20 years ago, I'd probably be richer than I ended up by just working on the software I was interested in)

Does SimpleBeanFactoryAwareAspectInstanceFactory not simply spring to your mind when you're coding solutions? /s

Can you explain what that is or what’s it’s trying to do?

The response I dont see represented is that we were more ambitious in the past. There were all kinds of negative effects, bad tools & out-of-touch architecting that was begot; we can acknowledge that, certainly. But that almost all representation of Java here so emphatically emphasize only the negatively bias, without recongizing also some of the idealism, hope, & progress is to our loss.

We were also trying really cool things with many of these java efforts. Rather than a main() routine that imperatively did everything, there were future-sightes notions that we could have a bunch of ambient systems available & interconnected. OSGI solved some thorny library versioning problems, and as a bonus allowed for a very strongly manageable runtime for modules. EJB brought objects up to a higher level, a greater management tier with more interoperability of objects across instances. We were interested in trying more, in tackling harder problems, that microservice era has masked over, deliberately chosen to ignore.

Over time we got more refined forms of many of these systems. Maven calmed much of the ad-hoc madness of Ant. CDI was a more reasonable EJB with event more competent management. JSP kept getting better component libs. OSGi actually was... decent all along, sometimes legit useful.

But the idea that Java was absurdly troubled seems to endure & be enormously popular hatred/FUD to spread. The fads of today are exactly the opposite of the better managed more flexible runtime trend, where the JVM keeps rising as a more capable, flexible, malleable, adaptable, controllable way to tackle processes than the processes at an OS level. And certainly, there's a lot of ease we've bought ourselves by doing much much less. Statically compiled binaries are an example of yet another means of doing less, of finding easy routes; powerful simplifications that have greatly eased many folk's lives.

Less is great, sure. Isolated container instances with dedicated apps is pretty easy to deal with. But there are still constant undercurrents, pushing us towards more interesting complected runtime arraingements, where virtual machines (not fully virtualized os'es but systems like glassfish or erlang, where many sub-processes are co-resident) come back. The idea of very quick very easy to spawn sub-processes allows for interesting security & modularity, where-as today services are almost all monolithic app servers that have complicated multi-user authentication & authorization concerns. We could make simpler safer software if we had more complex runtimes; it's a trade-off, and im deciding we have gone too-deep, unleashed monsters, we have perhaps too polarized ourselves against many of the great capabilities we one tried for.

Right? We already over the explosion. Backend development is not complex, it's mature. With spring boot + initializer + bean based configuration, REST interfaces everywhere, maven and gradle are both great what's not to like?

Can you tell me exactly what kind of beans you have in your simple spring boot app, and how request processing works step by step?

I’m 27 and my first job was OSGI at a Java shop.

That was some convoluted shit. Class path errors, coupled to blueprint XML and fancy IDE stuff.

I’ve worked in different languages and frameworks since and love the simplicity of Go. Excited to learn Rust

Hum... I've done this, and no, the "multiple interdependent microservices" over kubernetes architecture is much worse.

No mention of CORBA? I think it is on the top of my personal "what the heck is going on there and why on Earth anybody would want it to happen to them?" list.

I used CORBA once about 20 years ago to talk between a C++ server and a Java EJB server. It was pretty ok? It beat hand wiring TCP protocols. I'm not sure hand-wiring REST calls is much better. At least we have gRPC now.

My first real paid job was using CORBA to communicate from the Java applet running in the browser to the Java proxy running on the web server and then to an app server. Later I also worked with RMI, DCOM, etc.

I also used both DCOM and CORBA at later jobs for Soft-Realtime/Embedded distributed systems.

I’d say statically typed IDLs are much better than all these REST/HTTP/JSON/CRUD approach. That’s why I prefer GraphQL and gRPC/ProtoBuffers.

Not the least important thing is that having a statically-typed and well documented schema reduce the amount of required communication between the developers, and also reduces the number of misunderstandings, mistakes, and bugs.

I really don't think this is a productive question or will produce productive answers that aren't tribal and arbitrary.

Some folks will blame "the system" or "the man", others will claim it's not hard and you just need to RTFM, still others will explain how much harder it was "back in their day", and some folks will question the question (like me).

But nobody is going to drop some single piece of information or knowledge that'll "answer" the question; "In 2004 Linus Torvalds declared that all backend systems must contain three additional technologies than the currently mandated two." If only it were that simple.

Things are hard until they aren't, complex until they're simple, convoluted until they're straightforward as can be. The line is different for everyone, so nobody knows what your line is, which means nobody's going to answer the question in a way that will satisfy you, and if they do, it'll only be helpful to a few folks (your folks) and possibly actively harmful to others.

Maybe if you had a specific question that illuminated where you're coming from, folks could provide better answers...

Lovely answer, should be cross posted to the frontend thread :)

I see your edit, fair enough!

I tend to agree with you; stuff that felt impossible for me to understand at one point is now second nature to me, and over time I've learned how to do more. UI and BE, as well as orchestration and infra.

AWS used to scared me, and now I maintain a dozen microservices (and I may slightly regret breaking everything up lmao) for my startup in AWS by myself because Terraform is pretty easy to pick up and at this point AWS makes a lot of sense (hello Dunning-Kruger, my oldest colleague).

Frontend terrified me because I didn't know how to compose UI cleanly (not just JS), but then I learned React and now feel confident I can build, maintain, and deploy complex frontends because honestly it wasn't that hard to grok.

The single strongest piece of advice I've been given, the single idea that gets me through like... 99% of work related strife is this:

How you feel now is not permanent and will not last forever.

I guess the more poetic way people say this is, "This too shall pass"[0] but I just think of Gandalf so...

[0] https://en.wikipedia.org/wiki/This_too_shall_pass

worked at two big tech companies. there is perverse incentive to grow your organization. gets you to director or VP. if you engineer it goes to your promotion packet to get you promoted.

pattern i saw is teams do not build customer feature but work on “efficiency” or “cost saving” project. Ok, so you build something that other teams should use because it will save them time. Sounds very good on paper.

so you do these things to entrench your team DEEP into the tech stack. now people depend on you you use it as reason to get even more head count. ok now the engineers got their promotion but they struggle to support these teams when things not working. So they leave. they got promo, why stay?

managers now use all this ammo to ask for even more hc. it is self serving cycle. Now..development harder. harder to push new features. new people from college come. first they have imposter syndrome and think this is all great and they just not smart to get it. within few year they get it..see processes and tools are idiotic. they learn to play the same game. circle of life continue

This is also encouraged by vendors. Old-school "boring" solutions (that you can't rent-seek over) are shunned in favour of proprietary reinventions of the wheel that do the same but poorly or more expensive. This in turn incentivises "engineers" to do resume-driven-development and implement those suboptimal products/services at their own company, and the cycle repeats.

The scope of back-end development has grown considerably. In fact, it's precisely "one level of indirection" bigger: "commit is the new save", "push is the new commit", and "repo is the new module". Plus, "full-stack development" now includes containers and orchestration! Which is, of course, insane: distributed systems are, in general, not well understood. Expecting people to understand front-end, back-end, and then just tacking on an academically challenging area is beyond foolish.

I strongly believe that we're now at the limit of human cognitive ability (as evidenced by the tight, and tightening, employment market) and we human programmers are painting ourselves into a complexity corner and we will weep when our employers happily hire the only minds capable of dealing with mechanical complexity, artificial ones.

I have never met a full-stack developer that is comfortable with the full stack. I am also pretty sure full-stack was coined by recruiters and marketing folks to create a technical middle class.

Really? In how many years experience?

I've met many full stack devs who are comfortable from DB (db design and stored procedures, sql) - backend code (java, spring framework) backend services (consuming first party and third party APIs) - frontend code (javascript, typescript, jquery, angular, react) - frontend services (Google analytics, GTM, etc. )

Edit: forgot to add in CI, CD, deployment platform skills, server less functions etc

The only time I met half competent full-stack devs was when the stack was limited to JavaScript on front and back ends along with API calls to serverless functions (eg Cognito) for "more complex" things. Never met one that could quit vim or design a database with anything other than varchars.

That is not full-stack, that is knowing a syntax that happens to have ended at both ends of the stack.

Hmm. Yes. https://dev.to/cubiclebuddha/i-m-sorry-but-this-full-stack-m...

I dont even have a clue what a full-stack even means these days, its just a resume wank-word somewhere between "cloud" and "framework"

"Full stack developer" is supposed to mean someone who will just figure shit out irrespective of where the bug/enhancement resides.

I say this, half tongue-in-cheek.

It means that you're supposed to be equally comfortable with VLSI design as you are with Photoshop and everything in between.

Is the employment market really tight?

I think the backend development is still simple it's the deployment that has grown overly complex with Kubernetes/Docker and CI/CD pipelines

hard agree. then all the choices of cloud servers, learning those, getting up to speed and actually being productive. it's good there's DevOps, but... yeah good luck explaining to pretty much any business without one that they need 'em.

> "Why can't you do it?"

Because everyone is prentending that they need to be infinetly scalable like the FAANGs, exacerbated by Cloud providers marketing spiel. In reality nobody knows how to build distributed systems but everybody needs to build them as resume fodder, because they might one day apply for a job at a FAANG... sigh...I sometimes pine back to the days where I saturated the NICs of two quadcore servers with C++ std::unordered_map and FastCGI. Especially the days when I ask what problems the service mesh solves which isn't solvable with internal DNS and loadbalancers in our use case.

BE is simultaneously easier than ever and harder than ever. The "new" complexity in the BE is stuff that has always been there. However, in the past, the infrastructure required so much manual work that it was handled completely independently from the BE.

Now, I can write a BE, configure a Dockerfile, and deploy to a server nearly anywhere in the world without leaving my chair.

I don't think it has. We just have a fashion trend where people love to brag about managing 100 micro-services using Kubernetes. But there's still companies out there where the entire back-end is a single binary on a single beefy server, potentially with a hot spare server running alongside it.

Would you happen to know where I can find such companies?

Most government entities, big banks and companies using mainframes and AS/400 systems

There are mainframes & COBOL still around, but speaking as someone who's never worked in the sort of place that uses the trendy things mentioned on HN, I would say Oracle and things built on Oracle are my concept of "normal".

I don't know how decision makers think, but I get a general sense that Oracle is the safe "you can't be fired for choosing..." choice.

I don't think these places have just one server, though, far from it.

What, so you could DDoS them? Heh...

The products we're building are more complex than they used to be, but I don't know if its true that backend development has inherently become more complex than it was in the past.

I've worked on everything from dinosaurs older than me to new greenfield projects using all the latest tech. I think over-complexity emerges in many ways, notably: too much abstraction, ignorance, dogma, and just straight up carelessness.

I think the abstraction problem is the most common and severe though. In enterprise software, I've spent an entire day in the past combing through several layers of services and repositories just to get the full picture of how a response object is being created for a single controller method.

Let me give another example: think about the great debate over ORM vs No-ORM. It's fundamentally a battle about the balances and costs of abstraction in software development. On one hand, you have a (hopefully) simple interface for defining a db's schema, but do I really want to obfuscate the developer from the implementation details of that `findBy` method and trust that the ORM will be 1) used correctly and 2) generate a performant, sensible query? What about security handling? Should I have my developers concerned about preventing potential SQL injections, or just let the ORM take care of that too? The answer depends on a lot of things.

Anyway, I believe for every abstraction:

- something significantly useful should be added (this seems like a truism as I write this, but i've seen a startling amount of useless abstractions)

- the cost should be carefully calculated

- understand where complexity is being added and subtracted

I believe thinking about abstraction in this way would've cut down a lot of the complexity I've seen in softwares past.

To respond to OP's edit: frontend catches more shit because it's largely gotten worse at the same time as getting more complex. This isn't (only) outsiders looking in leveling that criticism—it's people who worked on GUI software in the 90s looking at the state of things and wondering why it's both constantly changing and, in many ways, worse than it was back then. Slower, worse tools producing slower, worse software, while constantly demanding that frontend developers run on a churn-treadmill for—it sure seems—no good reason.

Backend suffers from some similar issues, but not to the same degree. Frontend seems to be subject not just to the effects of the apparently-eternal fat/thin client cycle, as the backend also is, but to some other pressures that lead it to be constantly changing but not really getting better, and even getting worse over time.

> frontend catches more shit because it's largely gotten worse at the same time as getting more complex.

The "microservices by fad" based architecture usually comes with a huge decrease in performance and an even large one in reliability. The NoSQL for no reason fad came with horrible performance and some very bad data losses. Every once in a while we get some data leakage story here, certainly many are due to complexity (but nobody can tell which ones).

Frontend is not alone suffering for the complexity.

> The "microservices by fad" based architecture usually comes with a huge decrease in performance and an even large one in reliability.

Good point, that one was a big mess.

> The NoSQL for no reason fad came with horrible performance and some very bad data losses.

100% agreed. My perception at the time, though, was that it was largely driven by frontend folks who'd come to the backend via NodeJS buying (especially) MongoDB's marketing hype. There was a real "finally, all those barriers the backend folks had for no reason whatsoever are gone and this is easy now—better, even, you dinosaur, have you seen the marketing, it makes it so clear that this is the future—so I can be full-stack without having to learn much!" vibe to the whole thing. But maybe that's just what I happened to see and that wasn't really the main cause of that going as far as it did.

Maybe it's a general comment to both back-end and front-end. To me it seems that down in the trenches coding is getting more complex, even though frameworks hide more and more complexity. Languages and frameworks gain features and they don't necessarily shed old ones. I've been working in C# for near 15 years. Each C# and .net release brings more to the table. Often these are features, which simplify things, at the same time the old ways still work. Googling often returns N solutions, each from a different era. I imagine it can be confusing for newcomers to even start. As for the growing complexity in the world of backend, I think, when needed, the best tool against it, is pragmatism.

sarbanes oxley, other reporting requirements. As systems become faster, companies need to save more data. Where I work, we need to be able to reproduce finance reports created years ago using the original data. Doing that makes things far more complex.

When I started out decades ago, systems were far less complex.

Note, the IRS is our best friend. Without them, most of us would be doing something else (if we are very lucky).

Because development jobs pays too damn well, and are too easy to get. Someone famous once said something along the lines of "any idiot can make a thing more complicated, more complex, but it takes the stroke of genius to go in the opposite direction"

Basically, backend dev is so complicated because it's done by fools. When it's not done by fools, it's either not complex or has some very difficult requirements, perhaps sometimes both.

Edit to add, I say this as a foolish backend Dev myself, no offence meant to my own kin, of course!

I think a lot of things actually became easier lately. Deployment? Docker, k8s and cheap virtualization made it much easier than before. Exchanging data and interoperation? Finally people are starting to use comprehensible things like REST and GRPC, instead of trying to invent some byzantine "Enterprise Solution". C/Assembly is no longer the only option if you want performance. Sure, there are more tools around now, but I think the tools themselves are getting somewhat better and more sane.

Backend development? Have you seen what SRE/DevOps people have to deal with?

People literally use hundreds of different tools for no reason at all.

> People literally use hundreds of different tools for no reason at all.

Slightly Disagree. Resume driven development (RDD) is a thing.

Microservices and services in general where they weren't needed in the first place. Not saying they don't have their use but most companies just ride the hype train and start using technologies for the sake of using them. A couple of years later they end up with an unmanageable spaghetti of tightly coupled services.

Follow up question. Did backend development explode in complexity? What's your source?

I think it's artificial, and the reasons are ego inflation and job security. Nobody wants to admit that all their abstractions and knowledge are made-up and net-negative. But they largely are.

I think complexity is conserved. You can move complexity around in an application but in the end someone has to deal with it.

E.g. the whole microservices vs monoliths debate. While a single microservice is usually relatively simple, you have 10s, 100s or even 1000s of microservices and the entire system design becomes crucial. With a monolith you don't have to worry as much about distributed services and inter-service communication but designing a robust monolith is still a complex process.

Because managing Linux servers is now less complex than orchestrating containers. But people didn’t want to do the former.

The two biggest wrenches that get thrown into a backend system and blow up complexity in my experience:

- 100x in growth, aka scaling.

- Needing to set up reporting data pipelines. Metrics, auditing, 3rd party uses.

Does the frontend need to worry about either of these? Maybe for scaling, they need to do some CDN work, or is that still the backend developer's job?

Scaling is over-rated and over-sold to most businesses who generally have predictable, stable or no growth over their lifetimes and then see their new system running at 1% capacity for its lifetime.

I agree. But I've worked at startups and it's been a real issue that comes up when the business really starts to grow.

Best case complexity is not discouraged.

Worst case complexity is incentivized in more ways than this comment has room for.

We live in a society.

It is almost as if complexity is increasing regardless of the software field one is in. Maybe it's due to the fact that engineers exist to build complexity, stacking layers is what we do day to day.

Funny. I am a backend engineer and think so about front end.

Try Frontend development!

i think a couple of reasons:

  - 1. Nodejs
  - 2. Nodejs

I have seen much more unnecessary complexity from backend developers shoehorning GoLang microservices, kafka and k8s... Than from those who create Express.js apps

"Enterprise" Software development