I have seen half a dozen otherwise useful projects retired or criticized because "the original developers are all gone and no one understands it any more."
The code worked, and continued to work, and didn't need really active development, so the org decided the project's programmers should move on to something new, or that the knowledge they have isn't especially valuable, and the programmers aren't worth keeping happy.
And a couple of years later, no one knows how to update the code for a changing outside environment, updated input styles or whatever. And all that investment is gone.
Actually, in most cases - no, they can't. Or, rather, they can't figure it out as fast as they're "needed" to.
> if there's a real need with no alternative
What happens instead is that somebody with no real knowledge of software development says, "we need this change made in two weeks. We need this." That this might be impossible for any human being, living or dead, is completely ignored. If the programmer this is dumped on even suggests making any clarifying improvements to the code, even that is rejected because "we need" this.
And two weeks come and go, something was changed because there was a deadline, but the something that was changed broke a dozen other things nobody even remembers were there, and now there are bugs waiting to be discovered by unsuspecting users.
> it's a matter of incentives and prioritization
I know what you mean - it is, technically possible but it's never feasible because it's not predictable.
When you run up against this wall, it is often easier to write new software than to maintain the legacy code. Long term you are duplicating a ton of work at huge expense, but that is frequently the fate of successful projects: to be replaced with something sub-optimal that is nonetheless more tractable because there is less of it at the beginning and rapid feature development makes it easy to demonstrate value.
More often I've seen that all that fast development time is spent trying to replicate features of the old system that were missed or deferred when designing the new system. Strangely, when they have achieved feature parity, the new system often doesn't seem so easy to work with.
Loyalty gets you very little in today's climate. Why should I spend time understanding the current system that was developed on yesterdays hotness?
I need to spend my time here developing on the latest trends on a new solution (learning more from running into the same walls the current system did) - not learning the domain knowledge + old framework, so I can move on after a year and earn more at the next place.
Is this good for the world? No. But unless you want to swing the hammock and work at institions that value incumbents, the incentive is to do whatever work improves your resume, not what benefits the company.
I don't personally like it, and I often see old systems that are just fine get thrown out and replaced with the a less featured replacement that grow into the same kludge as the first, but I can't ignore that when you do that you learn way more and have better examples to point at in interviews.
> I don't buy this whole "nobody knows how to update the code" reasoning
Me too. At best, they would need the previous guys only in one condition: very short time constraint (have to figure it out "today"). Otherwise, it's not hard to figure out at all, even codes from a complete idiot. If he can code the program that worked, the code is very likely be figured out by experienced people very soon.
Sure, you can figure out how to change it to do some very specific thing that somebody identifies as needed. What you probably can't do (at least not in any reasonable amount of time) is figure out all the things you're going to break in making this change. If they followed good software development practices, left a good comprehensive set of unit tests and documentation then you could do that - but they didn't.
> even codes from a complete idiot
If I was wrong, there wouldn't be anything wrong with global variables or goto statements - any code that "works" would be isomorphic to any other code that "works".
That's how it works even with the same guys on projects. If you think it would break (no tests found), write the tests first. I don't believe someone would write a complete mess million line of codes that works without some tests and docs, that would make him kinda genius.
And global vars and goto is not hard to figure out, it's hairy for sure.
100% agree. One of my friends, in her first software engineering role, was handed a binder from 1987 and told to update some things.
Other coworkers had interacted with the system a little, but not extensively. It was a bunch of C for (rock & ore) mining software. It was intimidating and it took time, but she cracked it, and then happily (& justifiably) bragged that she added features to code that had been written before she was born!
I'd be proud if I were the original author, too. My code lived long enough that someone who wasn't even born when I wrote it was able to add features to it.
> I don't buy this whole "nobody knows how to update the code" reasoning
As an anecdote, your example demonstrates the truth of the reasoning. Absent extraordinary or unorthodox effort, such as patching the executable manually, programs do suffer from loss of understanding. Yes, you made it work, but can you really claim to understand the program and how it was written?
Of course, given enough time and effort, most code can be reverse engineered to a level of understanding good enough to work on it. In practice, most organizations are going to decide it's better to spend time and money on a replacement.
> I'm sure the source code wasn't lost, it would have been on a tape backup that people were too lazy to try and recover.
That's certainly one, rather uncharitable, way to look at it. But what constitutes sufficient effort, given time and money constraints in the real world? Unless that backup literally contained the only copy of information essential to the continued existence of the company, there's a limit. And what if they did find the backup, but it was on a magnetic medium that had degraded to unreadability?
I've made my career working on legacy code. There are definitely ways to understand systems even when nobody who originally worked on them is around, but business constraints have always determined whether or not the effort proceeds or is abandoned.
I think most devs don't want to figure it out. Let's face it, working with legacy code is just not fun unless the code is well written (and even then a lot of devs still don't want to touch it because it's "old tech" and doesn't help their resume).
Flip that: most companies don't want to pay what it costs to get someone to do this kind of work. Most devs would do it if the incentives were good enough.
Maybe in some companies developers work very hard and are short on time. Maybe you are in one of these companies yourself.
In my own experience, devs aren't often overworked and do have the time to do stuff like this. They just rather not spend the time on it for various reasons.
What incentives would get you to work on an old legacy project?
Are there consultancies that specialize in this kind of thing? "Help us kick this legacy can down the road the bit, nobody here understands it and it's not worth hiring for." Asking because I enjoy this sort of work :)
It's probably not much of a thing because companies don't want to spend a lot of money on maintaining legacy code, and are only willing to throw a lot of money at a problem when it's a huge emergency, but by then it's too late and your consultancy wouldn't be able to get the job done in time no matter how good you are.
Yeah, every consultancy. The ones that look like don't do that kind of work still do, because they somehow have the magic ability to turn a greenfield project into legacy code in the space of about 2 months.
My very first commercial program was a Fortran 90 thing that could write its own source code out to the filesystem if asked. Updated itself as part of the build. I still quite like that as a feature.
> ... Why would that be useful? Don't you have the source code in front of your face as you are working on it?
Maybe not, actually. Assuming that the code was written in the 90s, there probably wasn't any version control (only backups at specific milestones).
With no version control, there's a 50:50 chance that the source code you found for some deployed version is not the same as the code that was backed up.
Replying to a post saying
> the source code was "lost"
Sure, I had source that I had written. In source control even, was using fossil. But I thought my binary might end up somewhere decoupled from that source, with some other engineer tasked to modify it, and I hope they find the dump source option listed under -help useful.
> I opened up the executable in ollydbg, patched the executable so it worked properly, and life went on.
I've done simple binary patching, and thank my lucky stars that:
1. The replacement configuration value strings were shorter than the original
2. It was written in C, using nul-terminated C strings
If this was almost any other language, I would have more work even if the replacement string was shorter, because I'd have to fix a bunch of offsets everywhere.
Yes, but the comment I replied to implied that software will just stop working. If software has been working for years and then stops working, it's usually something fairly small than can be hacked quickly.
In my specific case, the tool had a whole bunch of hardcoded values. It checked authorization against an LDAP server, that all changed. Connected to a sql sever that moved + credentials changed. Assumed certain files existed on certain network drives which were moved.
Technically, it could have been solved by faking all the infrastructure. It also could have been solved by digging out the source code from tape drives. But everyone was running around saying it was "impossible". I was shocked how quickly people tossed out "impossible".
It could have been done with a simple hex editor, but the new values were longer than the old values, so the strings didn't fit. I put the new strings at the end of the .data section and then just changed a bunch of offsets.
I agree that if you're writing the next version of the software, you can't do that quickly when nobody knows how the current version works. But chances are, you're not writing the next version of the software quickly even without churn.
it's not really about how the software works. the source code says what it's doing. the thing is the source code is the product of a theory; a theory determines how you respond to the world. if that theory is lost, then updating--in the long-term sense--becomes hard, because the theory that you form might (probably) be incompatible with the theory that made the software. this incompatibility might make producing functioning software difficult, but more likely, the mismatch will make choices that don't make sense to, say, users, appear to be the obvious thing to do.
Sometimes it's worse when they start tacking on changes in an inconsistent way.
It's always painfully obvious when a company is shipping a product that has been acquired multiple times, and/or hacked on by contractors that don't care about the past or future of the project. I'm looking at you Aruba, Cisco, and pretty much every network vendor.
Software product is delivering value, maintenance isn't needed much, people move on to other projects[0], people forget about it, then a couple of years later a change is needed.
But how would the first response in that "couple of years later" scenario be "throw away all that investment and start from scratch" instead of "dig into how it works"? That's not a long time, really.
[0] this is a good thing, IMO. If every project needs constant maintenance from its "owner" you as a company are going to rapidly run out of resources to do new things.
(There's a closely related thing, here, though, that as an individual or a low-to-medium level manager, keeping track of these risks and low-level maintenance areas is part of your job to find ways to keep it from ever turning into a disaster. At some point - 10 years, 15, 20... maybe the knowledge gap gets way too big for the current team - then get proactive.)
The real disaster is management. This thing was working perfectly, and therefore was ignored. Then it became a frantic crisis because nobody saw that some other change would break it.
And then they responded with an absence of reality. A two-week deadline that can't be moved? Um, no. Ever hear of reality? You might visit there sometime. It's a bit harsh on people with ridiculous demands, though, so adjust your attitude before you go.
And then, nobody can fix the existing one? You not only have the ridiculous demand, you rule out the only possible way of meeting it? What do you think is going to happen, the elves are going to work all night to magically create a solution?
This sounds like an epic management failure from start to finish.
> But how would the first response be "throw away all that investment and start from scratch" instead of "dig into how it works"?
Because start from scratch is more predictable ("estimable") than "dig into how it works". "Dig into how it works" is better in the long run, but will take longer.
All software management is laser focused on speed and predictability rather than long-term quality.
Two terms you may want to refresh your memory on: second system syndrome and Chesterton's fence. I've never seen a rewrite come in remotely on time, and I've seen quite a few fail altogether. Often what happens is, in trying to solve the problem again, the developers come to understand the original system, which already works. It becomes impossible to articulate tangible benefits for the rewrite and it gets shelved.
"Dig into how it works" may take longer than rewriting or it may not. And the rewrite may (and often does) run into unexpected difficulties too.
I think it's a mistake to make the "fix the old" or "rewrite from scratch" decision up front from a position of high uncertainty about the old system.
I'd go with a strategy of starting a timeboxed exploration of the old system with the objective of understanding it better, hopefully well enough to have a reasonable idea of what it will take to fix it. Then make the decision from a better informed position. Yes sometimes the decision may still be to rewrite because you now know that the old system will be excessively hard to change but in many cases you may find out that the old system wasn't as bad as expected.
This is exactly the case where I think starting from scratch is the most difficult to estimate. This thing we need to update is a mystery-box, right? You don't know the code. You don't know all its edge cases. Tons of potential bugs could be introduced (or re-introduced) if you change it in the wrong way.
Replace-from-scratch has all the same risk of getting things wrong because of that lack of knowledge and a bunch more new-code-writing on top of it. Estimating new development often fails; estimating it in the face of business requirement uncertainty is even worse.
Can you quantify that to the management in that case?
To certain styles of management focused on "always building new products" to guarantee their worth/value, they absolutely will get blinders on and believe "new products are easy and more predictable than maintenance of old products". Those blinders are hard to take off once they get into that sort of pattern. All of their processes are built around "new products". That starts to be the hammer that wants to dismantle old products because they don't do maintenance (that's someone else's problem) but they surely know how to build new products. They are the build new products group, that's what believe they do. Building new products is exciting and they can be passionate about it.
I also think that's generally the wrong way to work and does plenty of harm and creates a lot of unnecessary stress on development teams caught in those cycles. But I've certainly seen plenty of management teams that do work that way in the real world and maybe don't question it enough.
Rescuing everyone with a shiny new system also takes more resources and looks (as you kinda note, too) a lot more like a real project, so product managers and project managers and various other managers can get comfortably involved and do their thing, with plenty of people to manage (= more prestige) and new UI to show off in powerpoints (= more prestige, doesn't even matter if it functions worse than the old one) and ordinary, regular development milestones.
Learning the old system enough to keep working with it, meanwhile, looks a lot like a couple developers staring at screens for months without results that look impressive to anyone who matters (as far as granting rewards goes) and with very uneven progress. Plus they've gotta be pretty good, experienced developers (so, likely expensive), or the risk of breaking shit as they begin trying to change the system, goes up a ton.
I'd like to disagree with you there, because of this one data point ;-).
I worked on an application where I knew just enough to make spot changes, while the other developer who had made the most core changes continued on with the core. Then he left.
So as more core changes needed to be made, none OMG due to Production issues, that made me dip into more and more of the legacy code, and I grokked what was going on. I wasn't happy with it, but I understood it enough to know that to make it truly maintainable would require a total re-do. But the code that was there worked, and as time went on I even took other developers' bolt-on additions and made their functionality take advantage of the sub-optimal idioms the core used, so that at least if someone else had to make changes, they'd only have to deal with one pattern. I've made wiki pages to describe it, but still any other developer who works on it ends up asking question after question and complaining why the documentation isn't in the wiki (it is; they just don't want to look).
We eventually moved to another servlet container, and I filled in the missing functionality the new server lacked, still keeping the original core code.
I've done that with two other applications at our company as well. Why haven't I moved on? Hey, I'm older now, and it's an 8 to 5 job you can't easily find any more.
It's not "gone", it served its purpose. Software is born, lives, and dies. By the time a project is 3-4 years old, unless it is a core central service, the business (if it is competing successfully) has most likely moved far beyond the initial assumptions laid out at its' creation.
It sounds like you haven't been in the situation the parent is describing.
I have and still am. We have stuck one ancient piece of software in a docker container and have our fingers crossed that it continues to work. This software is OLD, early Perl 5 old. Nobody in the department understands Perl or wants to. The business (read: product owner) has deemed it unimportant to update or understand, but I assure you that if it stopped working, it would affect millions of tax payers, upper management would find out (read: VP, CEO), and heads would roll.
Everything is fine, until it isn't.
Your thinking from the perspective of a startup might be true, but from the perspective of a company that is many decades old and has a dozen layers of management... perhaps not so much
This is absolutely a failure of management. Both the general management and _especially_ the engineering management.
Eng management needs to convey that this is an Achilles' heel that is important to understand & be able to update.
No engs want to work on perl5? Bribe existing employees, hire someone who wants to work on perl5, or a darker third option.
The product owner can even spin this as them doing a good thing to future-proof the business, even if it means their team loses velocity on other projects.
Nobody in the department understands Perl or wants to.
And that is the problem.
I don't know how bad that code is - a lot of Perl code from that period is not good - but learning enough Perl to have a shot at it literally should take a competent developer a few minutes if you had someone explain it to you.
You might find the language weird. $ means "the". @ means "these". % indicates you're building a lookup. Which you then access with braces. So %foo is a lookup. And $foo{$bar} should be read "the foo of the bar...".
Early Perl 5 means it probably doesn't use references or objects. You do use regular expressions. While regular expressions are built into Perl in a way that makes them easy to use, Perl-like regular expressions are in pretty much every language. So the effort you need to use them is directly portable to Python, JavaScript, Java, etc, etc, etc.
Fluency takes a lot longer. But getting to the point of "I can puzzle my way through a language I don't know well" should be doable very quickly. And it is a skill that every developer should build. Because I guarantee that whatever language and toolset you are using now, you'll need to be able to switch at some point in your career. And practicing in advance will help you. Plus being the person who can puzzle out old ganky code in another language lets you be a hero in fun ways!
> the business (read: product owner) has deemed it unimportant to update or understand,
That's the problem. It's not on a random person in the department to spend their own time trying to solve a problem the business doesn't want invest time or money into solving. I don't see you offering to do this for free, so don't claim that's a problem for others.
I've been around the block enough to read between the lines on this one.
All of the talk about how old and bad the language is, and how nobody wants to learn it is tech speak for, "We can't support it as is, we need a rewrite!" Anyone who has been near tech for a long time knows "rewrite" means "tech is going to break it for unclear reasons and uncertain gain, and in a few years will want to rewrite it again. Tech people always underestimate the cost and risk of a rewrite, and also overestimate the benefits.
Business is not being stupid in saying that tech can't rewrite the software. The software is there, works, and is being relied on. Tech should figure out how to support it without a risky rewrite. Don't stand on, "We don't want to learn Perl." Learn how to be a polyglot. At least when it comes to reading. And to the extent that tech understands the value of being prepared, tech should prepare in advance. And figuring out how to read the code is most of that battle.
Now you accuse me of not offering to do this for free. But in fact I offered the best tip I could for a non-Perl programmer who is trying to puzzle their way through an early Perl 5 script. Given that I'm an internationally recognized expert in Perl, that tip probably does more good than you realize. Learning how to navigate legacy code in a legacy language is a learnable skill. And you learn it by trying, then finding you can. And no matter how much you don't wish to learn any particular legacy language, having the confidence to know you can is valuable.
> All of the talk about how old and bad the language is, and how nobody wants to learn it is tech speak for, "We can't support it as is, we need a rewrite!" Anyone who has been near tech for a long time knows "rewrite" means "tech is going to break it for unclear reasons and uncertain gain, and in a few years will want to rewrite it again. Tech people always underestimate the cost and risk of a rewrite, and also overestimate the benefits.
No, it really isn't. I'm happy to learn a new language. The reason is correctly stated by a sibling post here (WesolyKubeczek).
That we've been able to containerise it has limited a lot of the risk, and any upstream issues we have going forward we should be able to write some middleware to sit between it and transform the data into the format we need. We can extend the container to proxy any requests from the ancient application to the new middleware. We have a strategy, the product owner is happy with that, but we've outlined that maybe the solution won't work as we think, there are always unknown unknowns...
The issue is that if a problem does arise, it will be immediate and the effects will be there for as long as it takes to get a fix up and running, which could be weeks to who knows how long. That's a real risk in my eyes, but it wasn't to our product owner.
It will affect a lot of people, it's a pretty widely used service! It's entertainment, so people won't die if it fails.
The risk was deemed low enough by the product owner that we can just leave it in a container until it's finally sunsetted in 5-10 years (or so...). We've had 2 product owners since the OG product owner made that decision, so it could just be the product owner who made the decision thought they'd be long gone by the time it became an issue.
You're making assumptions about people you've never met, you don't know me and you don't know what motivates me.
You aren't as good at "reading between the lines" as you think you are.
---
From your profile: My main programming language is Perl
Oh, it all makes sense, you're hurt that I called Perl 5 old! That's funny. I've never written production Perl, I am not making a judgement on the language.
> Because I guarantee that whatever language and toolset you are using now, you'll need to be able to switch at some point in your career.
Yeah, but the motivation to learn Perl 5 as your next language is about as wise as learning COBOL as your next language.
That's probably why parent is finding it hard to get his developers to work on it - lots of effort for negative gain[1]. Anyone wanting to learn Perl 5 in 2023, for no additional money, is probably legitimately crazy.
I don't understand why you don't see the problem that parent has - no one wants to learn Perl 5 because it is a dead-end, and takes up energy that would otherwise be spent moving your career forward.
[1] The opportunity cost is immense - the time used to learn Perl 5 enough to maintain the legacy code is time that could have been used to learn a current language that results in a payoff.
>It sounds like you haven't been in the situation the parent is describing.
Sure I have. And the first few times I too thought that it mattered. But it doesn't. None of the stuff we care about as software developers actually matters. We are just pawns for the people making money decisions. Completely replaceable automatons. If what you are doing is not directly supporting a strategic business goal, you are doing nothing but costing a business money. And cost centers are the first to be cut. You can sit there with your sense of self importance, thinking that the cleanliness and elegance of your code matters. But it just doesn't. Nobody cares. And the sooner you let go of thinking it matters the sooner you can focus your effort on what does.
The reality is that when whatever old house of cards you're running fails for whatever reason, the business will switch to a vendor that covers 95% of the use case, while being much cheaper and not requiring FTE support. It will be a tiny bump in the road for the business, replacing this thing that you thought was a world beating make or break technology.
What would be an example of such a software company?
I am always faced with this issue with non-software companies in industries where I've worked such as nuclear, defense, energy, and healthcare. At these places, I've confronted "hands-off" modules that are effectively black boxes of complex logic that has been vetted over decades, is written in C or Fortran, and nobody is allowed to modify.
I can't imagine that situation existing in a software company.
In my haste to give an example I missed the part about it being for only "software companies", we're not a software company, but we do have a lot of developers
The problem that your product owner cannot see any value in replacing/understanding/modernizing that piece of software, but should it fail, the heads would roll, speaks of a bigger problem in your organization than an ancient piece of software.
You have been able to put it in a container and it’s running, so unless the kernel itself changes drastically so old libc will just stop working, it’s going to be able to work unchanged for a decade more.
But the communications problem between management layers is way bigger than that, and could probably bite in some other situation.
3-4 years is very short as far as software lifespans go. Software libraries that are 10-15 years old are commonplace in both open source and at FAANGs, and some industries practice "extreme longevity": any domain where you wouldn't be surprised to run into a mainframe is probably also running code with a 30 year+ lineage.
Don't know what kind of projects you are working on, but in the world where I work, source-code that is literally decades old is common, and companies pay extremely good money for people who understand it.
This includes things like airline reservation systems, scientific computing models, compilers for obscure dsls, and the like.
If a business is competing successfully I don’t think you’re going to see software obviated every three years. Software isn’t like living beings that up and die. They evolve over time.
Companies doing rewrites every three years have a hard ceiling.
That's a blind jump. We don't know the purpose, we don't know how much was invested, how much was already recovered, and how much more could be earned if the program was allowed to live.
All we know is that it has recently hit fitness to purpose, and that the fitness caused it to be abandoned.
Fun story about the 3-4Y window, one of the execs at my company was having a Q&A, and he gave an example in another question a new set of components, saying that he expects them to be in use for the next 20Y. I asked why does he think he can keep the know-how and outsmart the competition (components were part of algo trading and other trading related things) for the next 20y with just this investment, and he just downplayed my question as an IT guy wanting more money/work. ¯\_(ツ)_/¯
The code worked, and continued to work, and didn't need really active development, so the org decided the project's programmers should move on to something new, or that the knowledge they have isn't especially valuable, and the programmers aren't worth keeping happy.
And a couple of years later, no one knows how to update the code for a changing outside environment, updated input styles or whatever. And all that investment is gone.