As a longtime autotools-hater I would say this justifies my position, but any build system complex enough to be multiplatform is going to be inscrutable enough to let somebody slip something in like this. But it really is a problem that so much software is built with what's essentially become a giant cargo cult-style shell script whose pieces no one person understands all of.
I think the common usage of bash and other languages with a dense and complicated syntax is the root problem here. If build scripts were written in python, you would have a hard time obfuscating backdoors, because everyone would see that it is weird code. Bash code is just assumed to be normal when you can’t read it.
I think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python.
You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
Man, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.
Shell is evil mainly because it's so old. It's largely unchanged since 1978 and had to run on incredibly resource limited machines. What it could achieve on those machines was amazing. Tradeoffs would be different today, e.g. PowerShell.
A big pet peeve of mine is that shell is written off as evil but the only reason I ever hear is basically a variation of “I don’t understand it, therefore it scares me.” The reality is, unlike _really_ old languages like COBOL or RPG, bash is still literally everywhere, it’s installed on practically every linux machine by default which makes deploying a shell script completely trivial. It’s certainly under appreciated and because it’s ubiquitous, widely used in build processes, there’s a responsibility to learn it. It’s not hard, it’s not a wildly complex language.
I don’t think these issues would necessarily be solved at all by waving a hand and replacing it with similarly complex build tools. Bazel, for example, can be a daunting tool to fully grasp. Any tool used should be well understood. Easier said than done of course.
As long as you apply the same standards to what seems to be everyones darling: Javascript.
Javascript has the same amount of footguns as PHP and Bash but has gotten away with it by being cute (and having a whole menagerie if support tools around it to make it possible for ordinary people to write workable code in it).
(Yes, I am qualified to rant about Javascript BTW. I wrote a working map rendering system with pan and zoom and automatic panning based on GPS location using ECMAScript and SVG back in the spring of 2005. I think roughly half a year before Google Maps became public. Back before all the modern JS tooling existed. All I had was JEdit with syntax highlighting. Perl at least let me put breakpoints in my code even back then.
In the context of build systems and the vulnerabilities that exist in xs, one of server-side JavaScript’s biggest footguns that cannot be ignored is its dependency management. Very few people I know ever really dig into the dependency tree and audit all packages 10 levels deep. The attack surface there is huge and objectively much wider than PHP/Bash. It’s also a built-in automatic entryway into a corporate network.
FWIW it has actually become better the last few years:
Now you can at least just stick to React and TypeScript and bundle it using Webpack and have months of relative sanity between each time you have to throw something out and replace it.
Shell has a lot of stuff going for it too, though:
-> REPL essentially for free (the language IS a REPL)
-> enormous installed base
-> No compilation (well, unless you have something--like autotools--using shell as essentially a transpilation target)
-> No need for "libraries" in most cases: the ordinary CLI that $vendor already ships can be used right away, with no need for a custom SDK for whatever "real" language you would otherwise be using. For example, if you are already familiar with the "aws" CLI program, it's trivial to treat it as an "API" for a quick shellscript instead of needing to dig into the boto3 docs to do something equivalent the "right" way.
-> Pretty good integration with standard *nix facilities (redirecting STD{ERR,OUT}, checking existence of files/pipes, checking/setting exit codes, etc.)
I’d argue that every language is loaded full of foot guns. If you’re encountering those foot guns on a regular basis, it’s an issue with the author.
That said, what can help drastically here are well-defined best practices and conventions built into the language which, admittedly, bash really doesn’t have.
Yep, every language has footguns and other kinds of quirks, but I contend that
the "footguns per character" ratio in shell is unusually high. (It is not unique in having a high ratio though; other popular languages, like c++ for instance, also have this issue.)
The worst (level of nastyness * usage) offenders all probably have a reason for being popular despite their flaws:
- Bash: installed everywhere you want to work (yes, who actually wants to work on Windows ;-)
- C/C++: when speed/size matters there was no alternative except Assembly until recently
- Javascript: until recently this was the most sane option for client side code on the web (Active X and Java applets existed yes but managed to be even worse.)
- PHP: Low cost hosting, Function-As-A-Service way before that became popular, shared nothing architecture, instant reload for local development bliss
- string vs. list "in" ('a' in 'a' is True, but 'a' in ['a'] is also True)
- cannot know which object attributes are private or public (and some classes use settable properties so you can't say "just don't set any attributes on non-dataclass objects")
I think a good way to evaluate languages from this perspective is through the lens of how easy it is to maintain understanding over time. I have learned bash well three or four times now, but I know now that I'm never going to remember enough of its quirks through the interim periods where I'm not focused on it, to be able to grok arbitrary scripts without refreshing my memory. This is very different for languages like java, go, python, and some others, which have their quirks, sure, but a much lower quirks per character ratio.
I might agree with "it's not hard to learn it", but I don't agree with "it's not hard to remember it".
I don't think PowerShell is a big improvement, though. Still allows no signature checking of functions. Shells are optimized for fast entry, not for writing comprehensible (or even secure) programs.
I really don't understand this point, its a script language, how old is it doesn't make any difference. I've come accross some Powershell scripts that were unreadable down to its verbosity with certain things, and if you don't already know all the flags and options for it, it's hopeless to try and understand.
> I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written.
When you use long options in bash (scripts), it becomes very readable, but it's not a widespread practice. I always use long options while writing scripts.
Consider these two examples, which is very straightforward:
IMO, that's not where the inscrutability comes from. Rather, it is things like many useful (and thus used) constructs being implemented with line noise like `"$(foo)"`, `2>&1 &`, `${foo:-bar}`, etc., finicky control flow syntax, surprising defaults that can be changed within unclear scopes, etc.
You're right, but you can also expand these. If not syntactically, by adding temporary variables and comments around these complex parts.
It's true that Bash and Perl has one of the most contractible syntax around, but it's not impossible to make it more understandable.
However, these parts of codebases are considered "supportive" and treated as second class citizens, and never receives the same love core parts of the codebases enjoy. That's a big mistake IMO.
When you make something more readable all around, hiding things becomes harder exponentially.
I have seen people write bash scripts with a lot of love and attention trying really hard to pay attention to modern best practices and avoiding known pitfalls and using shellcheck. It still looked like shit imo.
I challenge you to find a single easily readable >100 line bash script and link it here (I do think small scripts can be fine).
Yep, my experience of this was at google, where I heard "gbash is actually pretty good!", so I learned that system well (because shell scripts are incredibly useful!), but still found the results unsatisfyingly inscrutable.
The irony of talking about readability of an example of curl to sh. You really don't need to understand the flags here to understand that this is a voluntary RCE and while I'm generally for long options in scripts, in this case it might make it harder to see the real problem because it increases the distance between the download command and the shell invocation.
So are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on.
Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
FWIW, my comment wasn't meant to single out python as particularly good. I think the comparison I drew between its inscrutability and that of shell / bash would apply to nearly all other languages as well.
You already have to inspect everything if you want to review/audit a build script. Small functions - and I specifically mean functions being written small because of misguided ideas of "clean code", as opposed to e.g. useful abstraction or reusability - become especially painful there, as you have that much more code to read, and things that go together logically (or execution-wise) are now smeared around the file.
And you can't really name such small functions well anyway, not when they're broken down for the sake of being small. Case in point, some build script I saw this week had function like `rename_foo_dll_unit_tests` calling `rename_foo_dll_in_folder` calling `rename_foo_dll` calling `rename_dlls`, a distinct call chain of four non-reused functions that should've been at most two functions.
Are all Python build scripts like that? Not really. It's just a style I've seen repeatedly. The same is the case with inscrutable Bash scripts. I think it speaks more about common practices than the language itself (notwithstanding Bash not really being meant for writing longer programs).
Conan is a package manager for C/C++, written in Python. See: https://conan.io/.
The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.
Now, for the sake of this thread I want to highlight three things here:
- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;
- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;
- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:
Good luck keeping track of what exact code goes into your program, when using Turing-complete "recipe" programs fetched from the Internet, which fetch your libraries from somewhere else on the Internet.
It depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.
Oh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.
The problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly.
But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.
I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.
That's a lot of extra code to review, before you even get to any kind of language differences.
But then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.
> Nobody is going to review test, it was just generated by automake, right?
Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.
> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.
You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
This probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects.
On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
What if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.
Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.
Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.
In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.
However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.
In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.
There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
C is probably the best example where I would be fine with scripts in Python (for utility scripts, not build scripts). Though, if it were me I'd use Ruby instead as I like that language a lot better, and it has Rake (a ruby-ish version of Make), but that's a "flavor of ice cream" kind of choice.
or make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.
I think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user.
Something built on python won't cover quite as wide a range of (obsolete?) hardware.
Python actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script.
Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.
The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).
The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
...The main problem is some asshat trying to install a backdoor.
I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.
I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.
There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!
As a tester, nothing makes me angrier!
A pox upon them, and may their treachery be returned upon them 7-fold!
A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
You assert a dichotomy where no split really exists. Qt the end of the day, things like tr or xz or sed invocations are not part of bash as a language. They are seperate tool invocations. Python or any other programming language could have things hidden in them just as easily.
And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.
And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.
Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.
“Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.
I'm mot asking people to be better computers. Nor do I believe myself to blissfully free of making mistakes. I'm saying, from an information propagation standpoint, it is 100% impossible for you to make a knowing based statement without having first laid eyes on the thing which you are supposed to make knowledge based statements about.
This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.
This feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.
Yep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”.
This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically.
In bash land? “What the hell is [[?”
And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language.
And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should
acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
Yeah, nah.
I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong.
But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age.
The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable.
Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
It's rather easy to monkeypatch Python into doing spooky things. For something like this you really want a language that can't be monkeypatched, like I think Starlark.
Where are you going to hide your monkey patching though? As long as your code is public, stuff like this is always going to stand out, because no one writes weird magic one liners in python.
The way python's run time literally executes code when you import a module makes it seem pretty easy to taint things from afar. You only need to control a single import anywhere in the dependency hierarchy and you can reach over and override any code somewhere else.
This code was essentially monkey patched from a test script. Python automatically runs any code in a imported module, so not hard to see a chain of module imports that progressively modifies and deploys a similar structure.
I'm not a security-oriented professional, but to me a place I could hide this logic is by secretly evaling the contents of some file (like the "corrupt archive" used in xz) somewhere in the build process, hiding it behind a decorator or similar.
I’m not a security professional either, but that doesn’t sound very plausible to me. If you assume a maintainer who checks every commit added to the codebase, he’s hopefully blocking you the second he sees an eval call in your build script. And even a code audit should find weird stuff like that, if the code is pythonic and simple to read. And if it’s not, it should not be trusted and should be treated as malicious.
That was true for this project, which was almost orphaned to begin with. We'll run out of nearly-unmaintained critical infrastructure projects sometime. Larger projects with healthier maintenance situations are also at risk, and it's worth reasoning about how a group of honest developers could discover the actions of one malicious developer (with perhaps a malicious reviewer involved too).
> In the beginning of Unix, m4 was created. This has made many people very angry and has been widely regarded as a bad move.
autoconf creates a shell script by preprocessing with m4. So you need to know not just the intricacies of shell scripting, but also of m4, with its arcane rules for escaping: https://mbreen.com/m4.html#quotes
Yet it is python code which has the highest amount of security vulnerabilities found in public repos. And the most often times that pre-compiled code is commited as well.
I see what everyone is saying about autotools, and I never envied those that maintained the config scripts, but as an end-user I miss the days when installing almost any software was as simple as ./configure && make && make install.
Autotools need to go away, but most of the underlying use cases that cause build system complexity comes down to dependency management and detection. The utter lack of best practices for those workflows is the root cause of complexity. There is no way to have a mybuild.toml build config on the face of those challenges.
Agreed, and I think this leads to the question: how much risk do we face because we want to support such a wide range of platforms that a complex system is required?
And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?
> And how did we get to the point that a complex system is required to build a compression library -- something that doesn't really have to do much more than math and memory allocation?
The project in question contained a compression library, but was not limited to it; it also contained a set of command line tools (the "xz" command and several others).
And a modern compression library needs more than just "math and memory allocation"; it also needs threads (to make use of all the available cores), which is historically not portable. You need to detect whether threads are available, and which threading library should be used (pthreads is not always the available option). And not only that, a modern compression library often needs hand-optimized assembly code, with several variants depending on the exact CPU type, the correct one possibly being known only at runtime (and it was exactly in the code to select the correct variant for the current CPU that this backdoor was hidden).
And that's before considering that this is a library. Building a dynamic library is something which has a lot of variation between operating systems. You have Windows with its DLLs, MacOS with its frameworks, modern Linux with its ELF stuff, and historically it was even worse (like old a.out-based Linux with its manually pre-allocated base address for every dynamic library in the whole system).
So yeah, if you restrict yourself to modern Linux and perhaps a couple of the BSDs, and require the correct CPU type to be selected at compilation time, you could get away with just a couple of pages of simple Makefile declarations. But once you start porting to a more diverse set of systems, you'll see it get more and more complicated. Add cross-compilation to the mix (a non-trivial amount of autotools complexity is there to make cross-compilation work well) and it gets even more complicated.
I'm glad to see I'm not a minority of one here. Looking at autoconf and friends I'm reminded of the dreck that used to fill OpenSSL's code simply because they were trying to account for every eventuality. Autotools feels like the same thing. You end up with a ton of hard to read code (autogenerated bash, not exactly poetry) and that feels very inimical to safety.
I do not agree with your generalisation, the Meson build system is well thought, it has a clear declarative syntax that let you just express what you want in a direct way.
The designer of Meson explicitly avoided making the language turing complete so for example you cannot define functions. In my experience this was an excellent decision to limit people tendency to write complex stuff and put the pressure on the Meson developer to implement themselves all the useful functionalities.
In my experience the Meson configuration are as simple as they can be and accommodate only a modicum of complexity to describe OS specific options or advanced compiler option one may need.
Please note that some projects' Meson file have been made complex because of the goal to match whatever the configure script was doing. I had in mind the crazy habits of autotools to check if the system has any possibly used function because some system may not have it.
Meson is way better than autotools, but if you're too level to depend on python, you're probably needing to customize your build scripts in the ways you mention. I don't see meson being a silver bullet there.
Also, meson's build dependencies (muon, python) are a lot for some of these projects.
I haven’t used it for some time but autoconf always seemed like a horrible hack that was impossible to debug if it didn’t work properly.
That was bad enough back in the days where one was mostly concerned with accidents, but in more modern times things that are impossible to debug are such tempting targets for mischief.
The issue was that built artifacts weren’t immutable during the test phase and/or the test phase wasn’t sandboxed from the built artifacts.
The last build system I worked on separated build and test as separate stages. That meant you got a lot of useless artifacts pushed to a development namespace on the distribution server, but it also meant later stages only needed read access to that server.
The malicious .o is extracted from data in binary test files, but the backdoor is inserted entirely in the build phase. Running any test phase is not necessary.
So if you ran build without test files it would fail. I get that this is hindsight thinking - but maybe removing all non essential files when packaging/building libraries reduces the surface area.
