I think the issue is that build systems are often build on a completely different language - python with some weird build framework is likely as inscrutable as bash and autotools to someone who doesn't use python.
You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.
Man, this sounds right, but I dunno ... I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written. Just comparing my dotfiles - every bit of which I understood when I wrote them - to the gnarliest sections of a big python app I work on ... I just really feel like at least some of this inscrutability issue can truly be laid at the feet of shell / bash as a language.
Shell is evil mainly because it's so old. It's largely unchanged since 1978 and had to run on incredibly resource limited machines. What it could achieve on those machines was amazing. Tradeoffs would be different today, e.g. PowerShell.
A big pet peeve of mine is that shell is written off as evil but the only reason I ever hear is basically a variation of “I don’t understand it, therefore it scares me.” The reality is, unlike _really_ old languages like COBOL or RPG, bash is still literally everywhere, it’s installed on practically every linux machine by default which makes deploying a shell script completely trivial. It’s certainly under appreciated and because it’s ubiquitous, widely used in build processes, there’s a responsibility to learn it. It’s not hard, it’s not a wildly complex language.
I don’t think these issues would necessarily be solved at all by waving a hand and replacing it with similarly complex build tools. Bazel, for example, can be a daunting tool to fully grasp. Any tool used should be well understood. Easier said than done of course.
As long as you apply the same standards to what seems to be everyones darling: Javascript.
Javascript has the same amount of footguns as PHP and Bash but has gotten away with it by being cute (and having a whole menagerie if support tools around it to make it possible for ordinary people to write workable code in it).
(Yes, I am qualified to rant about Javascript BTW. I wrote a working map rendering system with pan and zoom and automatic panning based on GPS location using ECMAScript and SVG back in the spring of 2005. I think roughly half a year before Google Maps became public. Back before all the modern JS tooling existed. All I had was JEdit with syntax highlighting. Perl at least let me put breakpoints in my code even back then.
In the context of build systems and the vulnerabilities that exist in xs, one of server-side JavaScript’s biggest footguns that cannot be ignored is its dependency management. Very few people I know ever really dig into the dependency tree and audit all packages 10 levels deep. The attack surface there is huge and objectively much wider than PHP/Bash. It’s also a built-in automatic entryway into a corporate network.
FWIW it has actually become better the last few years:
Now you can at least just stick to React and TypeScript and bundle it using Webpack and have months of relative sanity between each time you have to throw something out and replace it.
Shell has a lot of stuff going for it too, though:
-> REPL essentially for free (the language IS a REPL)
-> enormous installed base
-> No compilation (well, unless you have something--like autotools--using shell as essentially a transpilation target)
-> No need for "libraries" in most cases: the ordinary CLI that $vendor already ships can be used right away, with no need for a custom SDK for whatever "real" language you would otherwise be using. For example, if you are already familiar with the "aws" CLI program, it's trivial to treat it as an "API" for a quick shellscript instead of needing to dig into the boto3 docs to do something equivalent the "right" way.
-> Pretty good integration with standard *nix facilities (redirecting STD{ERR,OUT}, checking existence of files/pipes, checking/setting exit codes, etc.)
I’d argue that every language is loaded full of foot guns. If you’re encountering those foot guns on a regular basis, it’s an issue with the author.
That said, what can help drastically here are well-defined best practices and conventions built into the language which, admittedly, bash really doesn’t have.
Yep, every language has footguns and other kinds of quirks, but I contend that
the "footguns per character" ratio in shell is unusually high. (It is not unique in having a high ratio though; other popular languages, like c++ for instance, also have this issue.)
The worst (level of nastyness * usage) offenders all probably have a reason for being popular despite their flaws:
- Bash: installed everywhere you want to work (yes, who actually wants to work on Windows ;-)
- C/C++: when speed/size matters there was no alternative except Assembly until recently
- Javascript: until recently this was the most sane option for client side code on the web (Active X and Java applets existed yes but managed to be even worse.)
- PHP: Low cost hosting, Function-As-A-Service way before that became popular, shared nothing architecture, instant reload for local development bliss
- string vs. list "in" ('a' in 'a' is True, but 'a' in ['a'] is also True)
- cannot know which object attributes are private or public (and some classes use settable properties so you can't say "just don't set any attributes on non-dataclass objects")
I think a good way to evaluate languages from this perspective is through the lens of how easy it is to maintain understanding over time. I have learned bash well three or four times now, but I know now that I'm never going to remember enough of its quirks through the interim periods where I'm not focused on it, to be able to grok arbitrary scripts without refreshing my memory. This is very different for languages like java, go, python, and some others, which have their quirks, sure, but a much lower quirks per character ratio.
I might agree with "it's not hard to learn it", but I don't agree with "it's not hard to remember it".
I don't think PowerShell is a big improvement, though. Still allows no signature checking of functions. Shells are optimized for fast entry, not for writing comprehensible (or even secure) programs.
I really don't understand this point, its a script language, how old is it doesn't make any difference. I've come accross some Powershell scripts that were unreadable down to its verbosity with certain things, and if you don't already know all the flags and options for it, it's hopeless to try and understand.
> I feel like even "simple" shell scripts tend toward more inscrutability than all but the most questionable python code I've ever written.
When you use long options in bash (scripts), it becomes very readable, but it's not a widespread practice. I always use long options while writing scripts.
Consider these two examples, which is very straightforward:
IMO, that's not where the inscrutability comes from. Rather, it is things like many useful (and thus used) constructs being implemented with line noise like `"$(foo)"`, `2>&1 &`, `${foo:-bar}`, etc., finicky control flow syntax, surprising defaults that can be changed within unclear scopes, etc.
You're right, but you can also expand these. If not syntactically, by adding temporary variables and comments around these complex parts.
It's true that Bash and Perl has one of the most contractible syntax around, but it's not impossible to make it more understandable.
However, these parts of codebases are considered "supportive" and treated as second class citizens, and never receives the same love core parts of the codebases enjoy. That's a big mistake IMO.
When you make something more readable all around, hiding things becomes harder exponentially.
I have seen people write bash scripts with a lot of love and attention trying really hard to pay attention to modern best practices and avoiding known pitfalls and using shellcheck. It still looked like shit imo.
I challenge you to find a single easily readable >100 line bash script and link it here (I do think small scripts can be fine).
Yep, my experience of this was at google, where I heard "gbash is actually pretty good!", so I learned that system well (because shell scripts are incredibly useful!), but still found the results unsatisfyingly inscrutable.
The irony of talking about readability of an example of curl to sh. You really don't need to understand the flags here to understand that this is a voluntary RCE and while I'm generally for long options in scripts, in this case it might make it harder to see the real problem because it increases the distance between the download command and the shell invocation.
So are some Python advocates, too. The thing that's worse than a bash script made of Perlish line noise, is a piece of "clean code" dead simple Python that's 80% __language__.boilerplate, 20% logic, smeared over 10x the lines because small functions calling small functions are cool. No one has enough working memory to keep track of what's going on there. Instead, your eyes glaze over it and you convince yourself you understand what's going on.
Also, Python build scripts can be living hell too, full of dancing devils that could be introducing backdoors left and right - just look at your average Conan recipe, particularly for larger/more sensitive libraries, like OpenSSL or libcurl.
FWIW, my comment wasn't meant to single out python as particularly good. I think the comparison I drew between its inscrutability and that of shell / bash would apply to nearly all other languages as well.
You already have to inspect everything if you want to review/audit a build script. Small functions - and I specifically mean functions being written small because of misguided ideas of "clean code", as opposed to e.g. useful abstraction or reusability - become especially painful there, as you have that much more code to read, and things that go together logically (or execution-wise) are now smeared around the file.
And you can't really name such small functions well anyway, not when they're broken down for the sake of being small. Case in point, some build script I saw this week had function like `rename_foo_dll_unit_tests` calling `rename_foo_dll_in_folder` calling `rename_foo_dll` calling `rename_dlls`, a distinct call chain of four non-reused functions that should've been at most two functions.
Are all Python build scripts like that? Not really. It's just a style I've seen repeatedly. The same is the case with inscrutable Bash scripts. I think it speaks more about common practices than the language itself (notwithstanding Bash not really being meant for writing longer programs).
Conan is a package manager for C/C++, written in Python. See: https://conan.io/.
The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.
Now, for the sake of this thread I want to highlight three things here:
- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;
- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;
- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:
Good luck keeping track of what exact code goes into your program, when using Turing-complete "recipe" programs fetched from the Internet, which fetch your libraries from somewhere else on the Internet.
It depends on the use case. Bash code can be elegant, Python code can be ugly. I'm not saying those are the average cases but complex code regardless of the language often is ugly even with effort to make it more readable.
Oh I get it. Sometimes bash is the right tool for the job. I think that’s just mostly an unfortunate historical artifact though. It’s hard to argue it’s intuitive or “clean” or ${POSITIVE_ADJECTIVE} in the average case.
The problem is that obfuscated bash is considered normal. If unreadable bash was not allowed to be committed, it would be much harder to hide stuff like this. But unreadable bash code is not suspicious, because it is kind of expected. That’s the main problem in my opinion.
Lots of autogenerated code appears "obfuscated" - certainly less clear than if a programmer would have written it directly.
But all this relies on one specific thing about the autotools ecosystem - that shipping the generated code is considered normal.
I know of no other build system that does this? It feels weird, like shipping cmake-generated makefiles instead of just generating them yourself, or something like scons or meson being packaged with the tarball instead of requiring an eternal installation.
That's a lot of extra code to review, before you even get to any kind of language differences.
But then you have the problem that enabled this backdoor. It's normal to have uncommitted autogenerated unreadable shell code in the tarball. Nobody is going to review test, it was just generated by automake, right? That makes it so easy to sneak in a line that something slightly different. At least with cmake, you have none of this nonsense, people need cmake to build the project, it doesn't try to save users from it by generating a ton of unreadable shell code.
> Nobody is going to review test, it was just generated by automake, right?
Well, there's your problem. If you have unreviewed code, anything can be snuck in. Doesn't really matter too much where in your system the unreviewed code is.
> It's normal to have uncommitted autogenerated unreadable shell code in the tarball.
You need to review everything that goes into the tarball. Either directly, or indirectly by reviewing the sources it gets built from. (And then making sure that your build process is deterministic, and repeated by a few independent actors to confirm they get the same results bit for bit.)
This probably varies widely, because unreadable bash is absolutely not considered normal, nor would pass code review in any of my projects.
On a slightly different note, unless the application is written in python, it grosses me out to think of writing scripts in python. IMHO, if the script is more complex that what bash is good at (my general rule of thumb is do you need a data structure like an array or hash? then don't use bash), then use the same language that the application is written in. It really grosses me out to think of a rails application with scripts written in python. Same with most languages/platforms.
What if your application is written in Rust or C? Would you write your build scripts in these languages, too? I would much prefer a simpler scripting language for this. If you’re already using a scripting language as the main language, you don’t necessarily need to pull in another language just for scripts, of course.
Writing anything in C is a bad idea these days, and requires active justification that only applies in some situations. Essentially, almost no new projects should be done in C.
Re-doing your build system, or writing a build system for a new project, counts as something new, so should probably not be done in C.
In general, I don't think your build (or build system) should necessarily be specified in the same language as most of the rest of your system.
However I can see that if most of your system is written in language X, then you are pretty much guaranteed to have people who are good at X amongst your developers, so there's some natural incentive to use X for the tooling, too.
In any case, I would mostly just advice against coding anything complicated in shell scripts, and to stay away from Make and autotools, too.
There are lots of modern build systems like Shake, Ninja, Bazel, etc that you can pick from. They are all have their pros and cons, just like the different distributed version control systems have their pros and cons; but they are better than autotools and bash and Make, just like almost any distributed version control is better than CVS and SVN etc.
C is probably the best example where I would be fine with scripts in Python (for utility scripts, not build scripts). Though, if it were me I'd use Ruby instead as I like that language a lot better, and it has Rake (a ruby-ish version of Make), but that's a "flavor of ice cream" kind of choice.
or make.go, for some project it makes sense to not add another language for scripting and building tasks. It way easier for every one to have to master multiple language.
I think the main issue is auto tools tries to support so many different shells/versions all with their own foibles, so the resulting cross-compatible code looks obfuscated to a modern user.
Something built on python won't cover quite as wide a range of (obsolete?) hardware.
Python actually covers quite a lot of hardware. Of course, it does that via an autotools nightmare generated configure script.
Of course, you could do the detection logic with some autotools-like shenanigans, but then crunch the data (ie run the logic) on a different computer that can run reasonable software.
The detection should all be very small self-contained short pieces of script, that might be gnarly, but only produce something like a boolean or other small amount of data each and don't interact (and that would be enforced by some means, like containers or whatever).
The logic to tie everything together can be more complicated and can have interactions, but should be written in a sane language in a sane style.
...The main problem is some asshat trying to install a backdoor.
I use bash habitually, and every time I have an inscrutable or non-intuitive command, I pair it with a comment explaining what it does. No exceptions.
I also don't clean up after scripts for debuggability. I will offer an invocation to do the cleanup though after you've ascertained everything worked. Blaming this on bash is like a smith blaming a hammer failing on a carpenter's shoddy haft... Not terribly convincing.
There was a lot of intentionally obfuscatory measures at play here and tons of weaponization of most conscientious developer's adherence to the principle of least astonishment, violations of homoglyphy (using easy to mistake filenames and mixed conventions), degenerative tool invocations (using sed as cat), excessive use/nesting of tools (awk script for the RC4 decryptor), the tr, and, to crown it all, malicious use of test data!!!
As a tester, nothing makes me angrier!
A pox upon them, and may their treachery be returned upon them 7-fold!
A good practice, but not really a defense against malice, because if the expression is inscrutable enough to really need a comment, then it's also inscrutable enough that many people won't notice that the comment is a lie.
You assert a dichotomy where no split really exists. Qt the end of the day, things like tr or xz or sed invocations are not part of bash as a language. They are seperate tool invocations. Python or any other programming language could have things hidden in them just as easily.
And the other major issue here, is that xz is a basic system unit, often part of a bare bones, ultra basic, no fluff, embedded linux deployment where other higher level languages likely wouldn't be. It makes sense for the tools constituting the build infra to be low dependency.
And yes. In a low dependency state, you have to be familiar with your working units, because by definition, you have fewer of them.
Unironically, if more people weren't cripplingly dependent on luxuries like modern package managers have gotten them accustomed to, this all would have stuck out like a sore thumb, which it still did once people actually looked at the damn thing*.
“Why can’t humans just be better computers!” is a naive exercise in futility. This is not the answer, and completely ignores the fact that you yourself certainly make just as many mistakes as everyone else.
I'm mot asking people to be better computers. Nor do I believe myself to blissfully free of making mistakes. I'm saying, from an information propagation standpoint, it is 100% impossible for you to make a knowing based statement without having first laid eyes on the thing which you are supposed to make knowledge based statements about.
This fundamental limitation on info prop will never disappear. There is nothing harder to do than to legit get somebody to actually read code.
This feels a bit like saying you can run as fast as Usain Bolt. Theoretically many things are possible, but I don't think I've ever seen a readable bash script beyond a trivial oneliner and I've seen a lot of bash in my life. Or to maybe explain from a different perspective, ask a room full of developers to write a bash if-else without looking at the docs and you'll probably come back with more different options than developers in the room. Ask the same for a language such as Python and you'll mostly get one thing.
Yep. Python is my day job. If anyone on my team put something up that was as unreadable as a typical bash script, without really good justification, I’d immediately hit “request changes”.
This isn’t just because I’m more familiar with Python. I don’t even think it’s the main reason. It’s just that Python is more likely to be able to be read in a ‘natural language’ sort of way. It’s better at doing what it says on the tin. It’s more able to be read via pure intuition by a programmer that’s not familiar with Python specifically.
In bash land? “What the hell is [[?”
And yes, I could come up with 20 ways off the top of my head that Python is a far-from-perfect language.
And I’m not even saying that Python is the right tool for the job here. Maybe we’re better off with one of the many more modern attempts at a shell language. The main thing is that we should
acknowledge that bash has overstayed its welcome in many of the areas in which it’s still used.
Yeah, nah.
I’m all for “every tool has its place”, “religious wars over languages are silly”, etc. Don’t get me wrong.
But anyone that claims that all but the 1% most simple bash scripts are “straightforward” and “readable” is, to be blunt, showing their age.
The reality is that we as a society have made meaningful progress as far as designing readable languages goes. And we LITERALLY have many orders of magnitude more resources to make that happen. That’s something to feel good about. It’s unreadable to continue to mischaracterise some sysadmin greybeard’s’ familiarity with bash as an indication that it is in any way intuitive or readable.
Like, sheesh, now we all sound like C developers trying to justify an absurdly footgun-laden standard library just because we happen to know the right secret incantations, or think that we know them, anyway. But now this is definitely becoming a religious war…
You can write pretty straightforward readable bash, just as I'm sure you can write pretty gnarly python. Especially if you're intentionally trying to obfuscate.