> though I hope existing well tested rust libraries are used rather than NIHing them and introducing new bugs
I think that's much more likely to introduce bugs.
Think of it that way, a lot of the Rust libraries are rewriting existing copyleft libraries in permissive licenses, so they cannot look at the original code, dooming them to repeat the mistakes that were made in the original code and having to fix them all over again on their own (as both go from "oh this is simple" to "oh another corner case").
I just want to translate code 1:1 to Rust, reusing my existing knowledge, design decisions, and tests. It should behave _exactly_ the same as before, just memory safe.
It's certainly what we aim for in APT. We do have an overwrite of course, since we need to copy uninitiated data around: The cache file is allocated as a whole and written at the end, but not all parts of it are used, but it triggers stuff.
Don't want to introduce complex code to only copy the parts that are actually reachable would be silly and introduce bugs.
But keep in mind valgrind is super buggy and we spend quite a bunch of time working around valgrind false positives (outside of amd64)
TBH most “false positives” that I investigate are wishful thinking or the result of ignorance of what is really happening. It looks like you are using Debian. That probably doesn’t help. Here is a typical Debian “bug” report:
10 years old. It never was a false positive. It was fixed a good few years ago. The fix did not involve suppressing the error.
Valgrind does need a lot of work, especially for missing CPU features and for Darwin. I’m not aware of many memcheck bugs that aren’t relatively obscure corner cases.
Like the last one was/is the inability to comprehend safety of large buffers on ppc64el because the stack clash protector code generated by gcc isn't understood. The one before that was more problems of that sort on armhf where it also didn't understand the clash protector - in more cases.
It's quite surprising and it takes days to weeks to debug each of these, going down to the assembler level and verifying that by hand.
Do keep in mind that a lot of the people involved in these sorts of things are neurodiverse in some ways, and may have significant trouble dealing with change.
As teh64 helpfully pointed out in https://news.ycombinator.com/item?id=45784445 some hours ago, 4ish years ago my position on this was a total 360 and I'd have had the same reaction to now-me's proposal.
All these changes requires work. Because of this, other priorities will get less attention. It would be ironic if bad security flaws are missed/introduced because of all the work switching to Rust. Its also very likely that all the new code written in Rust will be far less mature than the existing source bases. So the outcome might be (very probably actually) a lot of work to worsen security.
Most of the academic research into these sorts of typesafe languages usually returns the null result (if you don't agree, it means you haven't read the research on this topic). That's researcher for it didn't work and you shouldn't be using these techniques. Security is a process, not a silver bullet and 'just switch to Rust' is very silvery bullet.
It's not like I'm in a hurry to switch to Rust and will spend full steam on it. It's amongst the lowest priority items.
A lot of the Rust rewrites suffer a crucial issue: they want a different license than what they are rewriting and hence rewrite from scratch because they can't look at the code.
But here we're saying: Hey we have this crucial code, there may be bugs hidden in it (segfaults in it are a recurring source of joy), and we'll copy that code over from .cc to .rs and whack it as little as possible so it compiles there.
The problem is much more there on the configuration parser for example which does in a sense desparately need a clean rewrite, as it's way too sloppy, and it's making it hard to integrate.
In an optimal world I'd add annotations to my C++ code and have a tool that does the transliteration to Rust at the end; like when the Go compiler got translated from C to Go. It was glorious.
Isn't it also funny that all of these things are done by the same person?
In all seriousness though, let me assure you that I plan to take a very considerate approach to Rust in APT. A significant benefit of doing Rust in APT rather than rewriting APT from scratch in Rust means that we can avoid redoing all our past mistakes because we can look at our own code and translate it directly.
Honestly having seen trainwreck after trainwreck after trainwreck come out of Canonical for the last decade, I'm sure I'm not the only one that has strong doubts about anyone associated being able to "avoid redoing past mistakes" or to make things not suck.
Keep in mind APT is using the GNU dialect of C++, particularly C++ 17, shortly C++ 23. And it always exploits the latest C++ features while at the same time still having workarounds in the code for a lack of namespaces in C++ compilers prior to the C++ standardization...
Actual cryptography code, the best path is formally verified implementations of the crypto algorithms; with parsers for wrapper formats like OpenPGP or PKCS#7 implemented in a memory safe language.
You don't want the core cryptography implemented in Rust for Rust's sake when there's a formally verified Assembler version next to it. Formally verified _always_ beats anything else.
I should have clarified that I was primarily referring to the stuff dealing with all the wrapper formats (like PKIX certificate verification), not the core cryptographic algorithms themselves.
The core cryptographic algorithms, IMHO, should be written in a dedicated language for writing cryptographic algorithms so that they can get formally-verified constant-time assembly out of it without having to complain to us compiler writers that we keep figuring out how to deobfuscate their branches.
Sure. But assembly implementations by definition are not portable. And I don’t know what it takes to write a formally verified library line this, but I bet it’s very expensive.
In contrast, a rust implementation can be compiled into many architectures easily, and use intrinsically safer than a C version.
Plus cryptography and PKI is constantly evolving. So it can’t benefit from the decades old trusted implementations.
Formally verified in an obscure language where it's difficult to find maintainers does not beat something written in a more "popular" language, even if it hasn't been formally verified (yet?).
And these days I would (unfortunately) consider assembly as an "obscure language".
(At any rate, I assume Rust versions of cryptographic primitives will still have some inline assembly to optimize for different platforms, or, at the very least, make use of compile intrinsics, which are safer than assembly, but still not fully safe.)
It's insanely complex, particularly you want _verified_ crypto. Last year (or two years ago?) I had to fix a tiny typo in OpenSSL's ARM assembly for example, it was breaking APT and Postgres left and right, but only got triggered on AWS :D
You don't want to write the whole thing in assembly, just the parts that need to be constant time. Even those are better written as called subroutines called from the main implementation.
Take BLAKE3 as an example. There's asm for the critical bits, but the structural parts that are going to be read most often are written in rust like the reference impl.
I would like a special purpose language to exist precisely for writing crytographic code where you always want the constant time algorithm. In this niche language "We found a 20% speed-up for Blemvich-Smith, oops, it actually isn't constant time on the Arrow Lake micro-code version 18 through 46" wouldn't even get into a nightly let alone be released for use.
It seems that for reasons I don't understand this idea isn't popular and people really like hand rolling assembly.
I do think this is pretty much the one use case for a true "portable assembler", where it basically is assembly except the compiler will do the register allocation and instruction selection for you (so you don't have to deal with, e.g., the case that add32 y, x, 0xabcdef isn't an encodable instruction because the immediate is too large).
Ah, my apologies. I must admit I am not entirely clear on where the boundaries are between the different tools and other elements of the debian packaging process.
If anyone sees that horrible mess of hacks around pre-STL C++'s lacks of namespace in combination with latest C++ features as part of the C++ community I'd be very surprised :D
If APT were a hardcore C++ project surely we'd have like adopted namespaces everywhere by now.
I think that's much more likely to introduce bugs.
Think of it that way, a lot of the Rust libraries are rewriting existing copyleft libraries in permissive licenses, so they cannot look at the original code, dooming them to repeat the mistakes that were made in the original code and having to fix them all over again on their own (as both go from "oh this is simple" to "oh another corner case").
I just want to translate code 1:1 to Rust, reusing my existing knowledge, design decisions, and tests. It should behave _exactly_ the same as before, just memory safe.
reply