Hacker Newsnew | past | comments | ask | show | jobs | submit | eddyb's commentslogin

If you're not familiar, the way X11 works is comparable to "multiplayer notepad" for your pixels ("multiplayer MSPaint"?).

All your monitors are combined into a large canvas where every pixel can be written and read by any X11 client.

Screenshots (and screen sharing) could be silently performed with zero user feedback (or any good way to even detect when apps might be doing this maliciously, AFAIK).

This is one of the big "security implications" that motivated Wayland (and somewhat similarly, the Flatpak sandbox and the XDG Portal infrastructure that has by now outgrown it).

The infrastructure is already there, for 3rd party apps to request these abilities (with the user getting the choice of following through, or denying the request), e.g.:

- https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.fr...

- https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.fr...

Keep in mind that any action which doesn't require interactivity every time (e.g. restarting the screensharing of a previously-chosen window/display) could have "user gave permission" be remembered, but that seamless case still only applies to that combination (so that client can't peek at anything else than what it was offered).

Anyway, what the blog post is talking about is really just 3rd party apps that haven't been updated (and e.g. might already not function properly inside Flatpak).

> feature parity

If we are being honest, screenshots/screensharing was never a "feature" of X11, it was a big security hole abused as a feature.


> The infrastructure is already there, for 3rd party apps to request these abilities (with the user getting the choice of following through, or denying the request), e.g.:

Why can't the infrastructure include the user simply allowing the app to continue to have access without a specialized infrastructure of app request (that requires updating the app)?

> really just 3rd party apps that haven't been updated

What's with the downplaying "just" again? Have all the best or most popular apps been updated to ensure no disruption?

> screenshots/screensharing was never a "feature" of X11, it was a big security hole abused as a feature.

So what was the screenshot feature of X11? Also classifying use as abuse isn't that honest, only the lack of security is


> So what was the screenshot feature of X11?

In case I wasn't clear enough, there was never a screenshot feature. Any X11 client could read any pixel written by any other X11 client.

> Have all the best or most popular apps been updated to ensure no disruption?

I am not familiar with DE-agnostic "screenshot apps" for Linux, they always seemed more common on other OSes, and I've always used the DE-specific apps (which were the first to support such mechanisms, some of them even using more direct DE-specific private protocols instead of XDG Portals).

But I spent a few seconds googling for general screenshot apps, found Flameshot (which makes sense as a cross-platform app), and it turns out that support for the XDG Portal approach was added to it almost 5 years ago:

https://github.com/flameshot-org/flameshot/pull/1272

And if you peek around the diff, you can tell that KDE/GNOME-specific support, on Wayland - using DBus but not the XDG Portals protocol - already existed, in early 2021, in fact...

https://github.com/flameshot-org/flameshot/commit/a5df852268...

That's the commit that added KDE/GNOME-specific Wayland screenshot support.

8 years ago, in a 3rd-party app!

> Why can't the infrastructure include the user simply allowing the app to continue to have access without a specialized infrastructure of app request (that requires updating the app)?

I'd forgotten that this happened, but for screensharing from a X11 client, someone already went through the trouble of emulating it (on top of the XDG Portals + PipeWire infrastructure):

https://blog.davidedmundson.co.uk/blog/xwaylandvideobridge/

It's only a temporary hack, and it only matters for X11 clients running under XWayland - if an app can run as a native Wayland client, it should have XDG Portals-based implementations of relevant features.

> What's with the downplaying "just" again? Have all the best or most popular apps been updated to ensure no disruption?

Am I downplaying, or are you describing a vague category of "the best or more popular apps" without giving examples?

I feel like it's too easy for some of this stuff to end up in FUD-like arguments without considering the objective reality (of how far we've come in the past few years etc.).

Anyway, my subjective take is that X11 took a decade or two too long to die, and most (if not all) gripes users might have with Wayland can be traced back to X11 outliving its UNIX Workstation origins and having never been designed as a Personal Computing graphical environment.


> In case I wasn't clear enough, there was never a screenshot feature. Any X11 client could read any pixel written by any other X11 client.

Which is a load of FUD, the X11 security extensions from (checks google) 1996, restrict this.


The 1996 extension had severe limitations. Untrusted clients have no clipboard, but also no GPU acceleration at all and other features were barely tested using it so it was somewhat random if they would work. It breaks a ton of applications and was therefore used by approximately no one.


Ok, so instead of a couple UAC style prompts for screen readers, macro recording, desktop sharing, etc, and some tweaks to GDK, we got what? An entire new backend GDK windowing system, and a pile of broken applications? And its been decades?

And its not like actual flaws people found couldn't be fixed.

There is a word for this.


Did you consider that maybe when you hold an opinion different than the people actually knowledgeable about a topic - like the people developing desktop environments and the former developers of X building Wayland - it might be because you are wrong and have a poor understanding of the field and not because they want to annoy you?

The flaws were not limited to the 1996 poor security extensions. These kind of half broken extensions are everywhere in X11. At some point, if the tweaks you have to do is basically rewriting the whole rendering pipeline and adding new APIs for the most significant systems, what you are doing is strictly équivalent to writing a new piece of software which is exactly what the people behind Wayland did.

And don't worry, the change adverse people you see here complaining about limitations fixed years ago would be complaining the same if the effort was on rewriting part of X11. That's life. Armchair complainers and keyboard warriors will complain while actual doers push things forward.


> Which is a load of FUD, the X11 security extensions from (checks google) 1996, restrict this.

Wait, what ? X11 has extensions ? As in can be "extended" ? And has the same thing since ( for the sake of dialogue) 1996 ? That't why it must die. We need a monolith window system, with clear versions, all incompatible with each other. Only then, real progress can be made. /s


> All your monitors are combined into a large canvas where every pixel can be written and read by any X11 client.

Heh. I learned this hard way when trying to set up multiple monitors with different refresh rates


If your Linux distribution is handling Mesa packages correctly, you will never lack OpenGL/Vulkan drivers.

The reason is that Mesa includes "software rendering" drivers for both OpenGL ("llvmpipe") and Vulkan ("lavapipe"). As the name(s) might suggest, they use LLVM to JIT shaders for your CPU (supporting SIMD up to AVX2, last I checked - although typical compositing shaders tend to get pattern-matched and replaced with plain `memcpy`s etc.).

So you should always be able to run a fully-featured Wayland desktop (albeit limited in performance by your CPU), on any unaccelerated framebuffer, nowadays (and I remember doing this even before Plasma 6 launched, it may be older than usable Wayland desktops tbh - the Mesa code sure is, but maybe distros hadn't always built those drivers?).


Software OpenGL rendering technically works, but is IME unusably slow for compositing. What does work okay is direct software rendering. It might work to configure OpenGL to hit all the fast paths in the software backend (which you might need to add first), but I'm really not sure if you can do it without at least some unnecessary data copying.


you missed the "3d accelerated" part.


Not sure if there's a better section elsewhere, but at least this one touches on it: https://doc.rust-lang.org/reference/destructors.html#tempora...


This is effectively a rustdoc bug, the pattern is very much part of the body of the function, not the signature (and this split is explicitly represented in the compiler IRs, which rustdoc effectively undoes).

While it's not really how it works, the closest valid Rust that comes to mind is:

    const foo: fn(u32) = |mut x| {};
What rustdoc should be doing is stripping binding modes (`ref` and/or `mut`), or even whole patterns - with more complex patterns, like `(a, mut b): (T, U)`, it can be beneficial to keep `(a, b)` as a way to label the two components, but the `mut` is superfluous for documentation.


GNOME has kind of always been on the opposite end of the configurability spectrum from KDE, IME.

But also I'm not super clear on how libinput fits into the picture, I think there used to be some Synaptics-specific integration in certain places that I never went back to, to confirm the differences (switched to libinput several years before, and I've completely forgotten since what I was trying to solve back then).

Anyway, I ended up writing a mostly-personal-anecdote below, that would likely not help, so you can probably skip it (if you do want to try anything, KUbuntu and https://neon.kde.org are both on top of Ubuntu, and there's other distros, but I've not kept up with any distro I might recommend)

---

I tried Ubuntu (w/ the GNOME default) once (back in KDE 4 times IIRC) and the lack of almost any flexibility felt like I was stuck in a sandbox (a literal one, like from when we were kids, not that it's easy to remember that far back).

The only other time I felt like that was when a friend gave me their old iPhone (6 IIRC?) "to see what it's like" and I had to give up trying it out because both the OS, and the few apps I could find (for its max supported iOS version), had random chunks of features missing (tbh I should've jailbroken it, then it'd just be one more piece of hardware I have no immediate use for - but I digress)

KDE is far from perfect (switched to Wayland recently, and been tracking a few QoL leaps in the next couple releases), but I can at least try to tweak it - same goes for using NixOS or ZFS tbh (not going to even defend those, but I have done both Gentoo-like shenanigans, and randomly RAIDed a dual-SSD laptop, respectively, quite painlessly despite not preparing for it from the start, and the weirdness budget is personally paid for a thousandfold).

Meanwhile I run into e.g. GNOME apps using libadwaita nowadays needing environment variables (that appear deprecated?) to apply the KDE gtk integration theme so they don't stick out like a desaturated winamp skin.

I've never felt like "power user" applied well to myself, like I'm not doing the equivalent of weight-lifting for computers, never want to be doing sysadmin if I can avoid it, I just want to have enough control to make things seamlessly neat for myself.

Opinionated defaults are great in the same way you'd set up a house for a marketing reel, but if I actually buy it, why would I have to deal with a landlord telling me I can't paint the walls or move/replace some furniture to maximize my comfort?

The "personal" in "personal computing" is supposed to be the same as the one in "personal property", and so I'd have similar expectations of "can screw with it without asking for permision" for both (modulo real estate being seen as an investment, and building codes, etc. - should've picked a smaller example than a house, oh well).

To stretch the house analogy further, just like I have leased (some corporation's) private property as office spaces, I would be fine to doing the same (also for business reasons) with e.g. a cloud platform (most likely through GHA, whenever they announce the paid tiers) - tho be fair to everyone, this applies far more to walled gardens than opensource software like GNOME.

Also, to be clear about the "sandbox" thing: sandboxes (and/or ideally more objcap systems) are great, and I think that the XDG Portal work is incredible for what it allows: the apps are getting sandboxed, the user getting more power over them.

Even without Flatpak (which I keep meaning to try out), I was happy to see e.g. the KDE Wayland screenshare dialog outright has an option for "create virtual screen" (which can further be configured in the KDE settings, and e.g. partially overlapped with a physical monitor, etc.). If the app was in charge, they would barely enumerate some of the windows correctly, let alone provide new virtual screens.


So what are you using or recommending? Recently switched to kubuntu from gnome-ubuntu, and while some pain points have gone away (lack of global menu can partially be kindof mitigated, didn't have to fiddle with touch acceleration like on 20.04), I'm not impressed: middle-click gets in the way a lot and can't be switched off, power mgmt tray is lagging badly behind power events, weird dock preview and not-so-great app switching, somewhat fiddly touch targeting at times, FF crashing, updates not smooth, insists on chromium as default browser, point-/tasteless Windows-y sounds and looks, ...

All the while there are ZERO GUI apps or other capabilities I'm using that I didn't already use 15 or 20 years ago.

Eying a return to Mac OS which at least doesn't feel like you're treated as guinea pig by dicks with attitudes. Linux notebooks seem barely good enough for uninspired enterprise work on bloated IDEs and Docker/other container crap only there because said dicks couldn't agree on a set of (really old) lib versions and gui toolkits.


  > middle-click gets in the way a lot and can't be switched off,
What do you mean by this? What happens when you middle click, that you don't want or expect to happen?

  > not-so-great app switching
How do you like your app switching? I agree that the default isn't great, but in System Settings you can tweak it beyond all reason. I can help if you tell me what you want.

  > FF crashing
I haven't had that happen. What addons do you have in Firefox?

  > insists on chromium as default browser
Again, System Settings. Or, from within the Firefox settings, there might be a way to set it as default.

  > apps or other capabilities I'm using that I didn't already use 15 or 20 years ago
What apps or capabilities do you feel are missing?


Half of the time I want to press right-click or even left-click on the touch pad it actually registers as middle-click which is very annoying as it means a window gets closed rather than focussed; don't want middle-click at all. Undesired Middle-click-paste also happens a lot.

When I click a link from Thunderbird (weirdly as it sounds when Thunderbird is a Mozilla app) it opens chromium; dialog to set FF as default doesn't change this.

I need to retest and maybe reconfigure app switching from a (vertical) dock as you say; just doesn't feel fluent as it is.

Occasional FF crashing should probably be addressed at Moz. Maybe it's because FF on gnome gets more usage and testing. Btw, does SuSE (or Manjaro) still have a global-menu patch for FF because kubuntu (understandably) doesn't maintain such patches?


I don't use KDE or Wayland - I use awesome-wm with X11. I don't get a lot of Firefox crashes but one odd behavior I have noticed is that sometimes, after closing all Firefox windows, the firefox process continues to run (unresponding) in the background.

When this occurs, clicking links results in no browser windows opening. At this point I run `killall -9 firefox` and oddly, in that very moment, the link I clicked will open up in Chromium.

No idea why this happens.


> Half of the time I want to press right-click or even left-click on the touch pad it actually registers as middle-click

That's a hardware issue. If I press Q I am sure it will not be registered as ctrl+alt+Y for example.


  > Half of the time I want to press right-click or even left-click on the
  > touch pad it actually registers as middle-click
Are you sure that's not a hardware issue? This doesn't happen in other OSes on the same hardware?


Thanks for these helpful comments guys!

I guess I can remap middle-click to left-click using the wayland equivalent of xinput from a shell script or something. The apparent-FF-crash story described by johnmaguire and eddyb sounds very much like another thing I believe is happening on my notebook. Between these fuckups and the general alienation going on in Linux land (wayland, snaps/flatpack, systemd) I've got to say I'm not enthusiastic to go through those chores and teething probs just to get the same-old gui apps like Inkscape and GIMP (plus FF/Thunderbird) running, so right now I'm leaning to go back to Mac OS more than ever (used Mac OS on and off from 2003 to 2016).


FF crashing

(note that the preferred abbreviation is Fx, not FF)

Firefox tends to very much like its shared memory buffers and crashes with SIGBUS if they run out: https://bugzilla.mozilla.org/show_bug.cgi?id=1245239

How much shm is enough depends on both the browser's rendering engine and the display engine, but apparently 1.5GB shm use isn't out of the ordinary. You won't find any references to shared memory (or even general out of memory messages) if Firefox crashes in this way, so it's just another thing to keep an eye on.


> note that the preferred abbreviation is Fx, not FF

Abbreviations is never something up to the original creator to dictate. And "fx" is alredy commonly used for effect.

Next you will tell us that it's actually called Plasma Desktop and not KDE.


I have had KDE Neon running on one or more devices the last 6 or so years.

It is built on an Ubuntu base and is almost like Kubuntu, except I think it is set up by people in the KDE community.

For me the difference felt enourmous.


> So what are you using or recommending?

I am using KDE (aka Plasma5) in Wayland mode, on NixOS unstable.

I would not recommend NixOS, just like I mentioned, and I didn't really want to get into the weeds of why, but while Nix is something that more people should try out if they're already familiar with unfortunate asymmetries (e.g. "git's data model is really nice" vs "git's CLI has sharp edges and some workflows the data model implies are entirely unserviced") and/or like to play around with experimental unpolished software, I would maybe avoid it until they actually come up with a "more declarative"/"less computational" flavor for 99% of usecases.

I've used openSUSE in the past, and while YaST2 might be less relevant now, it was shocking how much similar things were outright lacking back then (a lot of this was pre-NetworkManager to be quite fair).

A lot of people like Arch, and if Debian/Ubuntu package management doesn't get in the way I suppose KDE Neon might be nice? (I keep forgetting KDE's Discover exists, it might also help with not dealing with package management directly)

---

> FF crashing

Quite ironically, if it is https://bugzilla.mozilla.org/show_bug.cgi?id=1743144 that's technically a gtk limitation (not only does it lead to the FF main thread having to poll gtk often enough to keep the Wayland connection from breaking, but when it does break it calls `_exit` so Firefox can't even do crash reporting, and they refused a patch to address this), and it can also happen for Chrome (which also uses Wayland through gtk AIUI).

If you want to check if it is the case, you can look in the logs (e.g. through `journalctl -o with-unit -r`) for "error in client communication".

> middle-click gets in the way a lot and can't be switched off

Are you talking about the feature controlled by System Settings -> Input Devices -> Mouse -> "Press left and right buttons for middle-click"? (that is, if you intentionally press both buttons, does it trigger middle-click?)

AFAIK that's off by default, but I am on a different distro and running KDE/Plasma 5.25.4 and maybe it changed at some point, or maybe it's specific to touchpads? (which I sadly can't test because I only have an older Nvidia laptop, that can't use the Wayland-compatible drivers, or rather I would have to switch to nouveau first and deal with that etc.)

> insists on chromium as default browser

I've had issues with this in the past, some apps provide their own configuration instead of going through XDG mechanisms, or at least have suboptimal defaults.

I would check the settings of the apps which cause chromium to start, and maybe play around with Flatpak/forcing the use of XDG Portal, but that might be too much to ask.


> GNOME has kind of always been on the opposite end of the configurability spectrum from KDE, IME.

which gets weird, when they push on you things like:

- opening WiFi-portals in whatever crap browser your distro ships with GTK. Executing random JS code in the process.

- make XWayland startup absolutely inconfigurable and hardcoded in C (or was it Vala) and for whatever reason MAKE IT LISTEN ON ABSTRACT SOCKETS (which no Xconfig anywhere else does)! (problem here: if any user-namespace container shares your network namespace and you do a naive xhost +SI:localuser:user - it works for any container, because abstract sockets are not stowed away in the filesystem).

At least the first issue I'd like to report, but don't know how and where...


> Maybe it would be called #[expect(deprecated)]

Both this and the reply saying you should open an RFC worry me that it was tongue-in-cheek.

Or maybe it's the obvious name choice, heh, because it literally exists with that very syntax: https://play.rust-lang.org/?version=nightly&mode=debug&editi...

Not stable yet, as you can see, but all the RFC and implementation work has been done, and a stabilization report was even put forward back in July: https://github.com/rust-lang/rust/issues/54503#issuecomment-...


That's hilarious, I had no idea! I'm glad I didn't embarrass myself by proposing an RFC. Thanks for the links!


> but also parts of rustc are implemented in a dynamic library ("librustc_driver")

Nit: 100% of rustc is found within librustc_driver-*.so (I mention it in https://news.ycombinator.com/item?id=32329062 - which also touches on stable ABI).

`rustc` just calls into the .so without any custom configuration: https://github.com/rust-lang/rust/blob/e141246cbbce2a6001f31...

(but rustdoc/clippy/miri all have additional configuration, passes, etc. specific to them)

Also, if you look at the `rustc` executable's size:

    $ du -bh ~/.rustup/toolchains/nightly-*/bin/rustc
    17K     /home/eddy/.rustup/toolchains/nightly-2018-01-01-x86_64-unknown-linux-gnu/bin/rustc
    2.9M    /home/eddy/.rustup/toolchains/nightly-2021-12-02-x86_64-unknown-linux-gnu/bin/rustc
    2.9M    /home/eddy/.rustup/toolchains/nightly-2022-01-13-x86_64-unknown-linux-gnu/bin/rustc
    2.7M    /home/eddy/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc
17K is what it should be, I'm not sure where the extra 2.7M are coming from - my current suspicion is that it's entirely statically linked jemalloc, which I guess we didn't want to impose on all users of rustc_driver, but it might be worth doing it anyway.


No, at most we would've invested more in e.g. ensuring¹ you cannot accidentally (or intentionally) mix-and-match shared objects not built by the same build process, or in ways to reduce the cost² of safe ("full relro") dynamic linking.

¹ via forcing the dynamic linker to eagerly resolve some symbols, global constructors, etc. - see https://github.com/rust-lang/rust/issues/73917#issuecomment-... for some recent discussion

² such as exporting a single base symbol per dylib (with a hash of code/data contents in the symbol name, for integrity), and statically resolving imports (to constant offsets from that base symbol), when linking object files into dylibs/executables - because of how position-independent code indirects through the "GOT", in practice this would mean that the dynamic linker would only need to lookup one symbol per dependency .so instead of one per import (of which there can be hundreds or thousands)

Also, "dynamic linking" in Rust was never really "dynamic" or about "drop-in replacement to avoid rebuilds", it was only about the storage reuse of "shared objects", with the "late-binding" semantics inherent in dynamic linking a negative, not a positive.

For example, rustc, rustdoc, clippy, miri, etc. are all relatively small executables that link against librustc_driver-*.so - on a recent nightly that's 120MiB, with only static linking that'd be half a GiB of executables instead. The relationship is "one .so shared between N executables" not "one .so per library". Also, if we could remove runtime symbol lookup but keep the separate .so (like ² above), we totally would.

---

At the same time, Rust's continued evolution depends on being able to (almost) "never ask for permission" when changing internals that were never guaranteed to be one thing or another.

Rust recently fixed most platforms to use simpler and more performant locks based on futex(-like) APIs (see https://github.com/rust-lang/rust/issues/93740 and PRs like https://github.com/rust-lang/rust/pull/95035), and this meant that e.g. Mutex<T> has a 32-bit (atomic) integer where a pointer used to be.

An even more aggressive change was decoupling Ipv{4,6}Addr/SocketAddrV{4,6} from the C types they used to wrap internally (https://github.com/rust-lang/rust/pull/78802) - that one actually required library authors to fix their code in a few cases (that were very incorrectly reinterpreting the std types as the C ones, without this ever being officially supported or condoned).

Imagine trying to do that in C++. The stories I keep hearing from people involved in the C++ standard are tragic, more and more otherwise-uncontroversial improvements are being held back by poorly-motivated stubbornness around ABI stability. Large-scale users of C++, like Google, have long since migrated from C++'s standard library to their own replacements (e.g. abseil) for many of their codebases, and I kept hearing Google were "forking C++" (over the frozen ABI debacle specifically), long before it came out as "Carbon".

---

The technology to coherently³ avoid today's rebuild-the-world cost in compiled languages⁴ isn't here yet IMO.

³ as in, guarantee identical behavior to the full rebuild from scratch, without introducing inconsistencies that could cause logic errors, memory corruption, etc.

⁴ some/most C libraries of course being the biggest exception - and while it's possible to intentionally design libraries like this in any language that can do FFI, it's fundamentally antithetical to "zero-cost abstractions" and compiler optimizations - nowadays the latter are forced even on otherwise-patchable C dependencies via LTO, in the interest of performance

What I'm referring to is incremental recompilation with the following properties:

...

[oops, hit comment size limit! I'll post the rest separately]


Has the Rust compiler team considered using a busybox-style approach (single binary, multiple entry points) for rustc and friends? Even Windows supports hard linking, so AFAIK this should be feasible.


We explicitly support building against librustc_driver-*.so for both "custom drivers" (what we call those binaries I mentioned) and generally "rustc as a library" usecases. We should maybe rename it to librustc and remove as much potentially-confusing "driver" terminology as possible.

Pinning a nightly and installing the "rustc-dev" rustup component are both necessary, the former because internal APIs of rustc aren't stable, but it's a supported usecase.

Both clippy and miri are developed out-of-tree like that, and sync'd using `git subtree` (or `git submodule` but we want to move everything to subtree).


[continued from above due to size limit]

What I'm referring to is incremental recompilation with the following properties:

1. automatic correctness

` - that is, a compiler change not explicitly updating anything related to dependency tracking should at most result in conservative coarser-grained behavior, not unsoundly cause untracked dependencies

` - manual cache invalidation logic is a clear indicator a compiler isn't this - e.g. this rules out https://gcc.gnu.org/wiki/IncrementalCompiler (maybe also Roslyn/C# and the Swift compiler, but I'm not sure - they might be hybrid w/ automated coarse-grained vs manual fine-grained?)

` - the practical approach to this is to split workload into work units (aka tasks/queries/etc.) and then force information flow through centralized "request"/"query" APIs that automatically track dependencies - see https://github.com/salsa-rs/salsa for more information

` - research along the lines of ILC (Incremental Lambda Calculus) might yield more interesting results long-term

` - outside of a compiler, the only examples I'm aware of are build systems like tup (https://gittup.org/tup/) or RIKER (https://github.com/curtsinger-lab/riker) which use filesystem sandboxing (FUSE for tup, ptrace/seccomp-BPF for RIKER) for "automatically correct" build dependencies

2. fine-grained enough

` - at the very least, changes within function bodies should be isolated from other bodies, with only IPOs ("interprocedural optimizations", usually inlining) potentially introducing additional dependencies between function bodies

` - one extreme here is a "delta-minimizing" mode that attempts to (or at least warns the developer when it can't) reduce drift during optimizations and machine code generation, such that some fixes can end up being e.g. "patch a dozen bytes in 200 distro packages" and get quickly distributed to users

` - but even with history-agnostic incremental recompilation (i.e. output only depends on the current input, and past compilations only affect performance, not other behavior), function-level incremental linking (with one object file per function symbol) could still be employed at the very end, to generate a binary patch that redirects every function that grew in size, to additional segments added at the end of the file, without having to move any other function

3. propagating only effective "(query) output" changes

` - also called "firewalling" because it blocks irrelevant details early (instead of redoing all work transitively)

` - example 1: editing one line changes the source byte offsets of everything lower down in the file, but debuginfo doesn't need to be updated since it tracks lines not source byte offsets

` - example 2: adding/removing explicit types in a function should always cause type checking/inference to re-run, but nothing using those type inference results should re-run if the types haven't changed

4. extending all the way "down" (to machine code generation/object files/linking)

` - existing optimizing compilers may have a hard time with this because they didn't design their IRs/passes to be trackable in the first place (e.g. a LLVM call instruction literally contains a pointer to the function definition, with no context/module/pass-manager required to "peek" at the callee body)

` - can be theoretically be worked around with one object file per function, which https://gcc.gnu.org/wiki/IncrementalCompiler does mention (under "Code Generation" / "Long term the plan is ..."), but that alone isn't sufficient if you want optimizations (I've been meaning to write a rustc proposal about this, revolving around LLVM's ThinLTO having a more explicit split between "summary" and "full definition")

5. extending all the way "up" (to lexing/parsing/macros)

` - this is probably the least necessary in terms of reducing output delta, but it can still impede practical application if it becomes the dominant cost - AFAICT it's one of the factors that doomed the GCC incremental experiment ("I’m pretty much convinced now that incremental preprocessing is a necessity" - http://tromey.com/blog/?p=420)

` - outside of "rebuild the world", this also makes a compiler much more suitable for IDE usage (as e.g. a LSP server)

My experience is mostly with the Rust compiler, rustc, whose incremental support:

- started off as coarse skipping of generating/optimizing LLVM IR

- has since evolved to cover 1./2./3.

- while many passes in the "middle-end" were already both "local" and "on-demand", sometimes allowing incrementalization by changing only a dozen lines or so, 4. is indefinitely blocked by LLVM (at best we can try to work around it), and 5. (incremental "front-end") has been chipped at for years with several factors conspiring to make it orders of magnitude more difficult:

` - macro expansion and name resolution being intertwined (with the former mutating the AST in-place while the latter being a globally stateful algorithm)

` - "incremental front-end" used to be seen as essential for IDE usage and that notion started falling apart in 2018 or so (see rust-analyzer aka "RA" below, though RA itself is not itself directly responsible, nor do I hold it against them - it's a long messy story)

` - this work (without the IDE focus, AFAICT) has mostly driven by random volunteer work, last I checked (i.e. without much "official" organization, or funding, though I'm not sure what the latter would even be)

- its design has been reimagined into the "salsa" framework (https://github.com/salsa-rs/salsa - also linked above)

` - most notable user I'm aware of is the rust-analyzer (aka "RA") LSP, which hits 1./2./3./5. (4. isn't relevant, as RA stops at IDE-oriented analysis, no machine code output) - without getting too into the weeds, RA is a reimplementation of "Rust {front,middle}-end", and ideally eventually rustc should be able to also be just as good at 5. (see above why it's taking so long)

I am not aware of any other examples of industrial compilers having both 1. and 2. (or even academic ones but I suspect some exist for simple enough languages) - every time someone says "incremental compilation? X had that Y decades ago!", it tends to either be manually approximating what's safe to reuse (i.e. no 1.), or have file-level granularity (i.e. no 2. - the "more obviously correct" ones are close to the "separate compilation" of C/C++/many others, where a build system handles parallel and/or incremental rebuilds by invoking the compiler once per file), or both.

While 2./5. are enough to be impressive for IDE usage all by themselves, the interactive nature also serves to hide issues of correctness (lack of 1. may only show up as rare glitches that go away with more editing) or inefficiencies (lack of 3. leading to all semantic analyses being redone, may still be fast if only requested for the current function).

Another thing I haven't seen is distro build servers talking about keeping incremental caches around (and compiling incrementally in the first place), for builds of Rust "application" packages (most of them CLI tools like ripgrep, I would assume) - it's probably too early for the smaller stuff, but a cost/benefit analysis (i.e. storage space vs time saved rebuilding) would still be interesting.


Don't you want "interrupt ABIs" for that specific usecase?

IIRC `#[naked]` are only really guaranteed to allow inline assembly, so it's like defining a function in `global_asm!` except for monomorphization and generally interfacing better with the rest of the language.


I'm not aware of any proposal for interrupt ABIs, but in the meantime naked functions essentially allow you to create ad-hoc interruput ABIs. Do you have a link?


https://github.com/rust-lang/rust/issues/40180 was the first one; and Phil has created a pre-RFC a few days ago: https://internals.rust-lang.org/t/pre-rfc-interrupt-calling-...

(That said you're right that naked functions still have a place for doing this, I can't imagine the compiler would grow stable support for every random ABI someone could come up with. But for more well-known ones it's a nice thing.)


It's a shame LLVM isn't more explicit about calling conventions (especially since the front-end has to lower most of the platform-specific C ABI details, short of the exact list of registers or stack offsets, into LLVM IR anyway).

(if it was more expressive, we could have e.g. specifiers similar to inline `asm!`, but on a function signature, to fully describe its call ABI)

In the meanwhile, I agree - if you need some entry-point to have a custom ABI (especially if you need it parameterized through generics), `#[naked]` + inline `asm!` is the only viable option for now.

Especially with const generics, you can do some cool stuff to make very compact trampolines that e.g. embed immediates directly into instructions (though it's sad that `const` operands in inline `asm!` aren't stable yet).

But for interrupt ABIs I would hope that LLVM can fully support them and avoid any inefficiencies that "whole body is one inline `asm!`" could create by blocking LLVM from doing optimizations, clever register allocation, etc.

(that is, they let the user control when they do something special with inline `asm!` vs regular code that LLVM understands)


> For instance `(*role).name` creates a `&mut &'static str` behind the scenes which is illegal, even if we can't observe it because the memory where it points to is not initialized.

Where is this coming from? It's literally not true. The MIR for this has:

        ((*_3).0: &str) = const "basic";
        ((*_3).2: u32) = const 1_u32;
        ((*_3).1: bool) = const false;
So it's only going to do a raw offset and then assign to it, which is identical to `*ptr::addr_of_mut!((*role).field) = value`.

Sadly there's no way to tell miri to consider `&mut T` valid only if `T` is valid (that choice is not settled yet, AFAIK, at the language design level), in order to demonstrate the difference (https://github.com/rust-lang/miri/issues/1638).

The other claim, "dereferencing is illegal", is more likely, but unlike popular misconception, "dereference" is a syntactic concept, that turns a (pointer/reference) "value" into a "place".

There's no "operation" of "dereference" to attach dynamic semantics to. After all, `ptr::addr_of_mut!(*p).write(x)` has to remain as valid as `p.write(x)`, and it does literally contain a "dereference" operation (and so do your field projections).

So it's still inaccurate. I believe what you want is to say that in `place = value` the destination `place` has to hold a valid value, as if we were doing `mem::replace(&mut place, value)`. This is indeed true for types that have destructors in them, since those would need to run (which in itself is why `write` on pointers exists - it long existed before any of the newer ideas about "indirect validity" in recent years).

However, you have `Copy` types there, and those are definitely not different from `<*mut T>::write` to assign to, today. I don't see us having to change that, but I'm also not seeing any references to where these ideas are coming from.

> I'm pretty sure we can depend on things being aligned

What do you mean "pretty sure"? Of course you can, otherwise it would be UB to allow safe references to those fields! Anything else would be unsound. In fact, this goes hand in hand with the main significant omission of this post: this is not how you're supposed to use `MaybeUninit`.

All of this raw pointer stuff is a distraction from the fact that what you want is `&mut MaybeUninit<FieldType>`. Then all of the things about reference validity are necessarily true, and you can safely initialize the value. The only `unsafe` operation in this entire blog post, that isn't unnecessarily added in, is `assume_init`.

What the author doesn't mention is that Rust fails to let you convert between `&mut MaybeUninit<Struct>` and some hypothetical `&mut StructBut<replace Field with MaybeUninit<Field>>` because the language isn't powerful enough to do it automatically. This was one of the saddest things about `MaybeUninit` (and we tried to rectify it for at least arrays).

This is where I was going to link to a custom derive that someone has written to generate that kind of transform manually (with the necessary check for safe field access wrt alignment). To my shock, I can't find one. Did I see one and did it have a funny name? (the one thing I did find was a macro crate but unlike a derive those have a harder time checking everything so I had to report https://github.com/youngspe/project-uninit/issues/1)


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: