It might be worth using a lightness estimate like OKLab, OKLrab[1], or CIE Lab instead of the RGB luminance weighting, as it should produce a more perceptually accurate result.
The other issue with your code right now, is that it is using euclidean distance in RGB space to choose the nearest color, but it would be probably also more accurate to use a perceptual color difference metric, a very simple choice is euclidean distance on OKLab colors.
I think dithering is a pretty interesting area of exploration, especially as a lot of the popular dithering algorithms are quite old and optimized for ancient compute requirements. It would be nice to see some dithering that isn't using 8-bits for errors, is based on perceptual accuracy, and perhaps uses something like a neural net to diffuse things in the best way possible.
If you are interested in color dithering with different color difference metrics [1], I've implemented just that [2]. You can find an example comparing metrics in my docs [3].
If you want to do true arbitrary palettes you also need to do projection of the unbound Oklab space onto the convex hull of the palette points. This is a tricky thing to get right, but I've found that the Oklab author's published gamut clamping for sRGB also translate well to arbitrary convex hulls.
I moved my canvas library's reduce-palette filter over to OKLAB calculations a while back. The calculations are more computationally intensive, but worth the effort.
It's an array of pre-calculated values that I extracted from an image donated to the Public Domain by Christoph Peters (the link is an interesting read about bluenoise - recommend!) - http://momentsingraphics.de/BlueNoise.html
No textures or masks, just brute computing on the CPU.
The other issue with your code right now, is that it is using euclidean distance in RGB space to choose the nearest color, but it would be probably also more accurate to use a perceptual color difference metric, a very simple choice is euclidean distance on OKLab colors.
I think dithering is a pretty interesting area of exploration, especially as a lot of the popular dithering algorithms are quite old and optimized for ancient compute requirements. It would be nice to see some dithering that isn't using 8-bits for errors, is based on perceptual accuracy, and perhaps uses something like a neural net to diffuse things in the best way possible.
[1] https://bottosson.github.io/posts/colorpicker/