Overflow being UB is not something you run into easily with typical math, index and size uses (not as often as your run into unsigned issues, in my experience). Yes, bit-twiddling should be unsigned, but it is very easy to make this code isolated, and convert from signed values storing these bits, if necessary.
But I am going to defer to authority here: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p142...