Windows IoT Core is a pretty good RTOS, competitive with VxWorks. The NT kernel is good engineering. It’s the Win32 userland baggage that causes all the problems.
That's why I mentioned IoT Core: it is, in fact, Windows without the Win32 userland. As such, it has no shell other than a web interface, and only runs UWP executables.
If you've ever interacted with an Xbox One in dev-kit mode, that's basically the experience of using Windows IoT Core.