I think most of the complexity here comes from trying to write as little unsafe code as possible (and not repeating any unsafe operation in multiple places, hence the many layered abstractions).
If you were to implement Vec without layering, it would be no more complicated than writing a dynamic array in C (but more verbose due to all the unsafe blocks and unabbreviated names)
Notably, some of the abstraction is actually there to prevent the compiler from generating a lot of redundant code. The process of monomorphization (turning polymorphic generic code into usable machine code for particular types) can seriously bloat the size of binaries.
If you were to implement Vec without layering, it would be no more complicated than writing a dynamic array in C (but more verbose due to all the unsafe blocks and unabbreviated names)