I wouldn't be so quick to dismiss it. Maybe that exact situation won't often arise, but in other languages like C++, Java and C# it's not at all uncommon to have nested collections.
Yes sure; but it is rare to have indirection for every layer of a nested collection (i.e. each collection will essentially be a pointer to some data, with a little bit of metadata (like length & capacity for a Vec), so having Box<Vec<T>> is a pointer to a pointer to the data: essentially pointless!).
In other words, one would write
Vec<Vec<Vec<T>>>
That might be considered ugly, but it's not ugliness caused by the `~` change.
I've never seen `~~` except for workarounds caused by the current lack of dynamically sized types (which is being fixed). `~&` is not very useful, as you'd be placing a stack-bounded lifetime on the heap (except for 'static, I suppose, but I've never seen that).