You certainly see OPA replacing IB -- I personally know several examples, though some HPC vendors now seem to be making it difficult to buy. It has a fairly healthy showing at top500, considering it hasn't been widely available for long. I know the Mellanox propaganda, and have some experience of it, along with promises of help to make it work that evaporated.

I can't remember where to look for OPA's features to help MPI implementation, but someone else might be able to comment.