There's going to be a lot of refactoring either way, whether there are custom abstraction layers in place, and whether there aren't.

The abstraction layers that were originally intended to ease a transition between ORMs or DBs often end up making such a transition far more difficult. They just end up being another huge amount of code that needs some type of refactoring.

ORM swapping is YAGNI.

IDbSet<T> is enough. If you swap your ORM, you're swapping mindset more than just methods, unless you're mapping ORM entities to POCOs in your repository object.