Passport is pretty large so it can be confusing. IMO, it's much easier to not use the session stuff in passport and just do your own thing letting passport handle the flow. You can use the BasicAuth strategy on a /login url to sign someone in and grant a token, and then use Bearer auth strategy to check the token on the rest of your urls.

Doing it that way, Passport doesn't require an ORM at all. You'll need to obviously provide a way to auth a user and verify a token, but that's then up to you.

Now, if you want to actually use OAuth it can get complicated because the flow.