Well, originally what was wrong was that non-vertex customers got much lower limits and about twice the time to first token as vertex customers. So I used a vertex key setup. However, the cli had me in auth hell trying to connect up with vertex. Like literal spinning. So, thank you. I wiped my vertex environment variables and upped my gemini key level, and it mostly works now.
https://ai.google.dev/gemini-api/docs/api-key?authuser=1