This is unfortunately just the physics of a fixed lens system. You’re experiencing https://en.wikipedia.org/wiki/Vergence-accommodation_conflic...

To overcome this, there would need to be a significant development in optics to enable varifocal lenses.

Meta have made a few public attempts but the results have huge tradeoffs in weight, mechanics and response time with todays technology.