Your point is a good one - what you're describing I've heard referred to as "the novelty effect". In fact, a lot in this article reminds me of another essay critical of the short-sightedness of many A/B tests:

https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-dest...