I'm not a fan of huge displays but 4" is actually pretty tame by today's standards. Apple may have come up with the perfect screen size for a SmartPhone circa 2007 but today the vast majority of SmartPhones sold in the world have larger displays so clearly 3.5" cannot be the only perfect size. I do think they have stood by 3.5" in part to see if this trend was just a temporary divergence. Now that is has endured for a couple years it's hard to ignore.
The other factor here is I think as Apple has scaled iOS to include more features they have begun to hit the wall of physical UI space. Let's use Safari as an example. If Apple wanted to add 2 new major features to Safari that had to be quickly accessible where would they even put them? The UI is just about full. They would have to increasingly rely on pop-out controls and other less-than-obvious UI tricks which adds extra taps. The situation is even worse when the keyboard is active taking up half the screen. With the status bar and maybe a visible app toolbar you have about a quarter of the screen available. Reading text messages on an iPhone has actually become painful. You can rarely see the entire message without scrolling or hiding the keyboard. A taller screen fixes that. They may just really need an extra ~170 pixels of height to keep scaling these apps with new features that require UI space.
The other factor here is I think as Apple has scaled iOS to include more features they have begun to hit the wall of physical UI space. Let's use Safari as an example. If Apple wanted to add 2 new major features to Safari that had to be quickly accessible where would they even put them? The UI is just about full. They would have to increasingly rely on pop-out controls and other less-than-obvious UI tricks which adds extra taps. The situation is even worse when the keyboard is active taking up half the screen. With the status bar and maybe a visible app toolbar you have about a quarter of the screen available. Reading text messages on an iPhone has actually become painful. You can rarely see the entire message without scrolling or hiding the keyboard. A taller screen fixes that. They may just really need an extra ~170 pixels of height to keep scaling these apps with new features that require UI space.