I came across a blog post[1] (HN thread[2]) recently that sheds some light on the issue. The tl;dr is that the PR view can render over 100 000 DOM nodes, many of which are invisible inline SVG nodes, and SPA routing makes navigation a lot slower.
That blog post discovered that hard refreshing the page is faster than GitHub's SPA navigation, which led me to make this browser extension which makes GitHub navigation twice as fast:
[1]: https://yoyo-code.com/why-is-github-ui-getting-so-much-slowe...
[2]: https://news.ycombinator.com/item?id=44799861