I read somewhere that SM64 collisions have no fancy optimization but are just brute force and a first glance at the source seemed to confirm this. Which shows that
1) you really don’t need to optimize prematurely
2) something is very wrong with engines used in modern (indie) games of similar complexity when they perform badly on modern hardware
I don't know how fancy these optimizations are, but it is optimized.
* Tris are sorted by whether they are static (level geometry) or dynamic (moving platforms). So only dynamic tris need to be reinitialized every frame.
* Tris are further sorted into which cells they touch. A cell is a spatial partition of a level. So collision detection only needs to check against tris in your current cell.
* Tris are further sorted into walls, floors, and ceilings (depending on their normal). The find_wall_collisions, find_floor_collisions, etc. functions are each optimized for that particular kind of tri.
* Tris extend their hitbox along a coordinate axis (the up axis for floors/ceilings, the one closest to their normal for walls), instead of along the normal itself. This is cheaper to test, since you can just project down along that axis and it becomes a point-in-tri test in 2D.