That video's amazing because the part of the UE5 demo I can't wrap my head around is where Nanite magically turns 1 billion triangles of source art to 20 million triangles of rendered art, per frame.
I was wondering about that too. My best guess is that they put the heavy geometry items in an acceleration data structure and then ray trace / cone trace these in a first pass to get the points where the camera rays intersect the geometry. Then generate triangles on the fly from these points in every frame. This would be quite similar in flavor to GigaVoxels [1] or the (IMO unjustly much hated) "Unlimited Detail" engine [2], except that instead of shading / visualising the intersections directly they just use it to generate triangles as another input to their overall pipeline.
If you look at the video closely all the heavy geometry items are static, i.e. they don't deform. This matches with a classic limitation of ray tracing as deformations invalidate the acceleration data structures used for making ray tracing efficient.
In the video they mention that the "heavy geometry" you are talking about is 100s of billions of triangles. Storing 1 billion triangles using 32-bit floats takes ~12 Gb, so we are talking about N * 12 Tbs only in triangle storage for this demo. Building an acceleration data-structure on top of this would also take up a lot of storage, so I don't think how you can make this approach successful on a PS5 with limited GPU memory.
To me this tech sounds more like they finally figured out all the quirks around how to properly construct "seamless geometry atlasses" (geometry images) and reconstruct geometry from them, and managed to bring that to production. It required quite a bit of compute, and there were also a lot of quirks about how to automatically "cut" an input geometry so that you can put it in a 2D image, how to compress and decompress the image or parts of it without artifacts (such that you don't get jumps across triangles), etc.
That approach doesn't store any triangles, but a 2D xyz (rgb) image that you can use to reconstruct the original geometry from, and to which you can apply all usual image compression algorithms, use all GPU texture features (but for geometry), etc.
That's ~15 year old tech, and they mention they have been working on this for over 10 years, so timeline-wise it would fit (search for geometry images, texture atlasses, etc. there was a siggraph talk from ~2005 about wavelet compression of geometry images).
It's quite possible that they make heavy use of instancing, which might reduce the memory footprint by a lot. Not every pebble or rock needs an individual geometry.
From the video it sounded to me that the 100s billions of triangles were from single instance geometry. The models they mention in the video are in the ball park of >1 billion triangles each (that's 12 Gb per model). So if they have 100 models of that resolution, that's 12 Tbs, and whether they instance each model 1 or 100 times doesn't matter much because the problem of having to fit those 12 Tb in triangles somewhere still remains.
That's why it seems much more plausible to me that they are not storing triangles anywhere, and instead just storing geometry images.
If one mesh is 1 billion triangles, and you have 100 different meshes, you need to store 100 billion tringles, unless you do something radically differently.
If you have 1 triangle mesh, and instance it 100 billion times, you don't need to do that.
> "more than 33 million triangles."
The assets the company they mention makes have 100s of millions of triangles, and billions as well. So I wonder if they are not using those original assets after all.
IIUC they are, but they are only rendering at lower triangle counts and the LOD happens automatically.
I'm not convinced, because mixing ray-marched and triangulated geometry is quite doable. If you've already computed a ray intersection, generating a triangle and putting a burden on vertex processing makes little sense if the geometry is static. If you wanted it to be deformable, then yes you may want to use a vertex shader. But in this demo, none of the high resolution geometries are deformable AFAICT.
This doesn't really seem volumetric in nature to me.
Firstly, UE4 has broad-phase that I understand they are quite happy with, I doubt they would add in a ray-trace, since not only would that not really improve anything, it would also not integrate well with any non-nanite techniques of rendering geometry, and my understanding is at least for now, nanite can only handle (mostrly)static geometry, and I doubt they would want to give up the ability to work with other rendering techniques.
If I had to take a wild guess, I would guess they have a somewhat standard process to generate a low detail mesh, which is what get's dispatched to the GPU, then, atop of that mesh they may build triangles acceleration structures parameterized across the surface perhaps similar to this paper [1] From there, you could do similar to what you suggest, and just generate additional triangles such as is in that paper.
However, given how poorly GPUs handle pixel-sized polygons, that may not be the best approach. so I wouldn't be surprised if tessellation is used to around a quad level and then from there, the rest is either done using compute-style rasterization in the pixel shader, or some outputs are written and additional rasterization is actually deferred for a compute job.
I'm sure there are all sorts of exceptions and edge-cases, but I wouldn't be surprised if it looks something like that general workflow. On the other hand, I'd be very surprised if there was anything that looked like ray-tracing on the scene level, perhaps traversing the surface-space data structures looks a bit like ray-tracing.
I was wondering about that too. My best guess is that they put the heavy geometry items in an acceleration data structure and then ray trace / cone trace these in a first pass to get the points where the camera rays intersect the geometry. Then generate triangles on the fly from these points in every frame. This would be quite similar in flavor to GigaVoxels [1] or the (IMO unjustly much hated) "Unlimited Detail" engine [2], except that instead of shading / visualising the intersections directly they just use it to generate triangles as another input to their overall pipeline.
If you look at the video closely all the heavy geometry items are static, i.e. they don't deform. This matches with a classic limitation of ray tracing as deformations invalidate the acceleration data structures used for making ray tracing efficient.
[1] http://gigavoxels.inrialpes.fr/ [2] https://www.youtube.com/watch?v=gM6QkPsA2ds