My starting point would be Chromium's documentation, as - presumably - chrome is one of the most widely used and battle tested, user-facing, third party sandboxes running on end user machines.
With the caveat that I wouldn't necessairly assume this is the cutting edge at this point, and there might be other resources to invest in for server-side sandboxing involving containers or hypervisors, and that I've only actually engaged with the Windows APIs based on that reading.
Windows: https://chromium.googlesource.com/chromium/src/+/main/docs/d...
Linux: https://chromium.googlesource.com/chromium/src/+/main/sandbo...
OS X: https://chromium.googlesource.com/chromium/src/+/main/sandbo...
With the caveat that I wouldn't necessairly assume this is the cutting edge at this point, and there might be other resources to invest in for server-side sandboxing involving containers or hypervisors, and that I've only actually engaged with the Windows APIs based on that reading.
I wrote `firehazard` ( https://docs.rs/firehazard/ , https://github.com/MaulingMonkey/firehazard/tree/master/exam... ) to experiment with wrapping the Windows APIs, document edge cases, etc. - although if the long list of warnings in the readme doesn't scare you away, it'll hopefully at least confirm I hesitate to recommend my own code ;)