I worked on something very similar for inference on video streams. To avoid the limitations of the config files mentioned in a sibling comment, I added a tool to convert a config to plain Rust. Your primary focus has to be the quality of the Rust API, and the config files are syntactic sugar for the beginning or simpler projects.