I've been struggling with this myself in various cases. The conclusion I've come down to, is that the app has to take some responsibility to provide for consistent backups, either via checkpoints / write barriers, or via a quiesce command that can finalize in-flight transactions, and pause / buffer operations long enough for a volume snapshot to take place. The other thing I've seen apps do is provide a data export functionality, so you end up backing up the export file not the live data. But that requires extra time and disk space. Another option is to run the app in a VM, and do a live VM snapshot which includes the running state of the VM.

Bottom line is if you can't live backup your app, file a bug report with the app vendor. Although it would be really nice if there were standards in this area, so that a backup tool would just need to call one operation to put all supported apps into a hot-backup mode.