So you set the output timestamp to the newest of the relevant input files as I mentioned. For example a blog post output file has as input the content file, configuration settings and templates. The output timestamp can be set to the newest of those.

Or the sitemap can use the input files to get the timestamps in which case the timestamps of files in the output directory won't matter. They would affect if-modified-since so it is preferable to have them deterministic.