First, let me just say that it never even occurred to me that my iPhone volume would control the TV so I just never even tried it. I would have expected volume buttons to appear in the remote app.
That said, I just tried it and it does not seem to work for me in my setup. I just have a Samsung TV with the Apple TV connected. No other "AV system". The volume buttons do not seem to do anything with the TV. Now that I know it works in some situations I will start digging around.
The Apple TV remote controls the volume of the TV fine but I seem to recall that I programmed it for the power and volume of my TV when I set it up.
I have a specific annoyance about that: I have an Apple TV HD that I bought second hand that had no remote; it's connected to a monitor so there's no TV remote or HDMI-CEC or anything of the sort; when using AirPods with the Apple TV the only means I have to adjust volume is that iOS App, the universal remote I have for otherwise controlling the Apple TV can't adjust it. Oh well. I'm thinking about buying that new remote and taking one of the other siri remotes I already have and putting it on that one.
For me, using both the "built-in" remote app on the control panel and the "real" remote app launched as a proper app, the volume is controlled simply with the iPhone's volume buttons. I'm not sure how it could get any more ergonomic.
