I think what you're seeing is aliasing. You need to filter for the Nyquist frequency of whichever is smallest, the input or the output. When upscaling the input is always smallest so it's easy. But when downscaling you need to adjust the width - for a 0.5x scale you need a filter 2x the size. I have my own hand coded Lanczos-5 that I use for all my scaling needs, and it's perfect.