The OP mentions that the COUNT query is a simplification that has the same performance behavior as the original -- in the actual application, the data inside the rows is needed.

Your idea is a good one for when the count is what's needed, though. Even in more complex cases, maintaining aggregate data ready to go can help a lot.