So don't use it to do any operations which require you to absolutely trust the smartphone?
When it comes to interacting with the government, security is often more important than convenience. Having people go to their local police station to scan their card on a secure terminal is definitely worth it for high-profile operations.
Maybe that could work for something like transferring property or other operations most people do only a few times in their lives, but there are tons of medium or low risk governmental things I would love to be able to do on my smartphone.
For that, I'd rather have a relatively secure scheme that people can actually use, rather than a hypothetical perfect one that nobody owns the hardware for.
As a concrete example: My government started out with very high ambitions. Every citizen got a smartcard holding a personal X.509 certificate (usable both for e-government, PDF signature, and S/MIME email signature/encryption!), and the government even gave USB CCID card readers away. The result? Absolutely nobody used it. A few years later, they replaced that with a solution that stores everybody's private keys on a central server, combined with SMS-OTP for signature approval...
Tapping my ID card on my phone and entering a local PIN to sign, even if the phone isn't particularly trustworthy and signature requests could be spoofed, would be much, much more secure than that.
When it comes to interacting with the government, security is often more important than convenience. Having people go to their local police station to scan their card on a secure terminal is definitely worth it for high-profile operations.