Whether you agree with it or not, at least the Patriot Act was passed by Congress and not simply Executive Orders because it's too inconvenient to work with Congress on legislation.
It's very difficult to challenge because the secret nature of it.
In order to sue, you have to prove standing and in order to have standing, you have to know you were harmed. It's hard to prove you were harmed if everything is top secret.
Whether you agree with it or not, at least the Patriot Act was passed by Congress and not simply Executive Orders because it's too inconvenient to work with Congress on legislation.