I mean worry about more than the 90% who have been fired and (presumably) don't have access to NSA computers anymore. Also, I don't think leaking NSA secrets is the main problem. I think their main problem will be finding and retaining talent. Why would you go work a government job like this if you don't even have job security. It's not like there's a shortage of demand for sysadmins in the private sector.
Recall that the CIA really likes to hire Mormons for their loyalty [1]. I'd be REALLY interested in knowing what % of the remaining sys ads are Mormon.