Say you need to get a penetration test for PCI compliance. There are literally hundreds of vendors that offer these services. Your CTO would like to use X vendor because he read a paper / saw their name at Defcon / recommended by a partner.
When the vendor comes to perform a penetration test, they launch a Nessus scan against the target ranges. They compile the results and manually validate the findings to ensure they are not false positives. The end product is a report that looks something like a checklist: SSLv1 in use, self-signed certificates internally, missing the latest third party software patch on a server.
According to the penetration testing firm, you are probably at a low / medium risk level. The tacit implication is that as long as you fix those issues, you should be good to go.
The problem is that first, a vulnerability scanner is an imperfect piece of software and does not test anything a real attacker would. A real attacker might try phishing, or guess "Password1" on a user account. Maybe the attacker would attempt a man in the middle attack or set up an evil hotspot. Once you have AD credentials, now you can find which users have local administrator access, which then you can see if there is a shared Administrator password across all workstations.
The other problem is that the first penetration test does nothing to address potentially systemic issues for why the security vulnerabilities occurred in the first place. The patches could have been missing because there is no formalized patch management program, or inaccurate change management, or an issue with their Puppet config.
Currently there's no way to separate the "good" (read: thorough) from the bad other than direct referral or looking at a sample report.
When the vendor comes to perform a penetration test, they launch a Nessus scan against the target ranges. They compile the results and manually validate the findings to ensure they are not false positives. The end product is a report that looks something like a checklist: SSLv1 in use, self-signed certificates internally, missing the latest third party software patch on a server.
According to the penetration testing firm, you are probably at a low / medium risk level. The tacit implication is that as long as you fix those issues, you should be good to go.
The problem is that first, a vulnerability scanner is an imperfect piece of software and does not test anything a real attacker would. A real attacker might try phishing, or guess "Password1" on a user account. Maybe the attacker would attempt a man in the middle attack or set up an evil hotspot. Once you have AD credentials, now you can find which users have local administrator access, which then you can see if there is a shared Administrator password across all workstations.
The other problem is that the first penetration test does nothing to address potentially systemic issues for why the security vulnerabilities occurred in the first place. The patches could have been missing because there is no formalized patch management program, or inaccurate change management, or an issue with their Puppet config.
Currently there's no way to separate the "good" (read: thorough) from the bad other than direct referral or looking at a sample report.