Ted Harrington Feature

So...

If you have a software system that protects valuable data or other assets, you probably want to have it tested for security vulnerabilities. That has perhaps led you to explore types of security assessments, and you’ve probably found that the most commonly referenced one is “penetration testing.”

What many companies don’t realize, however, is that “penetration testing” often isn’t penetration testing at all. Worse yet, they don’t realize that they actually might need something else.

When it comes to testing applications for security vulnerabilities, terms are used incorrectly all the time. If you don’t realize it’s happening, it can have dire consequences. Most people ask for penetration testing but are sold vulnerability scanning instead. However, what most people need is something else entirely: vulnerability assessments. Those are remarkably different things. Each requires different investment of time, effort and money. Each has different goals. Each produces different outcomes.

Penetration Testing

The most commonly referenced type of security testing is “penetration testing.” That has become a catchall term, and, unfortunately, it’s misleading.

Actual penetration testing is a tactical service suitable for robust, hardened, thoroughly tested systems. It’s a timeconstrained effort to measure a single outcome. For example, a penetration test might seek to determine “could an attacker escalate basic user privileges to admin rights?” The result is either yes or no. There is no other outcome.

Penetration testing is excellent when you have a mature, welltested defence, and you want to determine if that defence still stands up to a simulated attack. Think of it like when a carmaker seeks to understand how a vehicle performs in a crash situation, so they crash it into a wall to see what happens.

It’s great for understanding a specific scenario, but it is only suitable for a system that’s already been thoroughly tested.

And it’s only intended to inform what happens in that particular scenario; it’s not intended to be holistic.

Vulnerability Scanning

Unfortunately, the term is often used to refer to something else: vulnerability scanning. Thanks to misleading marketing and confused customers, “vulnerability scanning” has become synonymous with “penetration testing.”

It’s not.

Vulnerability scanning involves running an automated tool that looks for common vulnerabilities that are known to exist. The goal is to quickly and inexpensively find basic issues, including checking for unpatched vulnerabilities. Given that running a scanner is one of the first steps your attacker will take, it’s a good idea for you to do this, too. You want to see what they see. It’s good if you’re going to keep timelines and cost to a minimum (with the understanding that it will also limit the value you get as a result).

It’s like the diagnostic tool that mechanics use when the “check engine” light comes on in your car. The tool scans for known issues, spitting back readable codes. It’s easy, inexpensive, and quick. But it’s certainly not a comprehensive way to evaluate vehicle safety.

Think about that: you ask to simulate a car crash, yet are sold a way to remove the check engine light. Those are pretty different!

Vulnerability Assessments

The frustrating confusion doesn’t end there.

As if asking for one thing but being sold something else wasn’t frustrating enough, there’s this: neither of those delivers the outcome that people are usually after. When it comes to security testing, most people seek a comprehensive understanding of their system’s security vulnerabilities. They want to know what the problems are across the entire sytem and how to fix them, with a way to prioritize what to focus on first. Then they want to be able to prove that the system is more secure. That’s not what either penetration testing or vulnerability scanning delivers. But it’s exactly what vulnerability assessments offer. Vulnerability assessments leverage experienced humans who solve problems manually to address your unique circumstances. In the real world, you’re defending against smart, motivated, problem-solving humans—not just scans. Vulnerability assessments help you defend accordingly. They’re great for both well-hardened systems and those still figuring it out (and everyone in between). Unlike a single crash test or running the diagnostic tool to clear the “check engine” light, vulnerability assessments are like the entire safety engineering department. They consider all of the different safety systems—from seatbelts to airbags—and how they all work together. It’s a holistic view of where the weaknesses are and how to improve them.

Know Your Goal

As you can see, each of these terms means something entirely different, so it’s essential to understand four things. First, there is a difference. Second, terms are commonly misused for each other. Third, they shouldn’t be. Fourth, it’s up to you to make sure you get what you need.

The best way to do this is to start with your goal.

What do you want to achieve with the testing?

If you have a mature, heavily hardened system that’s already been through extensive security testing and you want to know how it stands up to a simulated attack against a specific area, get penetration testing. If you need to find basic, common issues quickly, keep costs to a minimum, and are fine without finding custom exploits, get vulnerability scanning. If you need to find as many security vulnerabilities as possible— including custom exploits—understand their severity, fix them based on priority ranking, get vulnerability assessments. Be clear on your goal when discussing testing with your security company, and you’ll be able to get the outcomes you need, irrespective of which term is being used to refer to the testing.

Ted’s Bio:

Ted Harrington is the #1 best selling author of Hackable: How to Do Application Security Right, and is the Executive Partner at Independent Security Evaluators (ISE), the company of ethical hackers famous for hacking cars, medical devices, and password managers. He’s helped hundreds of companies such as Google, Amazon, Microsoft, and Netflix fix tens of thousands of security vulnerabilities. Ted has been featured in more than one hundred media outlets, including the Wall Street Journal, Financial Times, and Forbes. His team founded and organizes IoT Village, an event whose hacking contest has produced three DEF CON Black Badges.