How to Choose a Penetration Testing Company When You Can't Tell Good From Bad

By Paul Petefish. 20 years in offensive security, 1,000+ pentests scoped and delivered.

---

Short answer: the hardest part of buying a pentest is that you usually can't judge the work. Two firms can run the same scope, hand you reports that look almost the same, and deliver completely different value. One actually tried to break in. The other ran a scanner with a junior resource and cleaned up the output. The report won't tell you which one you got. So this guide covers why that gap exists, the five tells that give away a weak vendor before you sign, and the questions that pull the truth out of a sales call.

---

Two firms, same job, opposite results

A security leader at a big US bank posted something recently that I have been telling clients for years. He had two firms working the same engagement. Firm A went straight to an assumed-breach scenario, skipped the real phishing and social engineering, used off-the-shelf tools that got caught the second they touched the network, and ran their testers across several jobs at once. Firm B on the same account spent three weeks building a phishing pretext, brought custom tooling, and walked through the environment without ever being seen.

Same money. Same statement of work. Night and day.

The thread blew up because everyone in the comments had lived it. And the most upvoted explanation was not about talent. It was about what buyers reward. As one person put it, a lot of companies do not actually want a great red team. They want a clean report they can hand their auditor. When that is what you pay for, that is what you get.

Here is what should bug you about it. Anyone can put "penetration testing" on a website and start selling. That leaves you grading a highly technical service you were never trained to grade. But there is good news. You can bring standards to the scoping table yourself, even though the market never built one for you.

Why the big names are the safe pick, not always the good one

The mediocre-report problem clusters around the large, recognizable firms. It is not that they are short on smart people. They have plenty.

It is the billing model. Big shops cap the hours a tester can spend so tight that going deep is not on the table. A tester who wants to build a real pretext, write custom tooling, or chain three findings into an actual break-in cannot, because the clock runs out at a scan and a templated report. A boutique with an owner-operator will blow the budget on purpose, because their next ten deals ride on this one being good. Different math, different product.

So why do buyers keep signing the big names? Because nobody gets fired for it. If you are the one putting your name on a six-figure invoice, it is a lot easier to write the logo everyone knows than to explain to your CEO why a firm they have never heard of is the better tester. The safe career move and the good security move are not the same move. Most of the industry quietly runs on that gap.

That is the whole problem in one sentence. You cannot audit tester quality on your own, and the brand name tells you the firm is safe, not that it is good. So you need another way to separate real testing from theater.

The 5 tells that give away a weak vendor before you sign

You cannot watch the test happen. But you can read the signs in the sales process, the proposal, and the sample report. After scoping more than a thousand of these, here are the five that reliably point to a disappointing engagement.

1. They never ask why you are testing. The first thing a good vendor does is ask what you are trying to get out of this. Is there a compliance deadline, a PCI or SOC 2 or HIPAA box that has to be checked? Did the board or a big customer ask for proof? Or do you genuinely want to know where you are exposed? Each one potentially calls for a different scope and a different approach. A vendor who jumps straight to IP address counts and a price is selling a commodity, not solving your problem. You will get a technically valid test that answers a question you never asked, and you will not realize it until you are standing in front of the person who wanted it.

2. They will not tell you who is actually testing you. Ask the real staffing questions. W2 employees or subcontractors? Onshore or offshore? Human testers, or fully automated and AI-driven? Get names and certifications. This is one of the most common complaints buyers have: you meet the A-team in the sales call and get a rotating cast of contractors on the real work. If they dance around it, that is your answer.

3. The sample report is a scanner dump in a nice cover. Ask for a redacted sample before you buy. If the findings read like automated output, generic severity labels, no story of how the tester moved through the environment, no chained exploits, no sign anybody verified anything by hand, you are paying pentest money for a vulnerability scan. A real report reads like a story. Here is how I got in. Here is how I moved. Here is what I could access. When we grade a vendor's reporting, that is the bar: attack-path narratives that show how findings connect, remediation specific enough for a developer to actually use instead of a generic CWE link, and a log of what was really tested. Weak vendors fail this quietly, and you only find out when the report lands.

4. They lead with the methodology name instead of your risk. A vendor who opens with "we run assumed breach" or a framework acronym before they understand your environment is selling a process, not a result. The approach should come out of what you are trying to protect and what a real attacker would do to you. When the method is decided before the questions are asked, you are getting the assembly-line version.

5. They claim to be experts at everything. Ask most firms what they specialize in and the answer is all of it. External, internal, web app, API, cloud, mobile, OT, red team, and now AI. They are not lying, exactly. They have smart people who can pick up a published methodology and figure it out as they go. But following a methodology is not the same as expertise. A real application tester usually came up as a developer and knows how software actually breaks, not just what the checklist says. A real OT tester has spent time around industrial control systems and knows why you never just point a scanner at a PLC. Cloud, same story. The generalist who dabbles in all of it rarely goes deep in any of it, and depth is what you are paying for. So do not ask whether a firm can do the test you need. Ask how often they actually do it, and who on the team specializes in it. And do not stop digging until you get a straight answer, because this is the question weak vendors work hardest to talk their way around.

The questions that pull the truth out

Take these into your next vendor call. The good ones light up. The weak ones get vague. And before you even get there, notice one thing: did they ask about your goals first, or did they jump to a quote?

• Who exactly will test us, and are they employees or contractors?
• Can I see a redacted sample report from a job like ours?
• Walk me through how you would approach our environment before you have scanned a thing.
• What did you find on your last engagement that a scanner would have missed?
• How often do you run this exact type of test, and who on your team specializes in it?
• For application testing: do your testers have a software development background?
• How do you validate findings before the report goes out?

So how do you actually fix this?

You fix it two ways. You rotate vendors, and you hold them to a real standard, because the market will not hand you one.

Rotation matters because the same firm running the same methodology every year tests the same things every year, which means it misses the same things every year. A fresh set of eyes with a different approach finds what your last vendor kept walking past. If a firm's blind spots are baked in, and they are, then rotating is not disloyalty. It is how you finally see the whole picture.

The standard matters because "trust the brand" is not a plan. Remember our Firm A versus Firm B story? That is the gap Lion Security was built to close. We run every vendor in our network against more than 100 criteria across eight areas: core specializations, tester qualifications, testing scope, methodology and standards, platform and technology, reporting, engagement operations, and commercial terms. Vendors do not just check a box. They get scored on a maturity scale, developing to maturing to established, backed by how much of that work they have actually done in the last twelve months. And where CREST or CHECK accreditation exists, we validate and grade it too, at the individual and the company level.

You tell us your scope, and we match you with vetted vendors who have already cleared the bar, with competitive proposals, at no cost to you.

You should not have to become an offensive security expert just to buy it well. That is our job. Fighting alongside you is the whole point.

---

Frequently asked questions

How do I know if my pentest was actually a vulnerability scan? Read the report. A real penetration test walks through how the tester moved through your environment, verifies findings by hand, and chains vulnerabilities together. If the report is a list of automated findings with generic severity scores and no story, you likely paid pentest prices for a vulnerability scan.

Are big-name penetration testing firms better? Not necessarily. Larger firms often cap billable hours so tight that testers cannot go past a scan and a template, while a boutique owner-operator usually puts in more because their reputation rides on it. A recognizable name tells you the firm is a safe choice, not that it is a good tester.

How can I tell if a pentest firm is a real expert in the testing I need? Most firms claim every specialization because they have capable people who can follow a methodology. Following a methodology is not deep expertise. Ask how often they actually perform your specific type of test, not whether they can, and who on the team specializes in it. For application testing, ask whether the testers have a software development background. Frequency and specialization predict quality far better than a capabilities list.

Should I use the same pentest vendor every year? Rotating vendors is a best practice. The same firm using the same methodology tends to find the same class of issues and miss the same blind spots year after year. A different vendor with a different approach surfaces what your last one missed.

Is there a certification standard for US penetration testing firms? There is no mandatory US accreditation for penetration testing. Anyone can advertise the service, which is why buyers should hold vendors to a consistent set of criteria instead of trusting a brand or a website claim. Lion Security applies its own evaluation, more than 100 criteria across eight areas with defined maturity levels, and validates CREST or CHECK accreditation where a firm holds it.

What should I ask a penetration testing vendor before hiring them? Ask who is actually testing you and whether they are employees or contractors, for a redacted sample report, how they would approach your environment before scanning, what they recently found that a scanner would miss, how often they run your type of test, and how they validate findings before delivery. Vague answers to these are the clearest sign of a weak vendor.