• Dana Epp

Why security research in the MSP space is abysmal



So a few weeks ago Huntress announced that they were donating $100,000 to the Dutch Institute of Vulnerability Disclosure (DIVD) Bug Bounty Program to elevate SMB cybersecurity. Well, $50,000 is being donated to the bug bounty program. The other $50,000 is an investment in DIVD to continue its growth and effort.


That is still a worthwhile endeavor. And something to be applauded. If you don’t know much about DIVD and the work Victor and his team are doing, it’s good to learn about them.

If you don’t know Victor, he is a member of the Guild of Grumpy Old Hackers. Episode 87 of Darknet Diaries does a great job introducing the hacking group, and some of their exploits like hacking Donald Trump’s Twitter account. And episode 88 goes further into understanding what Victor is all about.


He’s a self-proclaimed janitor of the Internet…. making a life out of coordinated vulnerability disclosure by cleaning up after vendors. My kind of people.


This article isn’t about Victor though. Nor is it really about Huntress and the great work they are doing in the industry. No, it has to do with the fact that the whole idea that you can throw some money at a generic bug bounty program and miraculously attract security researchers to help MSP vendors to find and fix their vulnerabilities is flawed.


Absolute security is a myth. With enough money or motive, anything can be breached.
And all software has bugs. No matter what your vendors are telling you.

But to fix a systemic problem of poor application security (appsec) being built into the systems and tools MSPs (and their customers) rely on, we have to go deeper. DIVD can continue to recruit volunteers to help them look at some of these programs, but you aren’t going to get a mass of serious security research done into these products unless researchers are being PAID to do it.


I digress. I applied to the bug bounty program to see what it was all about the day Huntress announced it and have NEVER been contacted. But that’s OK. Chances are I wouldn’t participate anyway.


Why? Several reasons actually.


The first is because very few of these vendors have a publicly available vulnerability disclosure program (VDP) that has properly documented the scope under which we can look for vulnerabilities, and which clearly describes how the vendor will hold security researchers harmless through safe harbor for ethical security research.


The second is because most of these vendors don’t have a bug bounty program (BBP) that rewards such security research monetarily. Now, this is where DIVD could POSSIBLY come in, acting as a broker for these disclosures. Kind of how Trend Micro invests into the Zero Day Initiative to reward security researchers for privately disclosing vulnerabilities to them, and then turning over the research to the original vendor as a proxy.

ZDI has been around for over 15 years. And it’s given out over $25 million the last I heard. A hugely different playing field.

So why would a security researcher want to take the risk of even touching software written for MSPs when there are far more interesting places to invest our time that pays a whole lot better and has a proper VDP and/or BBP that clearly articulates what we can hack, how we can hack and how we will be compensated?


And while I can’t speak for other security researchers, I know for me I prefer to work directly WITH the vendor than to hide behind a 3rd party proxy agent, or a crowdsourced bug bounty platform like BugCrowd or HackerOne.


But there’s more.


Not only do many of these vendors NOT have a VDP, but there is also no clear process for security triage and incident handling inside their companies anyways.


I should know. I’ve personally reported several zero-days over the years to some of these vendors when we were building integrations to the more popular RMM and PSA products. While I will admit it was a while ago, it was difficult to communicate with these vendors.


Support had no clue how to escalate these reports to the right people, and far fewer could even accept encrypted messages and proof-of-concept (POC) exploits demonstrating business impact. In a couple of cases, the vulnerabilities stayed in the product for some time after the report (in one case over 2 years), and it felt like pulling teeth to get them to update us on their progress to fix the issues. I’m not talking about some self-XSS crap here. I’m talking RCE to SYSTEM level access on shared systems.

Many of these vendors look at security researchers as the enemy, not as an asset. And that is too bad.

What’s worse is how technical debt is handled in many of these organizations. If you look at some of the publicly reported security breaches inside the MSP space in the last few years you might have noticed a trend. Legacy components in systems that haven’t been properly improved over time.


As technology ages, it can become brittle. Time has to be given to the product groups to be able to refactor and improve code as things change.


And that’s not always happening.


I have spoken to product managers and engineering leaders inside several different vendors who struggle with this regularly. They are compensated for growth and expansion. That means attracting new customers and selling MORE stuff to existing customers.


Or in other words… NEW features. NEW products. Not always fixing the old technical debt that piles up.


And that makes this all worse. It puts us all at risk. There is a fine line between balancing the protections of profits over the protection of people through products. And unfortunately, we continue to see how this hurts us all.


But it's not all doom and gloom.


It’s unfair of me to be critical of the great efforts Kyle and his team at Huntress have made, and how he has attracted several other vendors to invest in the efforts of this idea of a bug bounty program for the MSP space. With another $75,000 being donated by a handful of vendors… I want to see their investment come to bear fruit.


And here’s how.


How to improve application security (appsec) in the MSP space

This is a checklist of things vendors in the MSP space can do to improve their appsec and engage with security researchers to improve the security of their products. While it’s high level, it is a roadmap to get you moving in the right direction.


Doing a lot of this already? Awesome! Celebrate it and publish it publicly so we can see it, and learn from it.


For everyone else, here goes…


  • Admit there is a problem. And accept it at all levels within the company. There HAS to be executive buy-in that security is a process, NOT a product. You can’t buy your way out of this. You need to invest in your people and processes to ensure things don’t get much worse. Start today.

  • Get a clear understanding of your security technical debt. Engineering leaders need to get with their product managers and need to ensure this is being properly tracked in their defect tracking system(s) and is reviewed regularly. This data has to be quantified and qualified and reported to the executive team so they know the exposure the company is under and allows them to make proper decisions to invest in fixing what’s most critical, and consider improving software engineering going forward.

  • Invest in security engineering. Appsec maturity comes from investing in a Security Development Lifecycle (SDL) that supports security assurance and compliance requirements. Software leaders like Microsoft have spent decades improving on this, and you can learn from their efforts. Here is a great resource to consider.

  • Establish clear processes for testing your appsec. And be willing to share the results, at least internally. Get an idea of how your products stand up to using a commercially-workable open standard like the OWASP Application Security Verification Standard. At least to Level 1. If this is new to you, I’d recommend you check out the talk I did for OWASP last year on Verifying appsec in the modern world.

  • Create a clear Vulnerability Disclosure Program (VDP), but don’t publish it (yet). You can leverage tools like disclose.io’s Policy Maker to assist you in creating your first one if you don’t have one yet. Now circulate this to all stakeholders within the company and ensure everyone understands its intent and is onboard. A few things to consider:

  1. Are your commitments to timely interaction with security researchers reasonable? If your support team is underwater for a normal caseload, what’s going to happen if you get a flood of security bugs coming in from the outside world?

  2. How about in your coordinated disclosure timeline? If you commit to fixing security vulnerabilities within 90 days, is management at all levels of the company on board to invest the resources to do that when needed?

  3. Does everyone who is in a critical path for the official channels of communication know what’s going on and empowered to engage with security researchers? If not, who is? Is this documented clearly?

  4. Are you properly protecting your PGP keys (if you have any) and ensuring staff are trained on how to use them when communicating with security researchers?

  • Test your VDP before publishing it. Ensure the process works. From the initial report into triage, all the way through to release management. Review your experience with all stakeholders, iron out the kinks, and then prepare to launch it to the world. This is a great tabletop exercise to run on a semi-regular basis.

  • Publish your VDP publicly. Don’t hide this deep in your website on some hidden legal page. I recommend you check out https://securitytxt.org/ and follow RFC8615, defining your security policies in a security.txt file that points to your VDP, typically stored at https://yourdomain.com/.well-known/security.txt. I also recommend that you follow the DNS Security TXT standard and publish your security contact details in proper DNS records, as well as linking to your VDP. You can learn more at https://dnssecuritytxt.org/.

  • Treat security researchers with respect. Not all hackers are criminals. If we reach out to you, assume our intentions are good. Yes, you are going to have some “interesting” experiences working with some of us. Initially, security triage might be bumpy as you get your groove on in how to validate reports. You are going to get researchers who read BurpSuite output as gospel and are going to report stupid non-impactful issues. But sometimes…. those same reporters might find footholds to actual issues that could have catastrophic consequences if chained together and weaponized. Treat all reports as opportunities to learn how hackers are attacking your products, and what they are seeing. Track security researchers' behaviors with your team and ensure there is a responsive, and respectable, two-way communication stream.

  • Engage with security researchers that are helping. If a security researcher asks for reasonable stuff (like access to your software), consider the request. We aren’t trying to get free stuff from you. Sometimes you will have deeper app logic bugs that can only be exploited with the right license (s) or access. If we are engaged with you, let us help you. Within reason of course.

  • Consider remunerating security researchers reasonably. I’ll leave you to decide what is considered reasonable. We know you aren’t Microsoft or Apple and can’t give us 6 figures for a critical vuln. But know this…. If you want to attract security researchers to invest time in your products consider how we are paid. I am routinely paid 4 and 5 figures for vulns I report to vendors I work with. Based on criticality, you want to make sure the most impactful bugs are being paid out appropriately. That will attract us to stick around and keep looking. There is real value here for all parties.


Conclusion

I’m excited to see the work Huntress is doing to make this top of mind for vendors building software in the MSP space. I want nothing more than to be able to celebrate their work by seeing software improve.


It’s going to be a hard uphill battle until vendors embrace security engineering internally and embrace security researchers externally. To help with that effort, I have established a public GitHub repository where I am tracking vendors in the MSP space who have Vulnerability Disclosure Programs and Bug Bounty Programs. I am including links to their programs, providing security researchers with a one-stop location to find those vendors who are willing to engage with security researchers.


Vendors, MSPs, or security researchers who wish to have a company included in the list can ping me on LinkedIn or by email at dana@vulscan.com with the information to be added, or provide a pull request for me to review.


Let’s all work together to improve appsec in the MSP space!