Spire Security Viewpoint

Modelling the Security Ecosystem - is exploit availability exceeding patch availability?

One of the papers at the Workshop on the Economics of Information Security (WEIS '09) last month was "Modelling the Security Ecosystem - the Dynamics of (In)Security" by Frei, et.al. The paper does an excellent job of defining the state changes and interval periods throughout the vulnerability and exploit lifecycles. They also create scatterplots for discovery, exploit, and patch intervals relative to disclosure date.

I think it is worth clarifying one aspect of the paper that is (unintentionally) deceptive. In their attempt to compare availability of exploits and patches over time, the authors use different sampling techniques and populations that are not representative of the whole pool of vulnerabilities. This skews the data and ultimately leads to an incorrect conclusion.

On the surface, it makes sense to use vulnerabilities with known exploit dates as the sample population when analyzing exploit timing. And the authors correctly point out that the number of exploits they use is a minimum number. However, when they use the number as a ratio using the smaller "known-exploit" population in the denominator, and not the overall population of vulnerabilities, it is no longer conservative. In fact, because the ratio over time reaches 100% (inherent to the sample technique), it becomes a maximum ratio.

To illustrate: By my estimate, approximately 40% of the vulnerabilities in the time period being analyzed had known exploits. So conservatively, any distribution function should top off at 40% over time. Instead, the authors discuss numbers reaching "94% 30 days after disclosure."

The fatal flaw here is that the authors assume that there is an exploit for every single vulnerability. This is true for the sample they used, but it is unlikely to be true for the total population of vulnerabilities being assessed. More importantly, it is certainly not a minimum number.

The problem is exacerbated when the authors perform a similar analysis on patch data but select a different sample (top ten vendors) in conducting their analysis. I think it is reasonable to assume that these vendors are either more responsive or less responsive than the overall population of vendors, but I have no reason to believe it is representative.

The final issue here is that after conducting similar analyses on exploits and patches using different, non-random, sample populations, the authors then compare the two results: "We found that expoit availability has consistently exceeded patch availability since 2000." Given the weaknesses in methodology, this is a faulty conclusion.

The work is important and useful, but it needs to be redone using appropriate populations if it is going to be accurate.

Exploiting Undercover Vulnerabilities

For a while now, I have been tracking "undercover vulnerabilities" and exploits. These exploits are a subset of zero day (0day) exploits - while zero day attacks are focused on vulnerabilities that don't have patches, the undercover exploit is focused on vulnerabilities that were unknown prior to the exploit occurring "in the wild." That is, we had no prior knowledge of the vulnerability before the attack took place.

I think this is significant to track because it is an indicator of two things:

1. It reminds us that even before vulnerabilities are disclosed they exist in the software, and it is possible that we should be working harder to provide protection.
2. It highlights alternative methods for identifying vulnerabilities other than the discover and disclose cycle that occur between bugfinders and vendors today.

There are a handful of other reasons why I think this tracking is important... but I am stuck wondering if this is useful for the community. It is remarkably difficult (and time consuming) to actually trace the origins of an announcement - it essentially involves taking all reports of "0days" and vendors being aware of attacks in the wild, and playing chicken and egg games to try to come to a conclusion.

Currently, the OSVDB has taken up the charge (thanks, Jericho), but I don't think they can do the full root cause analysis required to really get to a solid determination -My list has 21 undercover vulns since 1988 and theirs has 75 (10 in 2009), though I haven't updated mine since last October (did I mention the time consuming part? ;-)).

While I have not executed an all-out full-court press on vendors, the times I did ask for follow-up to see how the vulnerability was discovered resulted in somewhat ambiguous answers about having "no information" or "disclosure agreements" that prevent any discussion about them.

So, if you think the list is helpful, please let me know. And if you are a conspiracy theorist (I can't help but wonder a bit myself) I would be curious to hear why you think vendors are reluctant to provide this information - personally, I think we should care MORE about these vulns and not less.

Cognitive Dissonance in Security

Two continuous points of cognitive dissonance in security... as I read Brian Krebs' Security Fix post on firefox vs. IE:

1. If finding vulnerabilities makes software more secure, why do we assert that software with the highest vulnerability count is less secure (than, e.g., a competitor)? (It is even more ironic that this has become more questionable an action as "favored" software has fared worse in this category).
2. If disclosure date is the day that software becomes "at risk" why don't we try our hardest to prolong that date?

Conclusion: nobody really knows what the heck they are talking about when it comes to "secure software."

An alternative measure:
The Spire Vulnerability Rating
Why We Need the Spire Vulnerability Rating

The Other Side of Full Disclosure

Danny Quist of Offensive Computing has a guest blog post on Full Disclosure at ZDNet. (Note: this means I didn't bring it up, somebody else did).

Not surprisingly, I have some comments (mostly reiterations) to some of his points, excerpted in italics:

It floors me to think that it is acceptable for vulnerabilities to be left unpatched for a serious amount of time.

I can understand the emotion behind this comment but not the logic. As it is, there are many vulnerabilities that are left unpatched -- all the ones we don't know about yet.

I consider 90 days to be entirely too long to patch a vulnerability.

This sounds like an arbitrary number of days. It would be great to understand Danny's background with global software development to understand how he can conclude that this is "entirely too long."

You can disagree with full disclosure, but it is a useful motivational tool. ... Limited or closed disclosure creates complacency, which amounts to willful neglect.

Again, here are conclusions that are unsupported. I can believe that full disclosure is motivational, but I am not sure this brings about more secure software. In addition, I think it would be much more motivational for software companies to have to deal with exploits in the wild. The final point here is a simple opinion with no basis.

I wish there was some other way than full disclosure to motivate vendors. Unfortunately it is the only method available that has a proven track record of working.

This really is closed-minded thinking. With the process as broken as it is already, we need more people thinking outside of the box. There IS another way to motivate vendors - seek out undercover vulnerabilities being exploited in the wild. After all, these exploits are oft-mentioned and yet also ignored. These are the ones I worry about most.

Google's Native Client Vulnerability Contest

[hat tip: Ryan Naraine]

Google has decided to have a bugfinding contest for its Native Client. After its Borg-like introduction of the "Android Security Team" (no names, collective only), it also introduced the contest.

For vendors, this is probably the best option to take to address public vulnerability disclosure. The idea is to increase the value to researchers in order to focus efforts and presumably find most of the security bugs, rather than the traditional alternative of submitting to random public bugfinding.

I wrote a post back in 2005 that mentioned this option:

Of course, we don't live in that perfect world. We live in a world where random people look for random vulnerabilities in random applications. Randomness is kiling us. In this world, the best thing to do is to create a set period of time and have a contest, real or implied. Get everyone focused on one platform, perhaps by offering a reward (from the manufacturer, not this ridiculous stuff by third parties).

It will be interesting to see if the vulnerability discovery curve for Android is significantly different from other solutions. (It would also be great to determine rediscovery rates for vulnerabilities).

Corrected [2/27/09]: Fixed per Ryan's clarification in the comments.

The Disclosure Race Condition

The security profession has been debating vulnerability disclosure policies for years. The debate has heated up again with the latest Adobe "zero-day" (a true undercover vulnerability, I believe) resulting in specifics being published on Sourcefire's VRT blog, some concerned comments, and a blog post on Metasploit.

The arguments for disclosure first tug at the heartstrings with simplistic platitudes like "it is better to know than not to know" but then grounds itself a bit with the following logic:

1. The bad guys already have this information
2. The good guys need the information to protect themselves

There you have it - a classic fight between good and evil. But even this is incredibly simplistic. It treats two groups as holistic entities and not dynamic populations. And that matters a lot when evaluating risk.

Essentially, folks who support higher levels of quicker disclosure are betting that good guys can and will respond faster and more completely than the bad guys can attack. With discrete groups this may be true, but with dynamic populations I am not so sure.

Risk is a function of threats, vulnerabilities, and consequences. The variance in these elements is constrained by scarce resources on both the attacker side and the defender side.

The attacker makes his decisions based on a cost-benefit analysis that compares costs - skill, effort, and equipment - to the expected benefit discounted by potential penalties (the attacker's risk equation). The higher the result of this equation, the higher the risk to an organization (because threat is higher).

The defender makes a ROSI (return on security investment) assessment (typically ad-hoc) to determine her overall risk. The lower the cost of protection, the more likely that investment is a good one.

Finally, we shouldn't forget opportunity cost which compares these results to anything else the attacker or defender might want to do.

Looking again at the disclosure reasoning, the question is whether releasing more information helps the good guys or the bad guys more. The "bad guys already have this information" argument neglects the acquisition cost of this information and the skill level required to execute.

A basic illustration of cost associated with "effort" - some of you were no doubt a bit annoyed as you read my first paragraph above and wanted to see the source material in question - it didn't have links to the pertinent Sourcefire and Metasploit blogs. Of course, you "already had" this information in the form of your ability to use search engines to find it. Links are a part of the Web culture because we recognize that time is money and making things a bit easier for the reader lowers his/her costs.

So the short point is that distribution of information, and its corresponding ease of access, matters.

The "good guys need this information for protection" is perhaps a trickier. The huge majority of Internet users do not need the information provided because they have no capacity to leverage it for protection. (Guys like HD Moore can do wonders with it, of course). The users rely on the makers of products who DO need this information to provide protection.

It is clear from this case that many large security companies already had the information (they already had samples), so the added benefit to the "good guy" community must be adjusted with that information in mind.

In the end, I think it is less likely that good guys used this information for protection than it is that bad guys used it to compromise some user. I believe this is almost always the case, and my evidence is the aggregated number of exploits that occur after disclosure compared with the number of exploits of undercover vulnerabilities

Should Verisign sue Sotirov / Appelbaum?

[I am not a lawyer.]

With many security researchers supporting software liability (or at least some), it is useful to review alternative legal actions that may be taken throughout the discovery and disclosure process. I think the recent RapidSSL hack by Sotirov and Appelbaum is an interesting case.

In case you missed it, they demonstrated a hack where they could essentially spoof a Certificate Authority by taking advantage of known flaws in MD5 and (now exposed) weaknesses in RapidSSL's certificate issuance process (the serial number identification).

This is interesting primarily because it is uncommon. When Adobe and Cisco (and others) attempted legal action against security researchers, the researchers were demonstrating vulnerabilities in software that, if compromised, would impact their customers. This obviously is a concern, but in the end it is very difficult to show damages. That is generally true with most vulnerabilities.

Things change with cases like this (and potentially Software as a Service) where damages are borne by the vendor itself.

This vulnerability comes pretty close to demonstrating business fraud as far as I can tell. I don't know how far they went in their conference session, but if they actually issued a certificate, it seems to me that Verisign /RapidSSL could claim lost revenue (as minimal as it might be at this stage).

I am not sure what to make of their intentional secrecy to protect against legal action. I don't think it was a very smart move to tell people that you purposely didn't tell a vendor because they might have a legal right to keep you from presenting.

In any case, the approach by these folks completely disregards responsible disclosure (as anyone can at this stage) and the whole issue is couched in reasoning that could be applied to any vulnerability discovered -- anyone can withhold information from vendors under the assertion that legal action may be taken.

ISS outs Trend Micro

IBM / ISS released (partially redacted) security advisories against Trend Micro. I think John Pescatore got it right in this article:

"But in some ways, Pescatore said, X-Force broke an unspoken rule. "They definitely compete with each other," he said, referring to IBM's Internet Security Systems and Trend Micro. "Does the blog post warn users of the danger? That's what the vulnerability advisories are for. Would X-Force do the same thing if it found bugs in IBM's WebSphere? If IBM didn't patch fast enough or the patches didn't work too well, would they be blogging that, 'We've had it with IBM'?"

These kinds of competitive rivalries really bring out the worst in security companies and highlight the house of cards that is vulnerability discovery and disclosure. Perhaps more importantly, you'd think ISS would act differently given its experience with the Witty worm and its somewhat strange circumstances... although they may hold the record for the number of vulnerabilities found in competitor products (hmm, maybe I am confusing cause and effect here).

In any case, I doubt it would pass my litmus test. I really don't understand why the profession facilitates arbitrary target practice. Pescatore cuts to the chase with his IBM point, and I am tempted to challenge for ISS to out IBM sometime soon, except that it would increase risk. In any case, IBM would be a target-rich environment in an arbitrary world.

Is Microsoft's SDL Working?

I have been critical in the past of Microsoft's supporting evidence for its argument that its Security Development Lifecycle was working. Mind you, I generally assume that it is working, but am just not swayed by the data. So imagine my interest when I came across this tidbit on page 28 of its 150-page Security Intelligence Report for the first half of 2008:

"In general, trends for Microsoft vulnerability disclosures have mirrored those for the industry as
a whole, though on a much smaller scale."

So, if Microsoft is trending consistent with everyone else, then it is more difficult to see the benefit of SDL... This is one of the problems with using public disclosure data - it is inherently fickle and can't tell you nearly as much as, say, internal QA data.

I assume there is a different explanation since I haven't waded through the entire report yet. In any case, it seemed worth mentioning.

On Vulnerability Rediscovery

One of my biggest objections to vulnerability discovery and disclosure is simply that there is no reason to believe that we will find the same vulnerabilities that the bad guys find. This is a somewhat unique property of the set that contains the entire world's codebase (given that independent individuals can pick and choose whatever target software they want to review). A QA department doesn't have the problem because the much smaller codebase can actually be assessed.

If I were a statistician, I would calculate the (random) likelihood as a variation of the birthday problem that demonstrates how a room with 23 individuals has about a 50% chance of having 2 individuals with the same birthday. Except this problem involves one or more individuals with one or more vulnerabilities out of a set of all existing vulnerabilities.

In a recent comment on my full disclosure post, Michael Bennett writes:

Vulnerabilities are not discrete entities, they almost always fall into a class of vulnerability that is known and some of these classes are easier to test for or exploit. It is likely that two people will come up with the same or substantially similar vulnerabilities because easiest will be tried first.

In addition, pjhenry1216 had this to say in comments on Schneier's recent Wired.com article on full disclosure:

There's a statistically greater than random chance that the good guys and bad guys will uncover the same vulnerabilities. Researchers don't find random vulnerabilities. It's like electricity in a circuit: path of least resistance. Researchers look for the weak spots and go from there. This is true for both the good guys and bad guys.

To which I replied:

The "greater than random" comment is a reasonable assertion (any thoughts on how to model that? sort of like the "hot hand" in sports, I think) but I am not convinced it is true in the aggregate - since this is public bugfinding, the world's codebase is the target and I suspect it would still be at least close to random. In any case, the follow-on question is, how high would the overlap need to be in order to be useful? (There is a study by Ozment on rediscovery that found about 8% rediscovery rate).

The study I was referring to above is the only study that I am aware of that directly attempted to measure vulnerability rediscovery rates, by Andy Ozment: The Likelihood of Vulnerability Rediscovery and the Social Utility of Vulnerability Hunting. This study looked at rediscovery rates for Microsoft vulnerabilities based on credits in the notification bulletins. The study determined that 7-8% of Microsoft vulnerabilities are rediscovered.

It seems reasonable to suggest that Microsoft is probably a much larger target than other software companies when it comes to bugfinding (though I think this is probably reduced significantly from previous years), and it makes sense that the easiest vulnerabilities are the most likely to be rediscovered.

I guess the real question is whether an 8% success rate is enough.

More here: The Long-Term Impact of Vulnerability Research: Public Welfare