Spire Security Viewpoint

Response to Schneier on Full Disclosure

As I mentioned in a previous post, the MBTA v. MIT scenario is extremely distasteful to me. I do believe the MIT students have a "right" to disclose the information they had. I also believe they increase risk in the process.

[If you are fully versed in the full disclosure discussion and already have a firm position, no need to read further.]

Bruce Schneier has another Wired.com column out about disclosure. Here are my comments:

1. Even though it may "feel right" for security professionals to disclose as much information as possible, it provably increases risk for everyone.
2. There is a much more cost effective,appropriate way to "force" vendors to improve their code - by letting them respond to real incidents. Only 3-10% of vulnerabilities are ever exploited, so we are better off focusing on those that are, and historical information shows that disclosure does not help here.
3. Only a tiny fraction of all the vulnerabilities in the world are ever found, so discovery and disclosure needs to ramp up by orders of magnitude  or else it is simply security theater - there are too many other vulnerabilities for bad guys to exploit.
4. Since only a tiny fraction of vulns are ever found, and there are so many available, there is no reason to believe that the good guys will find the same vulnerabilities as the bad guys unless you believe in collusion between the two. Statistically,random collisions are highly unlikely and we'd need to find substantially all of the same ones that the bad guys do in order to be effective - close to impossible.
5. Every risk manager knows that the attackers' costs are a key component to the probability of compromise. Secrecy, though not perfect, keeps the cost of attack at its highest point; the threat increases with disclosure. The vulnerability level stays the same whether disclosed or not.
6. In order to encourage vendors to "build security properly rather than relying on shoddy design" one must define the level that is acceptable. If any single vulnerability is intolerable, then all software is shoddy and insecure and we should get rid of everything. If not, then we must answer the question of how many vulnerabilities are acceptable for some given software program.

As far as I can tell, there isn't any new ground here. I would certainly welcome any new arguments against the points I've made above. I know this all gets a bit repetitive, but I think it is important to highlight the contra-case every time someone brings up the topic.

Almost 85% of your patching is for naught?

I was skimming IBM/ISS X-force mid-year report and happened across the statistics that 89.7% of vulnerabilities found by research organizations and 82.5% of vulnerabilities found by indepedent researchers have no associated exploit code.

My math shows a real rate of around 85% The information provided is a bit convoluted, but I think it goes like this:

• 3534 vulnerabilities found
• 16% of vulns anonymously disclosed = 566 vulns unaccounted for and 2968 left
• 70% of the 84% of vulns left over were found by independents = 2077 vulns
• 30% of the 84% were found by research orgs = 891 vulns
• 89.7% of 891 vulns is 800 and 82.5% of 2077 vulns is about 1714
• so about 2500 of the 2968 vulns found didn't have an exploit for about 85% in reality

I can't really get to the bottom of the 16% associated with anonymous disclosure. It seems like the exploit numbers would be available, so maybe they were included even though the text reads (to me) like they aren't included.

MBTA vs. MIT

It is really difficult to hold the positions that I do in the face of specific examples like this recent MBTA vs. MIT bugfinder fiasco.The fundamental security design flaws are outrageous and I am not sure who to be madder at - the MBTA or the card vendor.

I get how maddening that is, and it is even more maddening for me because I have to defend the MBTA's right to protect itself against the exposure of this insecure junk, and that includes this silliness around exposing even more information during the protection phase.

So, I don't want to do it, but it seems clear that the MIT presentation was going to increase the MBTA's risk (keep in mind that this is slightly different than a general computing bug that affects many end-users) by making attacks easier/cheaper and putting the spotlight on this system. And of course the court disclosure information increased the risk even more, though a successful case against the researchers will contribute to the growing costs associated with bugfinding, resulting in a likely long-term decrease in risk.

I don't really know what the legal implications of any of this are, but I suspect in the long run things won't work out for the MBTA.

Microsoft Exploitability Index

It is truly disappointing to see Microsoft go its own way with an "exploitability index" that is so coarse in its own scale and so similar to other scales (e.g. CVSS and even MS' own ratings) that it simply adds noise to the overall vulnerability rating arena.

Dan Kaminsky wants the world's gratitude - should we give it to him?

I have a confession to make - I think many bugfinders out there are finding some really cool things, and I am glad they are doing this for us and not for the bad guys. It is easy to see how jazzed someone can be when they find a bug, and how quickly they can be consumed by it. But the "cool" factor really shouldn't be enough when dealing with OPR (other people's risk).

Unfortunately, it appears that these same bugfinders are very poor at risk assessment. I think this is the case with Dan's DNS Debacle. He clearly means no harm, and he actually thinks we should be happy that he did what he did in the way he did it. I think it is a pretty interesting exploit, and I can see that he tried hard to do the right thing, but it is also clear that the impact to the Internet is almost certainly net negative -- i.e. risk is increased.

This is a fundamentally simple point: in order for risk to be reduced, you have to believe there were more exploits across the Internet that used this attack technique prior to July 8th than there will be since July 8th and into the future. That is:

- More incidents prior to July 8th --> Risk is reduced.
- More incidents after July 8th --> Risk is increased.

This is the litmus test, adn there is no doubt in my mind that risk is increased. Can there be any doubt in yours? If so, I would love to hear how threats and vulnerabilities can be manipulated to come up with risk reduction.

Bugfinding is a strange world, because it can often feel like the world is better off -- to make matters worse, there are plenty of people who really think they ARE better off. But this is one of those cases where economists can identify irrational behavior and label it "human" due to the common biases that people have.

Update: Dan responds in the comments below (italicized here) and I clarify:

Well, here's the draft that would have led to this bug breaking publicly, right about now.

http://tools.ietf.org/html/draft-ietf-dnsext-forgery-resilience-05

(Don't be too surprised by the timing; Amit Klein's TXID randomness attacks were gated by TTL opportunities. This is a necessary consequence of that talk.)

Dan - First, let me say that I think you are a clever guy and this is a clever attack technique that you've invented.

I don't really get why you think this would have precipitated your attack technique. We've known about this problem for years. What I am trying to figure out is how many people could have figured it out prior to July 8th and how many people can execute it now. (If you are suggesting that lots of bad guys read the RFCs, I suppose I'll have to disagree).

Elapsed time is everything with this bug - sometimes you seem to suggest that this would have been hard to figure out - since you have been doing DNS for a long time - and sometimes you make it seem like it was easy.

I think we can agree that this bug would have been far riskier just going public one day, than this staged disclosure. In absolute terms, there would be more incidents from the inevitable rebuttal to Forgery Resilience than from a simultaneous patch event.

I do agree that it would be far riskier just going public one day and that, should someone decide to go public, you did it in a way that would keep the risk at its lowest level of increase. It is not clear to me, however, why you think someone else would have invented this attack technique, and why they would be bad guys.

Beyond that, I suspect that the long term damage to the Net of this bug staying in the hands of a few bad guys (it's too simple not to have) would have been -- has been -- silent but quite deadly.

This is where things get tricky. Just when I think you are agreeing that you've increased risk, you say something like this. It calls to question the capability of all the folks assigned to manage, monitor, and secure DNS. You see, I think that if this technique was in widespread use, it would have been identified. Statistically, given all the vulnerabilities and attack techniques that have been and will be discovered and invented, it is highly unlikely that someone would have come across this one.

In any case, I believe your disclosure (and the inevitable exploit code) has made it far, far easier and obvious to folks who would never have known any better, and they will certainly be exploiting it.

I guess it comes down to -- suppose you knew of a flaw that could really break things. Would you just leave it there, waiting to hurt people? Or would you try to do something about it?

I would do something about it. But I wouldn't be happy about it, and I wouldn't over-publicize it and intimate that everyone should be happy about it. Dan, I like positive press, too, but if you go back and look at what you've done, it was probably the biggest self-initiated ego-trip I've ever seen in bugfinding history.

Not only that, but I think it is arrogant to try to "order" people around with "Just.Patch.Now" kinds of assertions, when you have no clue what they have going on at their jobs and how this announcement fits into it. And when your supporters start getting self-righteous and calling people stupid for not patching, I find it even more offensive.

You found an arbitrary attack at an arbitrary time. There have been lots of these in the past and there will be many more to come. You say things like you can take down the "entire Web" (feel free to describe what you mean by that one, and why all those bad guys hadn't done it already if they are likely to have known about this).

I'm trying to do something about it. I'm not asking for anyone's gratitude. Actually, the security community is pretty pissed at me right now -- broke their rules, knowingly.

Yes, your entire tone makes it clear that you think you've done a great thing and that everyone else should think that as well. You told me in my last blog post that I should be happy about this.  The "security community"  (i.e. other bugfinders) are pissed off for a different reason. But you seem to completely ignore the entire population of Internet users out there... and you've increased their risk.

What's increasingly clear to me is we need good ways to deal with these flaws, and that just leaving them there until they collapse is just as dangerous an idea with internet infrastructure as it is with our roads and bridges.

I disagree - what I think is that we need to be secure without having to rely on bugfinding. There are plenty of opportunities for better representation of legitimate functionality (software safety data sheets) and enhanced monitoring. We will always be behind the game trying to deal with finding every flaw in the world.

I look forward to seeing what everyone has to say, when all this is said and done -- even you. Though I will say, the inflammatory titles do not help your credibility in that matter.

(If I wanted to ignore you, I would.)

No doubt, I am in the peanut gallery. I am pretty sure everything IS said and done, and this is what I had to say.

Mozilla-mania and Security Metrics

It is truly fascinating to see the volume of media and Internet coverage given to Mozilla in its quest for security metrics. Contrast this with, say, NIST's CCSS, a slightly different but related initiative where there was almost no coverage...huh.

In any case, given this level of attention Mozilla may be a good organization to push forth some useful metrics. It is definitely worth making a few comments:

1. First, though this discussion is around "security metrics" it is NOT oriented around "enterprise security metrics." I would rather Mozilla call this "attack surface metrics" but this is a common ambiguity used by many, so we can just move on after making the distinction.
2. The functional value of this metric comes during product selection phase, which is much less likely to be an ongoing exercise for most organizations. That means that these numbers will be more useful in the public "which is more secure" wars that go on between Firefox and IE (or Windows, Linux, Mac for that matter). In fact, for any given software type (e.g. browser) this is unlikely to come up more than once every couple of years for most enterprises.
3. This metric appears to be developing itself to be a better mousetrap than Jeff Jones' vulnerability counts, but is likely to suffer from the same problems. See below for a discussion and some ideas and recommendations.
   

Designing an Attack Surface Metric

The majority of people who have studied the problem of designing an attack surface metric agree that vulnerability counts are imperfect in some important ways. More importantly, many also agree that there are various attack surface measures, most notably the Relative Attack Surface Quotient by the folks at Carnegie Mellon but also possible the work being done by Nachiappan Nagappan, et. al. at Microsoft, that hold more promise but are harder to implement.

This makes it quite distressing that there are more promising options available and yet nobody is willing to go there. It also clearly means that we might say we care a lot even if we don't really care A LOT. Know what I mean? (Although I applaud the effort, the impact is not likely to have much significance in the long run.)

That said, there are still useful metrics that can be derived from information we do have. We know that information includes bug counts and dates of creation, disclosure, and patch availability (and perhaps patch application, but that is a stretch). It would be even more useful if it included complexity metrics about the software itself (lines of code, cyclomatic complexity, arcs and blocs, etc..) and development effort (most importantly, time spent in development). I believe Mozilla will include some part of the former but probably can't include the latter of these.

The Real Challenge w/ Vulnerability Numbers
In order to evaluate an attack surface metric using vulnerability counts, we must make a basic set of assumptions:

1. We can't truly know date of first discovery. That is, we should always assume the bad guys already knew about any vulnerability that good guys only recently discovered.
2. We must isolate out threat and vulnerability - vulnerabilities can be controlled, threats can only be controlled indirectly (through influence, for example).
3. When a vulnerability is patched, the software becomes "more secure," meaning that the vulnerable state of the software is reduced. There is certainly reason to consider an alternative here - that patches make software more complex and therefore more vulnerable - but I think it is reasonable to assert that a patch is much less likely to introduce more vulnerabilities into the software than it was created to address.
4. Over time, software gets more secure and not less so. That is, if all other things were equal, software that has been available for two years should be considered more secure (less vulnerable) than software of the same functionality created two weeks ago. (This is even more debatable, but I believe it is secondary to the discussion at hand.)

You can probably tell from the assumptions above that this is going to get tricky from a metrics perspective.  The conclusions you should draw are:

1. Higher vulnerability counts in shorter time periods make software more secure (again, less vulnerable).
2. Date of disclosure impacts threat, not vulnerability, and should therefore be ignored for an attack surface metric. The appropriate date is the creation date -- that is, the day the software went "live" with the vulnerability intact.
3. A "vulnerability level" curve should be sloping downward and not upward. (Good luck with this one ;-)).

In the past, I made an attempt to address these challenges when I created the Spire Vulnerability Rating (SVR) metric. I gladly contribute the name and the algorithm to the Mozilla security metrics project. (Related post here: http://spiresecurity.typepad.com/spire_security_viewpoint/2007/12/why-we-need-the.html).

Now, go forth, and multiply! (or is that divide? ;-)).

Are Bugfinders Being Gagged by Microsoft?

Short answer - No.

Long answer:

David Litchfield suggests that I think there is something nefarious going on between Microsoft and bugfinders. I wouldn't go that far. Microsoft is too smart for that and it doesn't serve their purpose. In any case, I do think that they genuinely want to reduce the number of bugs in their products (heck, we hear about how great they are at this every quarter).

I need to clarify my perspective on some of the other things David says:

Litchfield: what happened to the flaws that the researchers found but kept quiet?

Lindstrom: I don't think bugfinders contracted by Microsoft are finding new vulns and not telling anyone (or only telling Microsoft). I think they already believe they've inspected the code well enough and are less interested in finding more. And I think they are aware that finding too many bugs after they've done the contracted work would make them look bad.

Litchfield: "If the latter, then why haven't they been found by other good researchers who baulk at the very idea of working for Microsoft and would love to see nothing more than Microsoft being embarrased or by made a name for themselves by getting out an advisory or two or sold them to Verisign or Tippingpoint's ZDI?"

Lindstrom: It is not clear to me how many "other good researchers" are out there focusing on Microsoft products (the  app layer appears to be where it is at these days), nor is it clear to me that anyone cares that much about embarrassing MS anymore (witness this entire thread where some prominent former critics are now advocates). And things change in the world such that there is more money to be made in the undercover world - monetizing vulns appears to be worthwhile these days, so bugs found are more likely to be kept secret.

Litchfield: if the public vuln count was up, even marginally, you could bet that everyone would be screaming from the rooftops that SDL was a failure. Given that most people (even Pete and Ryan) think SDL was a success, why is it so hard to believe the opposite?

Lindstrom: It's hard to believe the opposite because there are a number of variables that could also explain a low number of vulnerabilities.... and a high number of vulnerabilities would have the opposite set of variables that I consider less plausible.

Microsoft's SDL - a second look

[This whole Microsoft Security Development Lifecycle issue is really pretty surreal – if someone had told me five years ago that a bunch of bugfinders would be defending Microsoft while I pointed out inconsistencies with what they were saying, I would have thought they were crazy.]

There seems to be some confusion about my point regarding SDL so I feel compelled to reiterate my message: I am not suggesting that SDL doesn’t work – on the contrary, I think it probably has had a significant effect. I am simply saying that public vulnerability numbers do nothing to prove it.

Let me see if I can explain my logic a little better and then come up with some better ideas for Microsoft if they want to prove SDL works.

As far as I can tell, the overall goal of SDL is to ship a product with fewer bugs. I believe the most significant effort around SDL was to train Microsoft employees, so that a) architects design more secure products; b) developers write better code (fewer bugs); and c) QA testers find more bugs (with the caveat that this last one may be difficult if the first two succeed).

The three assertions above – driving higher quality from each participant – are my interpretation of SDL objectives based on six years of public reports about the MS initiative. Some have suggested that SDL is a process and that simply increasing the amount of resources allocated to the problem, is within the scope of SDL. I believe this is disingenuous.

If all we need are more resources, we don’t need SDL (or anything with a special name to it). If all we need are more external reviews, we don’t need SDL. If the truth of the matter is that programmers are all at the top of their game already, we don’t need SDL. We simply need to work harder and not smarter.

That is not what I perceive to be the spirit of SDL. And if it is true, then it simply means that we’ve been cheated all these years and SDL really was always a marketing exercise. (And a very clever one at that). I choose to think more highly of Microsoft and believe their intentions were to really make their people better at what they do. If you disagree, there is no need to read further.

The major shortcoming of using public vulns as a measure of SDL efficacy is that there are too many other variables involved. Just because SDL’s goal is fewer bugs and there are fewer bugs disclosed in public doesn’t mean that one caused the other. Here are some other candidate reasons that could have contributed to a reduced public bug count in post-production:

1) Microsoft simply increased the amount of resources allocated to development;

2) Microsoft hired/contracted with all the major bugfinders before going GA;

3) External, uncontrolled resources spent less time and effort on Vista than they did on XP (due to unpopularity of Vista, more interest in applayer stuff, or simply happenstance).

4) Vulnerabilities have gone undercover and don’t get publicly disclosed anymore.

Now, if I were an external analyst (oh, wait…;-)), I might be forced to consider public vulnerability data as my only source of information to try to determine causation from SDL. In that case, I would use the vuln numbers and then try to come up with estimates to address the factors described above (at a minimum). But not only does MS have access (presumably) to much better data, but they also don’t see a need whatsoever to control for these other variables.

This is the core problem: Microsoft is much more confident in their hypothesis than the evidence dictates. So, not only is the accuracy of the information in question, but the approach itself uses one data point to make its case. That turns science into religion and forces a second look at the entire process. (I pause again to re-assert that I think SDL is probably working, but the proof laid out is not convincing).

I really don’t expect Microsoft to go through a scientific exercise to demonstrate the value of SDL. But I certainly feel like I am at a religious revival when I hear how poorly everyone else is doing and that others should adopt SDL and how insulting it is for people like me to question their beliefs. It sounds solely like a marketing and PR exercise to me, and nothing else.

And Microsoft can do better than that.

[Next Installment: What Microsoft SHOULD do to demonstrate the value of SDL.]

Microsoft's SDL has Saved the World!!

I can't believe it, folks - Microsoft has saved the world with its Security Development Lifecycle! Yay! You heard it here first, second, third... (hey, this sensationalistic stuff Michael and Robert promote is fun!)

Here's the straight scoop: Michael Howard, the "father of the SDL," is using Jeff Jones vulnerability counts as proof that the SDL works:

So if Windows Vista has more code than Windows XP SP2, why are we seeing a reduction in vulnerabilities? Simple: the Security Development Lifecycle (SDL)! Microsoft decided to change its development practices to enforce greater security discipline.

My gut reaction: I cannot believe Howard is actually going to suggest that the number of vulnerabilities found by external individuals is an indicator of SDL success. I defend Microsoft's SDL to many people and it is patronizing to see a metric completely abused.

Hmm, just for fun let me see if I can figure out some other reason why vulnerability counts might be down....(1 millisecond later)... Eureka! It could be that,

Microsoft has systematically hired and/or contracted with every one of their most vocal critics (and most seasoned bugfinders) to do the work behind the scenes and they don't count those vulns!

Now, it happens that I think this is a very, very smart thing to do. From a marketing perspective and from a threat control perspective. But it says nothing about the SDL because they don't reflect the real numbers. And it pains me even more to see this:

The only way you reduce security vulnerabilities is by focusing on improving code security, design security, reducing attack surface, education, tracking evolving threats, mandatory use of tools, banning known bad functionality, better compilers, better linkers, better libraries, etc. And that is what the SDL is all about and what our team is laser-focused on.

Just think how MS could have revolutionized the way we think about vulnerabilities. And they give us trash*.

Then, Robert Hensing has to go and "second that emotion" by acting insulted that people don't believe SDL works:

One of the most frustrating things for me is when ignorant non-believers <G> claim that the SDL is all just marketing hype / spin / FUD etc. (as so eloquently captured at the beginning of his article <G> and as the title of this post).  It's insulting to me.

I gotta tell you. This stuff is insulting to me, and as a frequent defender of SDL it is even more insulting to be ridiculed the way Microsoft has decided to ridicule those of us who think there are better numbers to be had. I am glad Microsoft has decided these things should be said because it helps divide us even more and makes me realize that I should be more vocal in my concerns.

Could it really be that SDL has done nothing to help MS developers write better code? Could it be that the only thing that makes them "better" is a stronger quality control cycle? There are so many ways they could do a better job proving the efficacy of the SDL that it begs the question why they aren't...

*It's not really that bad, but in context it is truly frustrating to see this stuff.

Asking the right questions about "virtualization security"

There are many vendors coming out of the closet about their virtualization strategy. New technology platforms always drive messaging around security - for example, we saw it with wireless and mobile devices in recent years.

I posted a set of 8 questions on the Burton Group blog that may be useful in tracking just what it means to be providing security in virtual environments. Here is an excerpt:

The only reason any of this matters is to assess the speed for which you need a security solution and the availability of solutions that support that need. There is nothing wrong with having a solution that is virtualization-agnostic, but it is useful to know how unique the solution actually is if it is a differentiator in your evaluation process. In addition, in the near-term security solutions should be able to leverage the characteristics of virtualization for beneficial features such as patching a VM that is not in service or isolating a host agent from the host it is monitoring. That is essentially the VMsafe story.

Read more here. Can you think of any questions I missed?