Microsoft's SDL has Saved the World!!

I can't believe it, folks - Microsoft has saved the world with its Security Development Lifecycle! Yay! You heard it here first, second, third... (hey, this sensationalistic stuff Michael and Robert promote is fun!)

Here's the straight scoop: Michael Howard, the "father of the SDL," is using Jeff Jones vulnerability counts as proof that the SDL works:

So if Windows Vista has more code than Windows XP SP2, why are we seeing a reduction in vulnerabilities? Simple: the Security Development Lifecycle (SDL)! Microsoft decided to change its development practices to enforce greater security discipline.

My gut reaction: I cannot believe Howard is actually going to suggest that the number of vulnerabilities found by external individuals is an indicator of SDL success. I defend Microsoft's SDL to many people and it is patronizing to see a metric completely abused.

Hmm, just for fun let me see if I can figure out some other reason why vulnerability counts might be down....(1 millisecond later)... Eureka! It could be that,

Microsoft has systematically hired and/or contracted with every one of their most vocal critics (and most seasoned bugfinders) to do the work behind the scenes and they don't count those vulns!

Now, it happens that I think this is a very, very smart thing to do. From a marketing perspective and from a threat control perspective. But it says nothing about the SDL because they don't reflect the real numbers. And it pains me even more to see this:

The only way you reduce security vulnerabilities is by focusing on improving code security, design security, reducing attack surface, education, tracking evolving threats, mandatory use of tools, banning known bad functionality, better compilers, better linkers, better libraries, etc. And that is what the SDL is all about and what our team is laser-focused on.

Just think how MS could have revolutionized the way we think about vulnerabilities. And they give us trash*.

Then, Robert Hensing has to go and "second that emotion" by acting insulted that people don't believe SDL works:

One of the most frustrating things for me is when ignorant non-believers <G> claim that the SDL is all just marketing hype / spin / FUD etc. (as so eloquently captured at the beginning of his article <G> and as the title of this post).  It's insulting to me.

I gotta tell you. This stuff is insulting to me, and as a frequent defender of SDL it is even more insulting to be ridiculed the way Microsoft has decided to ridicule those of us who think there are better numbers to be had. I am glad Microsoft has decided these things should be said because it helps divide us even more and makes me realize that I should be more vocal in my concerns.

Could it really be that SDL has done nothing to help MS developers write better code? Could it be that the only thing that makes them "better" is a stronger quality control cycle? There are so many ways they could do a better job proving the efficacy of the SDL that it begs the question why they aren't...

*It's not really that bad, but in context it is truly frustrating to see this stuff.

Comments

[Ryan Naraine]

yuo are my h3ro.

_r

[Andrew Jaquith]

Pete, I agree with you on this 100%.

[Ryan Russell]

I have to say, it appears that you have a bunch of self-conflicting statements there... but I'm biased towards vuln research. So, may I ask some clarifying questions?

You say you're a fan of the SDL. I assume that's not sarcasm.
-What do you think the purpose or benefit of the SDL is? Is it not more secure software?
-If the software is more secure, does that not me a smaller number of vulnerabilities (known or unknown)?
-Do you think the number of public vulnerabilities has a correlation with the absolute number of vulnerabilities?

[Pete]

@Ryan - Yes, I am a fan of SDL; yes, the purpose is more secure software; yes, a smaller number of total vulns is an indicator of more secure software; no, I don't believe there is a correlation between public vulns and total vulns.

Public vuln-finding is an ugly contest and MS isn't winning this anymore because they've bribed the judges.

[Ryan Russell]

If the total number of vulns if going down, doesn't that mean the number of public vulns has to go down as well? Or do the publishers increase effort to keep the number of things they publish constant?

Asking strictly as an indicator, not yet whether it's a good idea.

[Pete]

@Ryan -

No, the number of public vulns doesn't have to go down as well. I should mention that I am skeptical that there are a lot of silent fixes being applied, but a) public bugfinding is random in its focus of attention and amount of resources applied to the problem; and b) we have no information about the number of vulns that were found during development and QA.

According to ISS, the total number of vulns found overall is going down, and they are attributing it to everything BUT better coding. I am pretty sure that not everyone is trained in Microsoft's SDL, so determining cause and effect is extremely difficult.

[Ryan Russell]

I was asking about strictly pool size; For convenient-number-size sake, if Office 2003 has 1000 vulns, and Office 2007 has 500 (and everything else being equal) then the vuln finder has to do twice as much something to find 50 vulns in 2007 vs. 2003. Or he doesn't work any harder or smarter, and only finds 25.

I will agree that "everything else being equal" is extremely hand-wavy, and/or the numbers might be 100,000 and 50,000, making the difference in effort to find 50 a rounding error.

[Pete]

@Ryan -

My whole point is that the "ifs" and the "hand-waving" you mention could be answered definitively by Microsoft.

I can't really understand where you are going with your argument - I've already said I support the SDL and I think it probably worked, but they are using the wrong numbers to demonstrate the success.

It is not clear to me that there is some sort of linear relationship between effort and number of vulnerabilities - I think attack surface and/or code complexity probably factors in. But I reiterate that all of our assumptions would be unnecessary if MS came out with the real numbers.

[Balbus]

I don't think people are communicating. Suppose SDL resulted in less secure product but "vocal critics" found huge numbers of vulnerabilities that were found, fixed before the release and not reported. Yes, the product may be more secure, but it would be due to more comprehensive testing by "vocal critics" and subsequent fixing rather than SDL. Because we don't know how many vulnerabilities were found post development from all sources in Vista and its predecessors, we can't use found vulnerabilities as an indication of SDL's effectiveness.

[ms access]

Good job, Microsoft!

/sarcasm