Another Envelope: Vulnerability Growth Rates

Assumptions:

• [A1] Bureau of Labor Statistics (BLS) says: 435,000 computer programmers in the U.S. (projected to decrease over ten years, interestingly). How many in the world? Let's use 2 million (reasonable?).
• [A2] Various sources suggest 5 - 10 KLOC per developer per year. Let's use 6 KLOC/YR, 500 LOC/Month; 25 LOC/wkday.
• [A3] Previously, we settled on 1 security defect per 10,000 lines of code.
• [A4] IBM X-force says: 6,437 vulnerabilities found in 2007. Let's use 10,000 (conservative for our calculations and to account for suggested aggregation of vulnerabilities). That's 840/month; 40/wkday (that seems high to me - maybe 10k wasn't a good idea).
• [A5] SWAG says 1,000 skilled bugfinders in the world with 5 bugs in their back pocket at any given time.

Calculations:

• [C1] Number of new lines of code created every day -- 2m * 25 = 50 million [A1]*[A2]
• [C2] Number of new vulnerabilities created every day -- 50m / 10k = 5,000 [C1]/[A3]
• [C3] % of new vulnerabilities eventually found -- 40/5000 = .8% (99.2% of vulns remain undisclosed, mostly undiscovered). [A4]/[C2]

etc.

Comments

[ilja]

dude, that must have been some good shit you were smoking when you typed that. 25loc a day ? 2 million programmers in the world ? 1 bug on ever 10kloc (so with those stats you're saying the average dev makes 1 bug per year ??) those number just have to be bogus. they dont even sound remotely realistic.

[Pete]

@Ilja -

I hope you mean they are very conservative. In any case, I would love to know what numbers you think are more reasonable.

[ilja]

Hey Pete,
I don't think I have better numbers, I really don't know the numbers. And I think getting some of these numbers even in the same ballpark as the actual numbers would probably be hard. I guess it's just a gut feeling. but 2 million programmers in the world ? if I'd have to make A guess, India alone could probably account for those 2 million. Then again, I could be way off. The same with those 1 bug in 10000 lines of code. It just doesnt sound right to me, I'd be more inclined to say 1 to 1000 (and I'm not alone here) but again, I could be way off. An average dev doing 25 lines of code a day ? I hope that number is really bogus. it has to be. that means on an average 8 hour workday they'd write 3 lines of code an hour ? I would go for atleast 100 a day, but again, I could be way off. My point is that we're both probably way off and the outcome (using either your or my gueeses) is probably not realistic at all.

[Pete]

@Ilja -

I agree that estimation can be challenging. I also think it is worthwhile to add more precision to lines of reasoning (even if they are not yet accurate). In the absence of doing confidence interval work, I opted for something I thought was conservative. Judging from your thoughts on this, I got what I wanted.

Regarding your points about programmers - don't forget there are a lot of other things programmers do that don't involve coding. We are shooting for the final product over an extended time. So, if a programmer really only codes 4 hours a day and rewrites 25% of his/her code, those numbers can be affected significantly.

I think we can all get better at this if we keep trying (not necessarily for this specific case, but for estimation in security in general). I know the first time you do it, it seems difficult, so I appreciate you not just saying "its impossible" and actually taking a stab at it.

Thanks.