Spire Security Viewpoint

Setting the Record Straight on ROI in security

Trust me, I have no interest in bringing this up, but since someone else did, I feel compelled to respond to set the record straight. I had been casually following the Securosis work on business justification of data security and saw their paper pop up in the SANS reading room. As I was perusing the table of contents, I came across the item "Why there is no ROI in Data Security."

Trust me, I have been there and done that with the whole ROI in security thing. And I honestly thought I would read the section that said you can't get ROI because security doesn't generate revenue (I don't think that matters), along with the "and another thing" that ROI isn't even a very good model to use anyway.

Instead, I found this (my comments in between italicized excerpts):

Return On Investment (ROI): Repeat after me: There is no ROI for security spending. Anyone who tells you otherwise is wrong. Here’s why: When applying ROI to data security, you attempt to quantify loss, and then substitute loss as revenue.

No, you don't attempt to quantify loss, you quantify your costs and hold the revenue constant. That means ROI comes from reduced costs. Typically, you don't factor in losses, though that is a possibility for the high-frequency recurring events.

Besides always producing a negative result, this model is a fundamentally flawed way to approach security spending for a couple of reasons.

The "always producing a negative result" doesn't really make any sense when you apply the model to two different alternatives (your current state and the modified state).

The first is that security precautions do not create a return or generate revenue, so by definition they cannot be used to calculate revenue.

This is the standard 'no-revenue' objection which I disagree with but understand. The slip in the final part of the sentence perhaps explains the faulty reasoning - you aren't calculating revenue, you are calculating a return (based on the net between revenue - held constant - and costs).

 The equation is abused by substituting potential losses that cannot be reasonably quantified for ‘Return’, in place of quantifiable financial gains.

This is incorrect.

Further, expenses such as disaster recovery, legal costs, and regulatory costs can be estimated with a fair degree of accuracy; but indirect costs such as “loss of reputation”, brand impairment, and loss of future business cannot be accurately be [sic] assigned dollar amounts.

ROI generally comes from increased productivity - the gain that comes from substituting computing for human labor over a period of time. This, by the way, is the goal of essentially all IT. The items discussed above are again associated with losses and though they can be quantified as well, that is where ROSI comes in, not ROI. 

ROI is a well understood and commonly used financial equation, but it does not account for many of the relevant variables effecting [sic] revenue, or the non-linear costs for multi-incident breaches. The calculation is fine for controlled academic problems, but in context of losses due to data breach, it is a case of garbage in, garbage out.

I certainly agree with garbage in, garbage out.

Further reading:

Can you get ROI from reduced costs?

Ten Points about Security ROI and ROSI

What is threatening about ROI in security?

ROI is about value

Return on Security Investment (and a little ROI)

Breach Costs "per record"

It makes a lot of sense to normalize data coming in from many different sources so that it the individual values can be compared more easily to each other. This is done at a ratio level of measurement (the other levels are nominal, ordinal, and interval, in that order with ratio as the highest level.

However, using this scale assumes there is a reason to -- that the number of things being counted (in this case costs) correlates well with the normalizer (denominator).

It is common when estimating costs of data breaches to quote costs "per record". Most recently, Ponemon Institute released a study that asserted a cost of $202 per record for data breaches. But here's the problem -- the bulk of the costs of breaches are not variable costs (or at least it isn't clear to me that they are). These costs appear to be fixed costs.

In the Ponemon example, the variable cost - notification -  constitutes $15 of that $202, or about 7.5%. This doesn't strike me as significant enough to keep using per record ratios as a measurement.

Why does this matter? Because, to make this information useful, people may want to plan around it. And counting up the total number of records you have, then multiplying by $202 to figure out potential losses (from there, they would need to discount based on probability of loss, but that's another post).

It seems unlikely to me that when you have a million records versus ten million records that you will only lose 1/10th as much in, say, lost business.

All the studies use per record and I think this is misleading. Better to use numbers per enterprise, as a percent of revenue, or perhaps even percent of all costs (I'm brainstorming here).

McAfee: A Trillion Dollars in Losses

The press is awash with stories about McAfee's experts noting that losses are as high as a trillion dollars. Mr. DeWalt, CEO of McAfee, notes as much in this blog post:

Our Unsecured Economies Report shows that the loss of intellectual property, such as trade secrets, to be one $1 trillion or more annually.

There is a problem here. I can't find the word "trillion" anywhere in the actual report, which I took the trouble to download and read. In addition, I don't find anything in the press that asks how these numbers were derived.

Here's what I think. The report shows that the companies surveyed lost a combined $559 million and I assume McAfee is estimating based on these survey results. Assuming the companies are a representative sample of the world, then McAfee is suggesting that the companies surveyed represent .05% of all companies.

A trillion is a lot of money. Let's run a number or two:

• Amrit points out that the U.S. GDP is around $14 trillion.
• In 2007, the GAO said that cybercrime cost the U.S. about $117 billion for about 1% of U.S. GDP.
• The world's GDP is estimated somewhere between about $45 trillion and $75 trillion. 1% of that is $750 billion on the high side, which at this level isn't that far off from the $1 trillion.

It isn't necessarily too difficult to believe... but it sure would be nice to get a cleaner methodology and set of definitions down.

Thinking Strategically about Information Security Metrics

How many patches did you apply last quarter? It is an important question, but it isn't a strategic one. In order to think about risk and security, we need to elevate our discussion from the operational, such as patch levels and coverage, to the strategic.

One way to do this elevation is take a step back and consider where the risk comes from. It isn't uncommon to get caught up in our ACLs and tokens and signatures and simply work within the domain of security controls. So we associated our risk with the failure of controls. But that is only part of the story, and in a lot ways not even the most important part. Even regulations tend to focus more on control weaknesses than they do the underlying assets. And so we lose focus.

If we are thinking strategically, we must consider risk from the business technology perspective (you could even think more broadly if you like, but I find that a bit ambiguous from a metrics perspective and consider it out-of-scope). Thinking this way, it becomes obvious that the future "unwanted outcomes" that constitute our risk, and which we are trying to prevent, are attacks, exploits, compromises, and losses. Further, it is clear to see that these unwanted outcomes occur within the set of all systems activity and therefore this activity is a key aspect of the risk we are trying to minimize. For our purposes, consider these generic system activities that involve an input source and a target transactions.

We also know that risk has two components - a likelihood component that we can derive from the transactions discussed above, and a value component that identifies the amount of money at risk. We don't specifically need this latter one if we all qualitatively recognize the value of, say, a system compromise. However, understanding and quantifying value is an inherent part of what we do, and is crucial to our decisionmaking process. So the second piece of the model is value.

Now, we can start discussing the security controls in place and how they effect the transactions occurring in the IT environment. These controls are applied to transactions that may be good or bad, and result (simplistically) in a decision to allow or deny a transaction. We end up with four possible outcomes from any control test - good | allow; bad | allow; good | deny; and bad | deny. Controls are the third aspect to the strategic metrics model.

Finally, we have incidents. These are somewhat self-explanatory at the level we are operating on. Incidents are simply the manifestation of the unwanted outcomes. The frequency of these incidents within the context of the full volume of transactions provides historical information that can help estimate the risk associated with the population.

Runaway Numbers

Freedom to Tinker has an excellent post on journalistic skepticism. I agree wholeheartedly that numbers can take on a life of their own -- it concerns me even more because it is a real downside of estimates and measurements, of which I am a huge advocate.

I felt a similar way regarding (the book) Geekonomics' approach to estimating the $180 billion costs of insecure software, and dissected the number to understand the details.

I worry quite a bit that only the end results get reported without showing the work. But I don't really know any way around it, except to be fair and open throughout the process.

Intangible Asset Valuation -- The Art Edition

I was pointed to today's online New York Times, "A Textbook Example of Ranking Artworks." The article highlights the neverending problem of valuation, this time with art. Some excerpts:

"His statistical approach has led to what he says is a radically new interpretation of 20th-century art, one he is certain art historians will hate. It is based in part on how frequently an illustration of a work appears in textbooks."
[...]
"But what most bothers Mr. Galenson is his dismissal by art experts who, he writes in his forthcoming book, “almost unanimously refused to acknowledge the value that quantitative methods could have in their field.”"

This reeks of Bill James and Moneyball, as well as many other examples brought up in Ian Ayres' book SuperCrunchers. People who make a living flexing their qualitative viewpoint always feel like their livelihood is at stake with quantitative methods. In reality, there is a synergy between the two.

No doubt, it isn't difficult to challenge techniques, as does one critic:

Mr. Danto said: “I don’t see the method as anything except circular. The frequency of an illustration doesn’t seem to me to really explain what makes an idea good.

“Somewhere along the line you’ve got to find answers to why it’s so interesting.”

What I like best about methods like these is that they can be more precise - they force analysis that is deeper and usually clearer about valuation.

In the end, it is not clear to me how important this is to folks, given that the market is not that dynamic (as far as I know). It seems more like a bragging rights exercise than anything else.

Contrast that with the fluidity of information asset valuation, and the volume of decisions that must be made for usage and protection... my feeling is that security professionals should want this level of attention even more...

Calculating Risk - darn deep space probes

From The Concise Encyclopedia of Economics, by Henderson (p.447):

"One of the smallest risks that statisticians have measured - dying as a result of cancer caused by the release of plutonium from a deep space probe that loses control during its swing around the Earth to gain velocity and burns up in the atmosphere - measures in at three-millionths of one percent."

Found this email message archive that has some calculations in this regard. In case you were worried about dying this way.

A tale of two quotes - vulnerability disparity

First, from the Associated Press:

"Rouland contends the 2007 number would have been higher if not for the emergence of a black market that will pay up to $100,000 to computer whizzes who find such threats and sell the information to criminal gangs eager to exploit them."

This quote references the X-force report I mentioned in an earlier post - you know, the one where people are really concerned that vulnerabilities are decreasing.

Second, from a private mailing list:

"I'd bet that most skilled researchers have a number of vulns that they know of that are just lying around, where they didn't have the time or inclination to publish. Maybe they *know* an overflow is exploitable, but haven't put in the 20 hours to prove it; maybe they don't want to deal with the hours/days of labor it takes to disclose it responsibly; etc. Since we're on the back of the envelope - conservatively, I'd say that each skilled researcher would have a minimum of 5 issues that are just lying around that are not likely to get published. So, we're talking hundreds or thousands."

Hmmm, well $500k sure isn't chump change to me, and I suspect that would be true of many bugfinders. So, we have (potential) sellers with $500k of unrecognized income available to them, and buyers with a plentiful supply of vulnerabilities paying as if they are scarce.

I think this variance may be evidence of another thing as well - it is common for people to suggest that one reason to find bugs is that it is just as easy as the bad guys to find them. Keeping aside the fact that this argument is bogus to begin with (rediscovery rates are 7% at best), if these two perspectives really are rational, then it illustrates the disparity in the bad guy's ability to find vulnerabilities, and is perhaps a credit to every legitimate bugfinder who keeps his vulns secret.

To heck with gaming the numbers, it is great fun to watch people gaming the analysis (me included? feel free to clue me in where I am wrong...).

Back of the Envelope Math - Undercover Vulnerabilities

Assumptions:

• [A1] Schneier says: "They find, on average, one security flaw per 1,000 lines of code." Update: I can't substantiate the number but if memory serves me, 1/1k is a common rule of thumb for all defects. Also, I think the numbers here are more likely - around 1/10k. I will use this number instead.
• [A2] Tippett says: "Only 3 percent of the vulnerabilities that are discovered are ever exploited."
• [A3] The National Vulnerability Database shows 29,360 vulnerabilities found, ever (I suppose).
• [A4] Spire Security (i.e. me) lists 20 total vulnerabilities discovered via exploit. (undercover exploits)
• [A5] Spire Security (yup, me again) estimates there are 3 trillion lines of code in the world, perhaps 240 billion "active" lines of code.

Calculations:

• [C6] 240,000,000,000 / 10,000 = 24 million vulnerabilities in existence. [A5]/[A1]
• [C7] 30k / 24m = .125% vulns found (that's 99.875% of vulns undisclosed). [A3]/[C6]
• [C8] 30,000 * 3% = 900 vulnerabilities actively exploited (can this be right?) [A3]/[A2]
• [C9] 900:20 = 45 to 1 odds that an exploited vulnerability will come from the pool of known vulns. [C8]/[A4]

etc.

Update: I've had some feedback that suggests [A3] NVD numbers may be off by as much as a factor of 50 and that my list of 20 undercover exploits [A4] is "off by half" which I don't understand but will use 100 to be on the safe side. Here are the calculations in that case ([A3] = 1,500,000; [A4]=100):

 

• [C6] 240,000,000,000 / 10,000 = 24 million vulnerabilities in existence. [A5]/[A1]
• [C7] 1.5m / 24m = 6.25% vulns found (that's 93.75% of vulns undisclosed). [A3]/[C6]
• [C8] 1.5m * 3% = 45,000 vulnerabilities actively exploited [A3]/[A2]
• [C9] 45,000:100 = 450 to 1 odds that an exploited vulnerability will come from the pool of known vulns. [C8]/[A4]

Getting over the hump with vulnerability counts

Should our vulnerability counts be going up or going down? That is an important question every security professional should be considering when laying out a security program.

If you believe vulnerability counts should be increasing, then presumably you believe that we are only covering the tip of the iceberg with respect to the total number of vulnerabilities in production. In this case, you are taking a short-term view of what is happening in security - it is okay to be hoping the counts increase in the short term, but eventually you want them to decrease (right?).

If you think vulnerability counts should be decreasing, then you might be heartened by this bit of news from ISS' X-Force:

For the first time [in 2007], X-Force witnessed a reduction (-5.4 percent) in new vulnerability disclosures from the previous year.

The strange thing here is that X-Force wants to explain this decrease as a statistical anomaly. I think they should be pointing to it as a potential indicator of success, albeit with a need for more substantiation. 

So, do you want the number of vulnerabilities found in 2008 to be higher or lower than those found in 2007? (Btw, if you have some reason to expect 6000-7000 vulnerabilities to be found this year, and I believe you do, what are you doing to protect yourself from these "known unknowns" RIGHT NOW?)

Update: Funny! People are really vested in ensuring this number stays high. Here's what Larry Dignan at ZDNet/Zero Day had to say. Can you sense the worry? People get attached to this stuff so much that it becomes clear that they NEVER WANT TO BE DONE (not that they would, but presumably that is the end game for a process of finding vulnerabilities).