Mike Rothman doesn't like metrics. He gives his reasons here and here, and also refers to Ravi Char's perspective here. I have to admit that I find their perspective somewhat bizarre. In his August 7th post, Mike sounds like he is channeling Donn Parker's fatalistic and random finger-crossing approach to security where you either win or lose. And in his August 14th post, he essentially says the same thing, the same thing.
Here are my responses to some of his points:
[August 7th]...folks typically use metrics to show improvement and basically justify their existence...[and] saying you are great will go over like a lead balloon if half of your network went down yesterday...
So, in today's environment, this "all or nothing" approach ultimately ends in nothing. Mike's approach is to manage the PR around the incident. Regardless of "lead balloons" (I"ve never been afraid of them myself) it may actually be useful to do a root cause analysis and then determine what the likelihood of that incident was so that you can either accept the risk or make a change. Without measurement, you can't determine how frequent events like this occur with any level of objectivity and reliability.
I'd still rather folks spend time fixing stuff and reporting on what they are fixing. That's what auditors are interested in...
Again, the PR angle. I don't deny the need, but I certainly hope folks aren't fooled by audit interests. Also, feel free to replace "fixing stuff" with "spinning wheels" since it means the same thing. There is always plenty to do, regardless of whether it is useful.
...not some arbitrary metrics that may or may not represent security posture.
Ah hah! Now we get to the root of the problem (I think) in this final throw-away comment. Mike thinks metrics are arbitrary and may not represent the security posture, and (apparently) has much more confidence in the auditors he reference earlier than an objective view driven by metrics. (We'll see why in his next post).
[August 14th]...who at best are making up numbers to justify their existence... Metrics are deceiving, and I've always said that I can make a number say anything I want it to say. It's all about the positioning.
There it is again and again. No faith.
But ultimately, how do these metrics make my environment more secure? How does this type of initiative help me substantiate my existence better to the people that write the checks?
Well, metrics make your environment more secure as well as any other management tool (like a "to-do" list)... which is to say they won't. However, metrics can provide an objective, quantifiable basis to work from. These better decisions would substantiate existence much better than the Lemmings approach to security. (pssst, Mike - sometimes, the best answer may be to NOT write checks ;-))
As far as I can tell, Ravi's point is similar to Mike's - that there are bad metrics, so therefore you shouldn't use any of them. I agree that you shouldn't pick stupid metrics.
I have no clue why anyone would suggest that the "old school" qualitative approach to security is better than a more scientific one, given our collective cybersecurity track record. (If you despise metrics, please don't complain about the utter fruitlessness of TSA/airport activities - after all, it is YOUR approach TSA is using).
And to completely discard metrics on account of the possibility that some people will pick bad ones is ridiculous.
I am with you on this one Pete. It must be my automotive manufacturing background. You cannot improve without metrics. It is only recently that vendor products have begun to supply metrics that are meant to measure overall posture: numbers that include vulnerabilities, asset value, and threats. The next evolution will be for someone to publish benchmarks by industry vertical and region. Imagine the scramble by the IT department if their score fell below the norm?
Posted by: Stiennon | August 19, 2006 at 12:50 PM