Industry Perspectives

Measuring Success in Cybersecurity

If you had said 10 years ago that a company could fail as a result of a cyber attack you may have thought that you were watching a science fiction film. Today, the reality is quite different. Cyber is now a regular boardroom topic as the impact of stock price drops and executive departures post major breaches make headlines. In this environment, how do you define success with security? There are two prevailing methods:

Method 1:  We’re compliant!

There was a time when the metric of success for cyber security was simply the compliance of basic hygiene controls such as anti-virus and firewalls. Today, many industries are regulated by such controls that are focused on the practical application of good security practices. Yet the reality today is that incidents occur even when such regulations are adhered to. Be it a zero-day attack, social engineering or simple human failure, there are many reasons why incidents still occur -- which highlights that such compliance metrics alone don’t help the business measure the success of their investment in cyber security.

Worse, breaches are still seen as a failure and can lead to the departure of the CISO or key security staff. In some instances occuring due to core failings in strategy, more often the right fundamental controls were in place as dictated by the regulators. And this creates a Sisyphus effect where its very difficult to shift security budget based on risk. It's interesting to note, using IDC numbers, how budget allocations have changed very little in the past few years:

IDC shows that from 2003 to 2011, security spend has increased from $12B to $28B, more than doubling. Ironically, the allocation of that spend has hardly shifted. Today, CISOs rarely drop security controls.

Method 2:  We’re not in the headlines

Denial does not lead to success. Enough said.

So how do we measure success?

Today, many use some form of volume-based metrics. This can be the number of incidents that were blocked or succeeded, or the time to detect or respond to an incident, all of which are very useful KPI’s to evaluate how well security teams are functioning ... but not an effective way to measure success.

If incidents do happen, success must surely focus on how we manage the commercial impact they have, for which the scale between attacks is virtually from zero to catastrophic. Although the theory is easy, the practicalities and measures can be complex. Insurance firms have spent much time and energy in developing such models to validate what is the right policy to provide customers at the appropriate price. In cyber security, our goal must be do much the same: balance the right investment to prevent, detect, respond and contain proportionately to the risk.

That is very unlike the insurance market where years have been spent building actuarial tables for risks affecting homes, cars or people, all of which are relatively static and predictable environments.

But cyber risks evolve daily so metrics can’t work unless they incorporate incidents, especially time to resolution. Cyber risk must be considered a living scale. Much like a military DefCon status, it can be influenced by internal and external factors so good insight into what is relevant in your industry vertical, region and your technology deployment are paramount.

ISO offers framework for qualifying cyber risk (ISO27005) that takes business drivers and maps it against technology in terms of the business impact levels. If we understand the key business processes, we can understand how technology enables this and therefore assign aN impact level if that asset were to be compromised. We can then start to qualify the relevant threats and probabilities of an incident to these assets. To quantify the impact we must then qualify the direct and indirect losses that would occur during and incident. Map these together and we now have the foundations of business risk profile that can be discussed and agreed with the business.

This enables the most rudimentary measure of success. The identification and agreement of risk, and our ability to ensure we do not cross the “Maginot Line” of commercial impact agreed with the business. As much as we apply good hygiene, how we respond to incidents will be critical to this success.

Behind all of this we must have key KPI’s so the business can qualify our progress along the journey. Typically these could include some or all of the following:

  1. Can we effectively map the cyber risk to the board on a period basis?
  2. Can we determine what are the acceptable levels of impact to key business systems?
    1. Were we able to measure and achieve these goals (e.g. Uptime/response times/etc.)?
  3. How quickly can we determine a change to the risk profile either through external factors (such as new threats) or internal (such a new business drivers or technologies deployed)?
  4. Could we quantify the investment in security was proportionate to the risk?
  5. Did we achieve acceptable cyber hygiene levels?
  6. Did we achieve our regulatory requirements?
  7. Did the policies, procedures, people and products deliver the expected levels of protection?
  8. Were we able to identify incidents in a timescale agreed upon by the business that were relevant to the risk posed?
    1. Can we effectively qualify the impact of an incident?
    2. Do we understand the attacker and their motives?
    3. How long did it take to respond to the incident?
    4. Did we effectively manage the impact it commercially had?
  9. Are the incident processes we have well drilled and effective?


Measuring the success of cyber is and will for the foreseeable future be fickle, as it is dynamic and success can be achieved by both luck and judgment. What is critical is that there are an agreed-upon metrics between the business and the security team, as dependencies on technology only increase for most businesses the focus on cyber security grows. "Nothing bad happened," can no longer be an acceptable answer as it neither qualifies the investment made nor confirms we have been effective in our role. If we are to get the investments and resources we desire, we must ensure we can qualify and quantify the dynamic risk of cyber, having an agreed-upon threshold and management strategy to ensure that when we cross it, we can identify it and respond effectively.