It’s critical to establish a set of security metrics and KPIs that measure risk improvement and effectiveness of your overall security program for several reasons beyond reporting to the board. Metrics tailored to both your security requirements
and the business will help guide future security decision-making and improve the security posture of your organization.
Security leaders can make it easier to measure risk improvement across vulnerability management, product security and other areas by using a framework to build key performance indicators (KPIs). This piece provides detailed guidance on creating meaningful
security metrics, KPIs and benchmarking.
Types of Security Metrics
The two main types of metrics used in security are “deployment” (i.e., percentage of coverage) and “effectiveness” (impact on risk reduction).
Deployment Metrics
Most deployment metrics focus on percentage of coverage. A great example can be seen in the Center for Internet Security (CIS) Metrics. The CIS metrics are supportive
of the broad set of CIS benchmarks and the Top 18 CIS Controls.
CIS metrics break coverage into six “sigma levels.” These are progressive measures of deployment that range from 69% or less to 0.00034% or less (as follows):
- Sigma Level One: 69% or less
- Sigma Level Two: 31% or less
- Sigma Level Three: 6.7% or less
- Sigma Level Four: 0.62% or less
- Sigma Level Five: 0.023% or less
- Sigma Level Six: 0.00034% or less
For example: What percentage of the organization’s user accounts are not disabled if they cannot be associated with a business process or owner? An answer of 50% would be a Sigma Level One.
Other deployment metrics (also exemplified by CIS) focus on yes/no types of measures. In short, they state whether a control is deployed or not. For example: Does the organization automatically lock workstation sessions after a standard period of inactivity?
Overall, deployment metrics do not reveal the effectiveness of a control. They reflect how good you are at deploying security controls—not if those controls actually work well.
Effectiveness Metrics
Effectiveness metrics measure the impact a security capability has on reducing risk. An example effectiveness metric could be: 90% of critical vulnerabilities are remediated within two weeks. Strictly speaking, remediation is a process, and the capability
is vulnerability management. A security capability will have several metrics that measure its key processes.
The goal for most metrics is security effectiveness and improvement, but these metrics require more thought. They are often customized to an organization's improvement needs. The following high-level framework makes creating effectiveness metrics simpler.
READ: Establishing Key Metrics to Monitor Network Performance
A Framework for Better Security Metrics
There are five canonical effectiveness metrics. The first two are “shift-right” measurements, which means they focus on improving the effectiveness of capabilities for cleaning up risk. The last three are “shift-left” metrics that
focus on measuring capabilities that support risk prevention. Below, are vulnerability management examples, used for simplicity and wide usage. However, you can measure any security process this way, be it related to vulnerability management, threat
intelligence, product security or something else. The five metrics are:
- Risk burndown: This is a ratio of “risk removed” over “total risk.” For example: In January, there were 100 critical vulnerabilities discovered. During that time, 50 were fixed. The ratio is 50/100, which is a 50% burndown rate.
The next month, 50 vulnerabilities were discovered and 10 were fixed. When we add up the data for those two months, we get 60/150, which is a 40% cumulative burndown rate. More advanced measurements can then reveal the spread of this data and its
statistical baseline over time.
- Risk survival: This measures the “time to live” of risk. The simplest approach measures this in three groups: 50%, 10% and 5%. An example metric here would be: 50% of critical vulnerabilities are remediated in two weeks, 10% are remediated
in 90 days, and 5% are remediated in 200 days. Note: You may see improvement at the 50% mark, while the 5% group starts to take longer. More advanced methods can then cover measures across all units of time as a curve (this is formally called “survival
analysis”).
- Risk arrival: The simplest way to measure this is with an average over a timeframe. For example: Over the last quarter, critical vulnerabilities appeared at an average rate of 25 per month. Advanced methods can then allow you to predict the rate by stating
things like: There is a 25% chance of 40 or more vulnerabilities showing up per month, and a 90% chance of five or more. Effective shift-left controls can reduce the rate at which risk materializes.
- Risk inter-arrival: This is a more advanced metric used to measure how fast risk materializes. It predicts the rate at which the next risk will show up. For example: There is a 90% chance the next critical vulnerability will appear in the next two weeks,
and there is a 25% chance it will show up in the next four days. Changes in risk inter-arrival provide an early warning of changes in arrivals. A simple way to calculate this is to:
- Track vulnerability arrivals in the last 45 days: For example, Day 1 saw one, Day 2 saw five, Day 3 saw 10, etc.
- Track the difference in arrivals: The difference from Day 1 to 2 was four, the difference from Day 2 to 3 was five, etc.
- Average up those differences to come up with the average inter-arrival, i.e., 2.3 days.
- Risk escape rates: This is a canonical measure in software development. It is used to measure the rate at which software bugs get into production. It uses the exact same math as burndown rates. Imagine you discover 10 vulnerabilities in development. Prior
to deploying to production, you fix eight of them. Thus, two vulnerabilities escape into production. That’s 2/10, which makes for a 20% escape rate. This is a cumulative rate that baselines over time similar to burndown rates.
READ: Key Server and Endpoint Security Metrics to Track
How To Make Security KPIs
The five effectiveness metrics help ease KPI development. You can use them across capabilities like vulnerability management, product security, threat intelligence, incident response and more. The goal is to start simple. Start by picking one metric,
one capability and one process. For example:
- Metric: Risk burndown
- Capability: Vulnerability management
- Process: Critical vulnerability remediation
There are multiple reasons to start here. First, critical vulnerabilities must be fixed. Left unfixed they lead to exploitation and breach. Second, most organizations have vulnerability management programs. Last, measuring remediation can be straightforward.
Either your vulnerability management system or a ticketing system can source the data.
Now that you have your metric, you can create a workable KPI using the following steps:
- Baseline your current remediation rate: It will likely take 90 days’ worth of data to get a stable baseline.
- Select a target KPI: When in doubt, start with a greater-than-90% burndown rate.
- Set a final KPI: Assuming you can achieve the target, then based on your risk appetite, you can set a KPI for greater than 95%. (Note: You will have achieved that rate if you meet or beat it consistently over a 90-day period).
- Rinse and repeat: Your next step is to take the burndown metric and apply it to several other capabilities and their processes. The key is to search for processes that are problematic. Be discriminating. If something is working well, there is no need
to measure it.
Over time, you can apply all five metrics across myriad security capabilities and their processes, creating dozens of useful metrics.
READ: Operational Metrics CISOs Should Present to the Board
Creating Effective Information Security Metrics
Creating worthwhile information security metrics and KPIs for executive leaders and the board doesn’t have to be difficult. To improve your chances of
success:
- Follow the framework: The five canonical metrics can be used to measure any security-focused process.
- Start small: Begin with one metric, capability and process and move on from there.
- Start now: Baselining security metrics and KPIs takes time. Give yourself at least 90 days to achieve a workable baseline.
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in
connection with such information, opinions, or advice.