How to Track Phishing Resilience Using a Metrics Matrix

Phishing resilience is hard to measure, and most platforms focus on measurement of a campaign in terms of employee success or failure at a single point in time. To accurately gauge an organization’s ability to handle phishing attacks, metrics are required that demonstrate changing trends over time, identify problem areas and discern between different sophistication levels of phishing.  

This piece explains how to build an organizational phishing metrics matrix to better gauge organizational resilience and identify areas of concern to strengthen your security awareness program.

Create a Phishing Metrics Matrix: A Step-by-Step Guide 

Tracking phishing metrics on a team/group level enables both the security team, as well as managers across the organization, to identify resilience across different areas. The metrics should also take into account that different organizational functions have different risk levels, exposure to phishing and access to sensitive information. 

1. Identify Serial Clickers and Serial Reporters   

To solve the first issue of identifying problem areas (individuals, groups, etc.), consider creating a “serial clickers” metric. This will track anyone who’s clicked on a link in more than three consecutive phishing campaigns. This set of metrics applies across all other sets and will help direct assistance (in the form of training, awareness, intervention, additional controls, policies, etc.) toward the problem areas as the increased risk is recognized. 

You should also look to capture the other end of the spectrum: the “serial reporters” who are able to successfully identify and report more than three consecutive campaigns. This metric shows increased awareness and resilience and can be used to highlight the individuals or teams that exhibit such behavior. 

2. Create Different Levels of Phishing Campaigns 

The next step is to create at least three different levels of phishing templates: 

• Common: This is for the typical mass-mailing, nontargeted phishing seen in the wild. 
• Business: These campaigns target business-related assets and platforms, such as customer relationship management, enterprise resource planning, human resources, finance, infrastructure, and email. 
• Advanced: These are spear-phishing campaigns, that target individuals, such as business email compromise, whaling, etc. 

Creating these levels allows you to measure phishing resilience across additional dimensions beyond success/failure and adjust the sophistication of the phishing campaigns used for different types of internal groups based on their sensitivity or progress in the awareness training. The level changes can be applied to individuals, as well as to whole teams. 

3. Track Basic Indicators 

Next, you should measure the following key indicators across the campaigns: 

• Ignored: Did not open the email at all. 
• Opened: Opened the email but did not click on anything. 
• Reported: Reported the email as phishing. 
• Clicked: Clicked on a link in the email. 

You may also wish to measure “engaged,” which is when a person clicks on a link in the email and engages with the phishing website or downloads an attachment. 

The goal, of course, would be to see mostly ignored, opened and reported metrics (with a bias toward reported as an indicator of a more proactive level of security posture that is likely elevating others in the department). In more advanced settings, you can also measure reporting across out-of-band channels, such as Slack or email to non-security addresses that warn about the suspicious emails. 

4. Create the Matrix 

Once you collect the metrics, you can report them in a matrix format that clearly identifies areas of investment or reward (see Figure 1). 

The matrix makes it easy to identify the different levels of phishing sophistication targeting each department, as well as each department’s ability to identify and respond to those attacks. Tracking those metrics over time enables security teams to take specific actions (educational, disciplinary, communications, etc.) toward the right departments and track the efficacy of those actions. 

Organizations can then treat each class of action as a detractor or a promoter of phishing resiliency, where “ignore” and “open” are neutral, “report” is a promoter (i.e., better resiliency), and “clicked” is a detractor. They can also define the weights of promotion/detraction based on the type of organization and its risk posture. 

READ: 10 Ways to Identify a Phishing Email 

Phishing Resilience Metrics That Work for You 

Tracking which groups are susceptible to which types of phishing campaigns over time can be difficult, and tracking the results of various interventions (education, punishment, etc.) makes the process even more complex. Creating a metrics matrix can help clarify the process. To get started: 

• Choose the groups you want to track: This can be specific departments or groups within departments. 
• Track their engagement with various levels of campaigns: Tracking opens, clicks, etc., across the various levels of phish (common, business, and advanced) helps paint a more accurate picture of resilience. 
• Track changes post-intervention: Once trouble areas are identified and targeted for increased awareness, education, etc., use the matrix to track each group’s performance to see which types of interventions are most effective. 
• Don’t forget to highlight good behaviors: The matrix also makes it easy to see which groups are modeling the right behaviors so they can be highlighted/rewarded appropriately. 

Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in connection with such information, opinions, or advice.