Threat Hunting Basics
At its core, a threat hunting team’s mission is to look for indicators of compromise (IoCs) that have gone undetected by other security controls. As such, a hunt begins with the premise that the organization has been compromised but doesn’t
(yet) know it. At a minimum, that requires some level of understanding of the existing security controls. It’s also helpful to have an up-to-date understanding of threats.
Most of today’s malicious software and activities fall into one of only a few categories.
- Malware that modifies a system: It typically makes some unauthorized changes to a system’s persistent storage so it can survive a system reboot or be invoked when a specific action is taken on an infected system.
- Malware that doesn’t modify any media: This generally makes unauthorized changes in the system’s process memory space. Such malware can be nearly impossible to detect because it leaves no persistent footprints. On the other hand, it doesn’t
usually survive a reboot.
- Interactive compromise: This is accomplished through a network and usually is the result of some configuration defect (e.g., a system default “admin” username with a password of “admin”) or an exploitable vulnerability in a network-connected
service.
A hunt team should have the ability to detect any of these threats, even if the malware uses entirely new techniques and can otherwise bypass every deployed security defense.
How Treat Detection Tools Work
Most of today’s security tools, at least those intended to detect and/or prevent threats, operate by continuously monitoring a system (e.g., process memory, disk storage, network activity, etc.) and looking for IoCs, which are in the form of:
- Signatures (static or highly complex) of known attack tools and techniques.
- Known behaviors of malicious tools and techniques.
Vendors often speak of other proprietary techniques their products use, but they generally fall under those two broad categories.
Fundamentals of Detection
One of the problems with this approach is that detecting malicious activities and tools on a running system can be problematic. With the advent of rootkit techniques in the mid-‘90s, adversaries gained the ability to compromise and modify systems
to make malicious software exceedingly difficult to detect.
Thus, reliably detecting modern malicious tools is best done by starting from a known and trustworthy starting point. This is generally best accomplished by initializing the system being analyzed from otherwise trustworthy boot media, and then examining
the storage devices on the system being analyzed.
Of course, no technique is perfect. Even booting from trustworthy removable media can yield false results if the system has been compromised at the firmware or hardware level.
Nonetheless, the most consistently reliable method of detecting malicious activity on a system is to boot it from trustworthy media and examine its persistent storage for indications of unauthorized changes.
Process memory can also be examined for malicious tools that do not make changes to storage media, but since most of these do not survive reboots, they present another challenge for a hunt team.
READ: Setting Up a Successful Vulnerability Management Program
Getting Started with Threat Hunting
At the most basic level, threat hunting teams should:
- Examine systems for unauthorized changes, irrespective of how those changes were made. Once the changes are known, they can be examined to determine what caused them. They could, after all, have resulted from otherwise legitimate system administrative
activities.
- Examine process memory, but since that requires examining a running system, the hunt team should be mindful that a rootkit-like tool is already in place, so whatever they examine may in fact be incorrect. Also, process memory, by its nature, is dynamic
and constantly changing due to the system’s processes merely being used.
- Look for interactive compromises. System activity logs should be analyzed to look for anomalous behavior. Again, the possibility of a rootkit-like tool or technique being used to alter or incorrectly report system activity is very much in the realm of
possibility. Indeed, some of the Unix and Linux rootkit tools from the 1990s did exactly this.
Because of this, it is not sufficient to only examine system activities as reported by the system itself. One place to look for (relatively) independent indications of system activity is to do traffic analysis of a system’s network data over time.
Data such as NetFlow can be analyzed to look for anomalous or otherwise unauthorized network behavior.
From a practical standpoint, then, a hunt team should be built on numerous activities for detecting IoCs. These should include the following:
- Boot systems from trustworthy removable media and examine the non-volatile storage for indications of unauthorized changes. This can be done by comparing installed system components with trustworthy versions of the same components. It can also be done
by comparing the system components over time and looking for unauthorized changes. (These secure “snapshots” must be updated every time a system or application update is performed.)
- When booting isn’t feasible, the same sorts of examinations for unauthorized changes can be done using an administrative account, preferably after a reboot and running in single-user mode, or at least with no network services operating.
- Examine process memory for unauthorized processes. Similarly, look for network services running without authorization.
- Analyze logs. Examine system event logs, network traffic logs, etc., for indications of unauthorized network activity.
The above list is a solid starting point for hunt activities.
Automate Hunting
Each hunting activity carries with it myriad technical challenges. Some, but not all, of those challenges can be addressed through automation. For example, it may be helpful to automatically:
- Track the systems being examined by the hunt team.
- Tally results and scans.
- Maintain an inventory of trustworthy system components for all deployed versions of all operating systems and core application software used in an organization.
This is really just scraping the surface of the problem space. Because hunt team activities are relatively nascent, tools are scant and still fairly immature.
Threat Hunting Tools to Consider
Most hunt teams cobble together a tool chest of security tools and custom-made scripts and processes for analyzing systems. While certain tools can be helpful in looking for unauthorized changes in systems, they can sometimes seek to solve a different
problem set than what most hunt teams face today. Tools like Splunk can be enormously helpful for analyzing NetFlow, syslog and other system activity log sources, looking for signs of unauthorized activity.
However, when it comes to low-level analysis of critical IT assets, there remains plenty of heavy lifting in terms of developing operating procedures for booting a system from trustworthy media and examining all the installed software for indications
of unauthorized changes.
Scope of Hunt Teams
Most hunt teams operate on a limited scope and examine just a subset of an organization’s most vital IT assets. They rarely, if ever, are called to examine general-purpose desktop systems because the complexity there is enormous. Either that, or
they examine a mere subset of system attributes, such as an inventory of installed application software vs. an authorized whitelist of authorized application software.
One notable exception to this is special-purpose internet of things (IoT) devices such as security cameras, point-of-sale (PoS) terminals and such. Many of these ae built from open source components and many provide some form of administrative interface
for examining the firmware version or even entire file systems for indications of unauthorized activity.
While the same caveat regarding supply chain attacks exists on IoT devices, even relatively simple actions like verifying the checksums on firmware images can be helpful to a hunt team.
READ: How to Build a Proactive Threat Hunting Strategy
Setting Up a Threat Hunting Team
In summary, setting up a hunt team has no shortage of technical challenges. Most teams find it helpful to:
- Go slowly: It’s best to build your capabilities one small step at a time.
- Keep it simple: Even some simple but low-level data collection and analysis of business-critical systems can yield interesting results.
- Build on what you learn: Once you have those early results in hand, you can then justify further and deeper inspections.
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in
connection with such information, opinions, or advice.