The SolarWinds breach underscores just how complex supply chain attacks can be. Because it targeted network management software, the impacts were extremely damaging. Attackers gained immediate access to an extremely sensitive system, centrally positioned
in the network, with credentials that, in most cases, could be used immediately to move laterally to other systems.
If you were compromised by this SolarWinds software update, know that regardless of the procedures you followed for validating software updates, you were likely to have been impacted. There is very little organizations can do to prevent a software supply
chain attack like this, but threat modeling will provide the best possible outcome should one occur. This report outlines best practices for responding to breached SolarWinds Orion servers, including threat modeling network management systems in general.
SolarWinds Breach: What Happened
On Dec. 13, 2020 Reuters reported the U.S. Treasury Department had been breached. The Washington Post quickly followed with news the same attackers who had breached FireEye (reported the week of Dec. 7) were also responsible for the Treasury breach. The
Post also reported sources close to the investigation claimed a supply chain attack on SolarWinds’ Orion network management software was responsible for both intrusions, and that both intrusions were the work of Russia’s SVR intelligence
service (aka APT29/Cozy Bear).
Organizations using any version of Orion (not just the one confirmed by SolarWinds as compromised) should assume they have been breached and begin threat hunting for intruders. Unfortunately, this attack group was sophisticated, both in its development and operational activities.
Post SolarWinds Breach Actions
- Taking your Orion servers offline, or at the very least, isolating them from the internet.
- Examining logs of Orion and managed devices to determine if anomalous connections have come from the Orion server to managed devices in the network.
- Examining logs of non-managed devices to determine if the Orion server has communicated with them. If it has, assume they have been breached and act appropriately.
- Not trusting any version of Orion software. Do not limit assumption of breach to the specific versions identified by SolarWinds in its Dec. 14 SEC filing. Presume there is still data unknown at SolarWinds and act as if all versions have been compromised.
- Changing credentials for any service accounts used on the Orion server.
Inventory Your Orion Credentials
Due to the way the Orion user interface is built, it can be difficult to understand the full set of credentials configured for use by a SolarWinds Orion server. Unfortunately, this is critical in the context of evaluating the impact of a breach. Fortunately,
Rob Fuller (@mubix) has been researching Orion servers for years in the context of red team operations. He published a blog post compiling information he's learned from targeting SolarWinds Orion servers over the years. In it, he explains how credentials are stored across different versions of Orion and how they can be decrypted. He also provides links to some of his conference
talks discussing how attackers can use a compromised Orion server. Watching these videos is a must for security and operations teams seeking to threat model the impact from a compromised network management system.
Fuller also released a tool, called SolarFlare, that decrypts passwords and credentials configured in the SolarWinds Orion database. He says credentials previously configured in Orion may persist even after being removed in the Orion console. This is
likely the result of normal activity in the database engine, because deleted entries are frequently retrieved from database tables in forensic operations. SolarFlare performs a similar operation, so it stands to reason it can recover inactive credentials
operations that personnel believe were previously deleted.
Rotate Passwords and Key Material
Teams should also rotate any credentials used on a compromised Orion server. However, the credentials stored on the Orion server do not tell the whole story. Most Orion servers can retrieve data from other systems, often including network device configurations.
If attackers have a stolen network device configuration, they can recover a variety of sensitive data (as applicable to the specific device), including:
- Logging posture
- Access control lists (ACLs)
- SNMP community strings
- Hashed passwords
- Device logs (useful in locating network administrators)
- Private keys used for HTTPS and SSH
- Private keys used for virtual private network (VPN) services
The private keys should be considered especially important. While many attackers would find it difficult to operationalize a private key, the attackers associated with the SolarWinds breach are extremely capable and could make use of it in the context
of an intelligence operation if they have somehow compromised an ISP.
If the Orion server is configured with passwords providing access to web servers, SSH servers or any other device where keying material must be kept secret, those keys should also be rotated.
When in doubt, bring in red team members to evaluate the impact of compromised material. If dissent still exists on the necessity to rotate the passwords or authentication material, we recommend erring on the side of caution. Under no circumstances should
you think "the attackers won't figure out how to use this." While that might be a viable approach in some circumstances, it is not the case here.
Best Practices for Securing Net Management Systems
Several best practices can help protect against attacks that target network management software, including:
Limiting its communications: Limiting access to the internet is considered a best practice for network management systems in every case. Since the systems can provide a high degree of access to an attacker, their network communications
scope should be limited. Ensure network ACLs limit the system to only communicate with managed devices and only using the specific protocols required.
Reviewing its configurations: Most network management systems are configured by operations teams without consulting security. Security should review existing configurations.
Threat modeling: Security should also threat model the access attackers might gain by a successful compromise. Historically, organizations that threat model what a compromise of a critical asset would give an attacker fare the best in
extremely complex situations like a supply chain attack. In general, InfoSec should be involved in threat modeling the configuration and use of all orchestration and automation platforms. Security should not become a blocker to the use of these platforms,
but information security teams brings valuable perspective because it understands attacker tradecraft in a way operations teams may not.
Rebuilding Orion Server
It is not enough to simply install a new version of Orion. The Windows server on which Orion is installed must also be reloaded. And avoid restoring from backup. SolarWinds has not yet completed its internal investigation and we don't know for sure how
far back this intrusion goes. There are also questions about whether the new version of Orion that SolarWinds released is clean. While we have scanned this version with the Yara signatures released by FireEye and found nothing, this would be expected
even if the attackers are compromising the SolarWinds build process. There are countless researchers digging further into the latest Orion release. However, knowing how well the original backdoor was camouflaged and how large the installation is,
it could be days or even weeks before another backdoor is discovered if present.
The compromised Orion installation calls out to a command-and-control server to download secondary malware payloads (FireEye calls one such payload "BEACON," which is a reference to a loader for Cobalt Strike). This domain is used to bootstrap additional
operations, but it is now under the control of Microsoft, likely mitigating any direct threat from a freshly installed version of Orion. Microsoft controlling the domain is sufficient to prevent new breaches as a result of the backdoor, but it is
important to remember the attacker may have already downloaded other malware to an Orion server. All samples of secondary malware detected so far use other domains for command and control.
Use of IoCs Moving Forward
The group responsible for this attack is known to take countermeasures to avoid detection and practice active counterintelligence, adapting its operations based on what it believes is publicly known about them. This means any publicly available indicators
of compromise (IoCs) are unlikely to be useful in detecting the intrusion. By the time you start using the publicly available indicators in your environment, the attacker will have likely changed those indicators to prevent detection. Don't interpret
this as a requirement to more quickly operationalize indicators. Instead, interpret it as a need to think about logging.
By having logging available, defenders can execute the discovery course of action (CoA). The discovery CoA looks backward for newly discovered indicators, while the detection CoA looks forward. The sophistication of these attackers' actions are likely
to render detection CoA difficult. We recommend organizations consider what indicators they can execute the discovery CoA against and make logging changes to close gaps going forward.
As we continue to learn about more instances of confirmed compromise, we have yet to see overlap in network indicators, beyond the initial backdoor callback domain. While we predicted the attackers would change IoCs rapidly on disclosure, it now appears
they were also ready for the inevitable investigations. We should not anticipate network-level IoCs will be useful in this case.
A researcher named Ean Meyer is compiling information from threat researchers, including IoCs, into this GitHub repository. It is an excellent resource that is being updated frequently.
Determine Whether a Domain-level Compromise Occurred
The group behind this supply chain attack is extremely capable. Even without the privileged access provided by an Orion server compromise, it would likely elevate to domain administrator in most networks. If IoCs or anomalous logs are discovered that
lead the security team to believe a domain admin-level compromise occurred, be ready to take the appropriate actions to secure the do.
Once an attacker takes over the domain admin account, it is extremely difficult to track what they did that may allow them to regain access quickly once they rejoin the network. Consult with Active Directory (AD) security professionals to ensure malicious
modifications have not been made, including:
- Backdoor accounts
- Group modifications
- GPO modifications
- Delegation changes
- ACLs modified on directory services objects and attributes
Additionally, if the attackers obtained domain admin permissions, changes will need to be made to the krbtgt account to prevent them from immediately regaining these permissions using a Golden Ticket Attack (see this Microsoft AD documentation for information
on resetting the krbtgt password).
Securing Network Management Systems
The following recommendations apply to all systems that hold credentials, including:
- Security orchestration, automation and response (SOAR) platforms
- Password vaults
- IT orchestration systems
For all these systems, InfoSec teams should consider:
- Provisioning accounts with the absolute minimum privileges necessary to perform the specific required operations
- Never use accounts provisioned for orchestration for any other purpose
- Monitoring for use of orchestration accounts performing unexpected activities, such as interactive logins
- Using network ACLs to limit communication to only the destination IP addresses and ports required for orchestration
- Alerts on any communications from the orchestration platform that is not explicitly expected (this is impossible unless you have ACLs configured).
- Limiting communication from the orchestration platform to the internet.
- Configuring an aggressive audit posture on the orchestration server and forward event logs to a SIEM.
Prepare for and Defend Against Supply Chain Attacks
Supply chain attacks are amazingly complex. Even when properly investigated, attackers appear to come out of nowhere. Defending against these attacks is difficult as well. If your organization was impacted by this attack, know that you could have done
everything right and still have been impacted. This may not have been a failure of your information security
Organizations that move forward assuming their Orion server and managed devices have been breached will be in a better position to respond and repel the attackers. In addition, threat modeling and applying robust security controls around your network
management software should help reduce the risk of similar supply chain attacks in the future.
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in connection with such information, opinions, or advice.