In the past few months, dozens of organizations across numerous industries have been notified by government intelligence and law enforcement entities that their computer networks have been compromised. These notifications can vary widely in terms of detail - but usually specify internal systems that are compromised and potentially information regarding a phishing attack.
We can tell you from experience - the information you receive is typically the "tip of the iceberg" in terms of total number of compromised systems. Unfortunately, the government is prohibited from giving victims advice on how to address the situation. Therefore, many victim organizations treat it like they would with any other "virus" outbreak and respond to these notifications by immediately removing the corresponding systems from their production network and putting network blocks in place for the malware identified.
Unfortunately, by the time an organization is notified - the attacker has either been there for a long time (2-3 years in some of the cases we have responded to), or has had the ability to deploy second stage malware on other systems that you don't know are out there. Entering "remediation mode" prior to understanding the full scope of the compromise, without a well thought out remediation strategy, and without achieving the operational readiness required to address the attacker(s) is a recipe for disaster.
In this blog, we want to introduce to you three common, tactical remediation steps that many organizations take after discovering for the first time that they have been compromised by the APT. On the surface these steps appear to be obvious and correct when in fact they usually impair the effectiveness of your remediation efforts. These three somewhat intuitive steps are most often counter-productive and we often advise NOT to do these at the onset of your response:
Mistake #1: Immediately entering "remediaton mode" - also known as playing "Whack-A-Mole"
When most organizations are notified of an ongoing computer security incident, they immediately act upon the information provided by the messenger and perform the following actions:
- Remove compromised systems from the network.
- Implement IP, port, or DNS filtering as soon as network indicators are identified.
- Change administrator passwords when their use is identified.
When dealing with the APT, such "knee-jerk" responses:
- tip off the attacker - which causes them to change tactics further impedes the investigation.
- prolong the time and effort required to remove the attacker from the network.
- jeopardize the effectiveness of the remediation .
- have little or no impact to impede the intruder's access to the victim network.
- promote a false sense of protecting data.
In short, the victim organization is not executing remediation steps in the "strike zone" (See my last blog titled "The Challenges to Remediating from the APT".) Immediate and uncoordinated removal of compromised hosts is usually an exercise in futility. The victim organization's resources are likely to "chase" the intruders on a daily basis rather than perform a remediation effort that has lasting impact or potentially worse, they will think that the problem has gone away. In most circumstances, by the time the victim organization is notified the attacker has likely already deployed second stage malware and has already been stealing data from the network. Therefore, the first recommendation, in the spirit of fostering an effective and lasting remediation, is for victim organizations to discontinue the practice of immediate system removal when compromise is suspected. Understand the scope of the compromise first, then execute your removal of compromised systems in a more complete, coordinated and comprehensive fashion.
There are times when where these steps are the right thing to do - specifically for companies that have been fighting the APT for a protracted period of time and can identify an APT compromise within hours or days of the initial compromise. However, during MANDIANT's response to well over 50 APT victims, we have learned that these steps can be ultimately harmful to an organization that has just been notified of an incident to get rid of the attackers from their environment.
Mistake #2: Submitting malware to anti-virus providers as soon as you find it.
During an APT investigation at a Fortune 50 company, we had a "dang it, did that really happen" moment. We had fully scoped the compromise and were about to remove all the compromise at once when hours before executing the remediation plan, anti-virus agents at our client updated and detected some of the backdoors we had identified --- BUT NOT ALL. The attacker accessed 43 systems through a separate backdoor; installed new variants of old backdoors; and installed new backdoors that we had never seen before on systems that were not previously compromised all in an effort to maintain access to the environment. This unexpected AV update stopped a multi-million dollar remediation effort and forced us to continue the investigation and re-scope the compromise. During this time, the client continued to lose data and spend more money to deal with the problem.
We advise you to not submit your malware to AV until AFTER your remediation drill (if at all) for the following reasons:
- You want to remediate on your terms, not when AV companies decide you are remediating.
- When you submit multiple pieces of malware to AV, you will not know when the AV vendor is going to update their signature databases, or how complete their updates will be. In short, they may only solve half your problem on their first update, and not provide signatures for ALL the malware you submitted simultaneously.
- The bad guys have the same access to AV that you have. It is freely available. Ergo, they know when AV is updating for their malware, and they can change their fingerprint quickly.
Mistake #3: Preparing for a single battle, not a protracted war
Many APT victims think that battle is waged once, and then you can wash your hands of the APT problem. Organizations that embrace this mentality often push themselves in a sprint and burn out versus treating the investigation and preparation for future incidents like a marathon. You are the target of the APT because you have political, military, or economic data that is the target of an espionage operation. The fact is, you will have to maintain a sustained effort to detect and respond to the intrusions on a repeated basis. Users will continue to open the attachment or click on the link in the phishing email. When their systems are compromised, your goal should be to identify it quickly enough, have controls in place to limit lateral movement, and have access to centralized log data to make the next intrusion a one-day problem, not a 6-month exposure. With proper posturing steps that create visibility for your IT and security staff (a future blog), you will be able to accomplish this.