MANDIANT has been involved in numerous widespread remediation efforts following intrusions at large organizations. We have seen nearly identical recurring challenges emerge at these large organizations, and we believe conveying these challenges may be important to developing your overall approach to remediation should you be compromised by advanced and persistent threats:
- Remedial efforts usually take more effort and determination than anticipated. It is a good principle to begin managing the expectations of the business units as soon as possible, ensuring they are aware that the remedial efforts may involve continual effort, resources, and periodic adjustments based on the dynamics of the ongoing threat.
- Good remediation plans often fail because determination wanes if each phase of the plan is not short and concise.
- An ineffective remediation is initiated because the plan was implemented prior to understanding the tools and techniques of the intruder.
- An ineffective remediation plan was initiated because the remediation plan took too long to develop, allowing the compromise to become so widespread that remedial efforts become very time consuming and costly.
- The remediation plan fails because accountability for its execution is not clearly assigned to an individual. Each business unit should assign an individual who becomes responsible for the implementation of the plan.
- Remediation fails due to lack of resources - lacking the personnel, technology and processes to follow through on the remediation plan.
- Remediation fails because the organization tipped off the attacker. In past investigations the following actions have tipped of the attacker:
- Removing compromised systems as you discover them
- Changing one-off domain admin passwords when you see them used
- Blocking Command and Control (C2) channels as they are identified
- Remediation fails because the organization does not or cannot disconnect from the Internet while undertaking remediation activities.
In order to develop and execute an effective remediation plan, not only must you understand these points, but you must also execute your remediation at the right time. Remediating too early does not help your organization defend your networks, and remediating too late ensures you have lost all your trade secrets to the adversary. Therefore, below are some key criteria for you to use in order to assess your remediation "timeliness":
You are remediating early, and entertain higher risk of re-compromise or not fully remediating the current compromise when:
- Host-based indicators of the current compromise (the attacker's fingerprints) are unknown
- Network-based indicators are unknown or transaction based
- New compromised hosts are still being detected at a high rate (more than one per day)
- There seems to be no established pattern to assist your organization in anticipating the next compromised host
- There is little coordination between business lines (Staff) concerning remediation
You are potentially remediating too late when:
- Host-based indicators are becoming less reliable (they are not hitting on anything)
- Network-based indicators are becoming less reliable
- You know the attacker has been active and they just "vanish"
- Staff motivation and concern has waned
- Remedial activities have evolved from corporate-wide efforts to independent "splinter cells"
You are in the strike zone and remediating at the right time when:
- Host-based indicators are stable
- Network-based indicators are stable
- The delta to detect new compromised hosts is shrinking consistently
- Your organization is postured to actively identify and address the "next generation" of attacks - In many of our investigations we see the attacker, or a different attack group, return. We have seen it take 3 days, 3 weeks, or even 3 months - but inevitably they try and come back. Your organization has information that is deemed valuable and the attackers won't just "go away" when you kick them out).
- There is active communication and coordination between business lines (Staff) concerning remediation
Once you understand when to time the execution of your remediation plan, you should execute it. MANDIANT will be posting numerous blogs throughout the next several weeks to provide the technical details behind many of the longer term remediation strategies that cannot be implemented prior to eradicating the current incident.