This is part one of a series of posts I plan to make on what Mandiant does to "Find Evil and Solve Crime". These posts should help to make your organization better, faster and stronger at performing effective computer security incident investigations. And hopefully they will spark some good discussion about improving incident response. The first part is about focus.
One of the biggest challenges during an investigation is staying focused. It is all too easy to chase shiny objects, and sometimes even not-so-shiny objects, and never figure anything out. So in this part, I'll talk about what Mandiant does to help prevent an investigation from falling off track.
According to the Carnegie Mellon Software Engineering Institute:
"When an incident occurs, the goal of the CSIRT is to control and minimize any damage, preserve evidence, provide quick and efficient recovery, prevent similar future events, and gain insight into threats against the organization."
I think that statement does a great job summing up what the incident response process is all about, and what Mandiant commonly focuses on. But I have something to add. Mandiant pays special attention to specific high-level questions that need to be answered.
Based on discussions with our customer, we form high-level questions early on in the investigation. These questions define what topics will be in the executive summary of our final report. They also provide guidance for all investigative activities - the team should only perform tasks that contribute to answering those questions.
During an investigation, I consider myself a fact-finder looking for facts to answer those questions. I establish and search for facts that describe the incident. So it's very important to start the investigation on solid ground. We start by discussing the incident, in technical detail, with our customer. We directly verify and examine all initial facts about the incident by acquiring the data that shows the facts - forensic images, firewall logs, etc. Once we know we're on solid ground, we start to fan out - thinking of ways to look for evidence associated with our initial facts.
As we proceed, leads can take us down many different paths, some that help and some that hurt. To stay focused here, we take an old school approach: Who, What, When, Where, Why and How. Staying focused on answering these questions will help us Find Evil, and eventually Solve Crime.
One very important aspect that will help is to develop a clear picture of the context of the incident and evaluating every new investigative action against it. This will help keep the team focused and efficient. Here are examples scenarios Mandiant has seen challenge an IR team's ability to stay focused:
Problem: The team is not keeping a list of affected systems, malware found, indicators of compromise or dates and times of significant events. Also, no one is keeping a list of evidence gathered, outstanding tasks or responsible persons. Each day, team members are thinking someone else is working on a critical task but things are slipping through the cracks.
Solution: Assign a single team member to maintain a Web page, document or spreadsheet with all that information.
Problem: Team members are spending a lot of time analyzing data. Forensic images are scrutinized in painful detail. Lead generation is slow.
Solution: Before a team member begins analysis on any type of data, provide them with the following:
- Lead information (i.e., what do we already know)
- A list of questions you want answered
- A report template
- Discuss the elapsed time expected to complete the analysis
- Request the analyst inform you as soon as he or she thinks they are not on track with expectations
Problem: An investigation clearly identifies malware associated with the incident on a computer. While performing detailed analysis, additional malware is identified on the system. The team extracts the malware, performs analysis, and begins searching for indicators of that malware on other systems. They find the malware is on additional systems and begin investigations of those. Unfortunately, the malware was unrelated to the incident at hand, and the team wastes days of work.
Solution: A quick review of the context of the finding - as simple as checking the date the malware was created.
Problem: The team provides IP addresses and domain names to the proxy administrator to search for. While searching for those indicators, the administrator begins to look at the logs more closely than they ever have before. The administrator discovers entries that seem suspicious. They are reported to the team. The team begins to investigate these entries - responding to the computers responsible for the traffic. The team finds no correlation between these systems and the systems involved with the incident at hand.
Solution: A quick sanity check on the reported entries - are they similar to other known activity? Are they within the time frame of known activity?
Mandiant has found that the biggest challenge to most investigations is in how they are managed. We've found that sticking to the basics - documentation and communication - go a long way to help us stay focused to Find Evil and Solve Crime.
Stay tuned to M-unition for my next post, where I'll discuss collecting and analyzing data that's useful on an IR. I'll talk about some challenges and provide suggestions to help.