Sometimes our work here at VDA Labs falls into a more reactive mode than a usual penetration test or code review that is done before a problem is discovered. The stuff has already hit the fan, and some expert assistance is needed to determine exactly what happened and how it was possible in the first place. That means rolling out for a full on incident response.
Often times those activities involve many of the following components:
- Vulnerability Scanning / Gap assessment – looking for possible entry points that could have led to the compromise
- Malware analysis – if there is malware associated with the incident, or discovered during the investigation
- Endpoint investigation – checking into the event logs, AV alerts, etc found on any involved endpoints
- Network monitoring – when possible conducting threat hunting to determine if the incident is ongoing
Unfortunately though we often run into situations where the environment we are working with is not set up in a way that is amenable to the work we need to do on an IR engagement. VDA Labs works with organizations both very large and small, but it is most common in the SMB market that there are gaps in the IT infrastructure that hinder investigations. This post is about identifying some of those gaps so you can make sure your organization is prepared to respond when an incident happens.
Data Retention – Log (and retain!) All The Things!
Sometimes incident response can be boiled down to looking for a needle in a haystack. Sifting through logs (av, endpoint, firewall) is par for the course – but what if the logs you need don’t exist? It’s surprisingly frequent that we are called into an IT environment where log retention is painfully short, or no logs are kept at all. If we are looking for the first call out to a suspicious IP address, or a login on a system, that makes our work incredibly hard.
That’s why it’s necessary to see that you have sufficient logging in place across the board. We would recommend aiming for 1 year of log retention for all systems including endpoint and server system logs, antivirus alerts (especially AV alerts!), firewall / IDS / IPS logs, DLP systems, and more. Anything with security implications should be checked for sufficient logging. Keep in mind that certain regulations / compliance standards (DFARS, GLBA, HIPAA, SOC2, etc.) typically have data retention standards that must be met as a baseline, and that could be used to justify increasing logging capabilities.
Antivirus specifically should be treated with due care – not only should alerts be retained, but if possible, the quarantined threats (viruses, etc) should also be archived for later analysis if necessary.
Bonus Points: Enrich Logs with Sysmon
One often overlooked tool that is an easy win when it comes to tracing different types of activity across a windows network is Sysmon. Part of the Sysinternals suite, Sysmon augments Windows’ native event logging to add details that are very helpful when conducting investigations. Things like process creation, loading of drivers, and changes to registry are very commonly associated with malware, and the normal Windows event logs may miss critical details.
Fortunately Sysmon is free and there are several well known and vetted Sysmon configuration files (below) that you can use to add critical details to your logging that could prove invaluable later on. No excuses, do this now!
Bonus Points: Log Aggregation
The next step, after adding Sysmon, is to implement some sort of log aggregation to centralize log collection for easy storage and analysis. We won’t dive into much depth here, but this is possible through using Windows Event Collector or open source projects like Graylog2 or a full Elk Stack (we recommend Security Onion for those getting started here).
Having log aggregation allows for a centralized place to manage storage, and greatly enables anyone doing threat hunting across your network since all logs can be queried from a centralized location. Combine this with the level of detail provided by Sysmon and the results can be magical! All you really need to get going is a spare server, and storage space is cheaper than ever!
Get the Right People Involved
Another aspect of IR that is often overlooked is getting the right people involved. Often times the IT infrastructure of a SMB client is implemented, monitored, and maintained by a different mix of people that might or might not be on staff. Recently a client had one internal IT person, a separate MSP that managed their network, a third company that did network security monitoring, and a fourth company that was a vendor that operated some of their business critical infrastructure. In a case like this it is of critical importance that all of these people and partners are engaged to assist with incident response efforts.
When investigating a potentially ongoing threat time is critical. Having the ability to make changes to the network, track down credentials for a system, or whatever that might be, in a timely fashion is important. Another thing to consider is that there may be cases where certain staff members should not or could not be involved in a potential investigation due to circumstances or separation of duties requirements, so backup plans need to be in place.
Provide Access to ALL Systems and ALL Data
Lastly an area that we can run into issues is when our client only provides partial access or partial information to the IR team. This can really hamper IR efforts because IR is really supposed to be an open-book, white box activity where investigations can lead in many different directions. In some cases clients have done some investigation work prior to engaging our team, and that investigation can sometimes color results, or lead the investigation in a direction that isn’t correct.
Turning over as much (raw) data, and access to various systems, during the start of an investigation will help keep all routes open for review.
A Different Kind of Preparation
Generally people in infosec think of incidents as something to prepare for from an avoidance standpoint – “If we only had X in place that will prevent incident Y.” Incident preparedness means much more than that, however. You must also have the people (on staff or a trusted incident response partner like VDA Labs), processes, and technology in place to respond in an efficient and effective way. Hopefully the guidance provided in this post will help your organization proactively prepare!