Using memory forensics effectively for Linux incident response (and threat hunting)

In a world where attacker dwell times are measured in months, improving cybersecurity incident response time is critical to protecting enterprises. Breaches expand in scope and severity over time, so responders must act quickly yet carefully to identify and remediate threats to minimize the impact to the business. In “Alexsey’s TTPs (Tactics, Techniques, and Procedures)” incident responder Chris McNab shares his work in dealing with compromises of the computing infrastructure within large technology companies. The attacks he describes often start with the compromise of Linux servers and the use of infected Linux systems to harvest credentials that give the attacker access to additional systems.

Whether attackers exploit weaknesses at the ‘human point’, where users interact with systems and critical business data, or penetrate networks by other means, organizations must be prepared to respond to a breach. This applies to any part of their computing infrastructure, including Linux servers in data centers or the cloud. Effective response requires an understanding of the tools available to reveal the location and nature of malicious code and other artifacts.

Memory forensics is an essential step of cyber incident response, both to preserve evidence in volatile memory, as well as to uncover malware and other attack-related artifacts from memory. In scenarios like the ones described by McNab, memory forensics can reveal keyloggers, credential stealers and tools used to clean up or hide evidence of a compromise, such as rootkits.

The screenshot above shows a task listing from a memory integrity scan of an Amazon EC2 instance. The highlighted task, while appearing normal to a surface-level inspection, in fact contains malicious injected code.

Despite the value it can offer, when it comes to security incidents in enterprise or cloud Linux server environments, memory forensics is often seen as too complicated or time-consuming to pursue. Some of the most common challenges faced by those attempting to implement Linux memory forensics at scale include:

• Inability to gather full memory dumps from large numbers of active servers with large amounts of memory: issues include blurring, storage, and transfer of GBs of RAM
• Lack of compatible metadata and memory analysis algorithms: these must match the kernel versions of deployed systems for memory forensics to be successful; community-maintained resources are often outdated or incomplete
• Ineffective manual anomaly detection: hoping to spot the evil in long lists of running processes or loaded modules is not realistic – highlighting suspicious aspects of a system must be automated for use at scale

To successfully use memory forensics for Linux incident response these challenges must be addressed. First and foremost, security teams should implement a solution that automates memory acquisition and analysis tasks, with broad support for different versions of Linux. Next, a Linux memory forensics solution should provide detailed reporting and alerting on potential malware found in memory. It should also be able to perform efficient remote memory analysis, scanning systems using targeted acquisition, which yields rapid results without the overhead of a full memory dump.

Ideally, a security team responsible for Linux infrastructure should implement a memory forensics solution that can quickly acquire and analyze memory across their entire Linux environment. A solution of this nature enables the team to use memory forensics not only during incident response, but also proactively to hunt for currently unknown and undetected threats.

Forcepoint Threat Protection for Linux (formerly Second Look®) is an enterprise- and cloud-ready solution that uses memory forensics to enable security teams to detect threats and respond to incidents on Linux systems. Automating the tasks of comprehensive memory acquisition and analysis, with detailed reporting and alerting, it has solved the Linux memory forensics problem for you, performing rapid memory integrity scans across even large-scale Linux environments. This allows security teams to do their jobs faster and more effectively, reducing dwell time and, most importantly, the cost and business impact of security incidents.

The screenshot above shows the malicious shared object file that was injected into the process in the first screenshot. The dropdown menu makes it easy to navigate among the wealth of information produced by the memory integrity scan.

Forcepoint offers a free demo environment that can perform memory integrity scans of any system with a public IP address. It will provide you with a first-hand evaluation based on your own, real-world systems. You’ll see results quickly, and gain a tool you can use to detect and respond to attacks on your Linux-based infrastructure like those described by McNab.

More Information:

• Forcepoint Security Labs Blog: Detecting Register-Hooking Linux Rootkits with Forcepoint Second Look
• Forcepoint Security Labs Blog: The Horse Pill Rootkit vs. Forcepoint Threat Protection for Linux
• Case Study: Global Trading Firm Detects Advanced Linux Threats with Forcepoint Threat Protection for Linux
• Splunkbase: Forcepoint Threat Protection for Linux (Second Look) - App for Splunk