February 20, 2017

Detecting register-hooking Linux rootkits with Forcepoint Second Look

Andrew Tappert

Sometimes old threats continue to remain relevant for a long period of time. The longevity of the x86 CPU architecture means that rootkits leveraging its features to achieve stealth on compromised systems may have a long shelf life and enable attackers to evade detection over an extended period. In this article, we look at "Subversive"; (https://github.com/falk3n/subversive), a Linux rootkit that uses x86 debug registers to hook the operating system kernel.

Despite the last change in Subversive’s Github repository being in 2011, it compiled and operated successfully in a modern environment, on the latest release of Red Hat Enterprise Linux (RHEL) in the Amazon cloud (AWS EC2). We show detection of the rootkit using the memory integrity and CPU register verification capabilities of Forcepoint Threat Protection for Linux (Second Look).


One of the major use cases for Forcepoint Threat Protection for Linux (Second Look) is verification of the integrity of the Linux kernel in memory on a target system, to detect kernel rootkits and other low-level system modifications made by malware. Static executable code and read-only data of the kernel are verified out-of-band with a dedicated memory analysis engine against published reference kernels. Dynamic aspects of the Linux kernel are also tested for integrity constraints, such as kernel function pointers referencing only previously verified executable code. However, operation (and integrity) of the kernel depends not only on the contents of memory, but also on the values in processor registers. Certain key registers, such as those involved in system call handling, interrupt handling, and debugging, are particular targets for attackers.

For example, the debugging capabilities built in to x86 processors can be used by attackers to modify operating system behavior and conceal their presence on a system. This technique has long been known, and was covered in detail in 2008 in Phrack (http://phrack.org/issues/65/8.html). With this technique being in the public domain for many years, checking for abuse of the CPU registers that control the debug functionality, as well as other key registers that can be abused by attackers, is an important step in ensuring integrity of Linux systems.

How does the Second Look memory forensics engine get access to these register values? When analyzing virtual machine snapshots, the virtualization system provides them. For example, VMware snapshot (.vmsn) and suspend (.vmss) files include not just the VM’s raw memory, but also a CPU metadata section. When analyzing systems using the Second Look SSH-based Memory Access Agent, register values are captured in memory during acquisition.


Forcepoint Second Look alerts for 'Subversive' rootkit

This screen capture shows alerts generated by the Second Look memory forensics engine based on an automated scan of a RHEL 7.3 EC2 instance infected with the “Subversive” rootkit.


Detection of “Subversive”

Whereas a traditional kernel rootkit hooks system calls by modifying entries in the kernel’s system call table, this rootkit performs the same job without modifying that data structure. Instead, the rootkit sets up a debug breakpoint in the kernel’s system call handling code (using processor debug registers), and overwrites code in the debug interrupt handler to gain control.  Versions of Threat Protection for Linux prior to 5.3 are capable of detecting this rootkit based on the kernel code modification, but with the new capability of register acquisition and verification introduced in Release 5.3, the additional alerts on the registers make it that much clearer what’s happening on the system and provide greater assurance that other rootkits in the future can be detected based on register-based hooking behavior.

In addition to the red “suspect register value” and “kernel text mismatch” alerts in the screenshot, there are also yellow alerts on the rootkit kernel module and the “subversive_ctl” user-mode tool that is used to control the rootkit (alert types “no reference module” and “unverified ELF”, respectively). Note that “subversive_ctl” was not running at the moment this target was scanned with Threat Protection for Linux (i.e., the executable file was not mapped in an active process at that time), but it was nevertheless analyzed and flagged. See the screen captures following this article for a closer look at the detected user-mode control tool.

The pale green alerts were “filtered”, using the product’s flexible JsonLogic-based alert filtering framework. Filters enable operators to prevent alerts on conditions which would normally be considered alarming, but in a particular case are expected and non-alerting, from being propagated to a SIEM, e.g. to the Forcepoint Threat Protection for Linux - App for Splunk. In this case, the alerts relate to normal and expected dynamic code generation by the Foreign Function Interface library (libffi) in the RHEL “tuned” service. Threat Protection for Linux ships with ready-to-use filters for such common cases, and enables security professionals to easily create and manage their own.

Release 5.3 now available

Forcepoint Threat Protection for Linux (Second Look) is a unique product that provides cloud-scale Linux memory forensics for incident response and intrusion detection. Threat Protection for Linux checks the integrity of the running Linux kernel and kernel modules, user processes, and executable files cached in memory on Linux systems. Upon detecting rootkits, malware, and other unknown or unauthorized software in memory, Threat Protection for Linux alerts security professionals and provides deep context for the investigation of Linux systems.

Release 5.3 features a number of advances, including new capabilities to detect stealthy malware, enhanced parallelization for faster scans of memory, an enhanced alert filtering system, and support for memory analysis of systems running Linux kernel versions up to 4.8 (including the latest releases of Amazon Linux, CentOS, Debian, Fedora, Oracle Linux, Red Hat Enterprise Linux, and Ubuntu). Threat Protection for Linux leverages a repository of page-level hashes for the executable code from hundreds of millions of Linux software packages, and a collection of nearly 20,000 reference kernels, to analyze a vast range of Linux systems and alert on unknown and unauthorized code in memory.

More information

For additional information on Forcepoint Threat Protection for Linux, please see:

About Forcepoint

Forcepoint is the leading user and data protection cybersecurity company, entrusted to safeguard organizations while driving digital transformation and growth. Our solutions adapt in real-time to how people interact with data, providing secure access while enabling employees to create value.