The Horse Pill Rootkit vs. Forcepoint Threat Protection for Linux

The Horse Pill rootkit was presented at Black Hat 2016 by Michael Leibowitz, a security engineer and member of the Red Team at Intel. Horse Pill is a proof-of-concept Linux rootkit that demonstrates two interesting techniques: 1. infecting systems via the initial ramdisk, and 2. deceiving system owners using container primitives. In this article we explore those techniques and how our product, Forcepoint Threat Protection for Linux, fares against them.

Initrd Infection

The initial ramdisk, or initrd, is a compressed archive containing files needed by a Linux system early in the boot process. The initrd is loaded into memory, alongside the compressed kernel image, by the bootloader. Here is a listing of the initial ramdisk and corresponding compressed kernel image file on an Ubuntu 16.04 system:

-rw-r--r-- 1 root root  35M Nov 10 01:05 /boot/initrd.img-4.4.0-47-generic
-rw------- 1 root root 6.8M Oct 26 18:27 /boot/vmlinuz-4.4.0-47-generic

The initrd is dynamically generated on each system. You can see this by comparing the timestamps above. The kernel was created October 26, but it was not installed on this system until November 10, and that’s when the initrd was created. The reason it is dynamically created is usually so that kernel modules that are needed for a specific environment (hardware, filesystems, etc.) can be included.  Some system software updates may trigger the initrd to be rebuilt in order for updated components to be included. There will be a distinct initrd for every kernel available to the system.

The Ubuntu 16.04 initrd is a gzipped CPIO archive:

/boot/initrd.img-4.4.0-47-generic: gzip compressed data, last modified: Thu Nov 10 06:05:45 2016, from Unix

$ gunzip --to-stdout /boot/initrd.img-4.4.0-47-generic | file -
/dev/stdin: ASCII cpio archive (SVR4 with no CRC)

Within the initrd archive we find executables, configuration files, kernel modules, firmware, shared libraries, scripts, and more.

$ gunzip --to-stdout /boot/initrd.img-4.4.0-47-generic | cpio -tv
-rwxr-xr-x   1 root     root          976 Apr 28  2016 bin/dmesg
-rwxr-xr-x   1 root     root         3936 Apr 28  2016 bin/dd
-rwxr-xr-x   1 root     root         4872 Apr 28  2016 bin/run-init
...

There are over 1000 files in this initrd. At boot, the kernel runs the “init” script from the initrd, which in turn executes the “run-init” binary from the initrd to start the initial user-mode process of the system.  The “run-init” program is what Horse Pill infects. The modified run-init that Horse Pill puts in the initrd enables it to gain control of execution early in the boot process. Here’s what the initrd contents look like after infection (note the change in size and timestamp of the run-init binary):

$ gunzip --to-stdout /boot/initrd.img-4.4.0-47-generic | cpio -tv
-rwxr-xr-x   1 root     root          976 Apr 28  2016 bin/dmesg
-rwxr-xr-x   1 root     root         3936 Apr 28  2016 bin/dd
-rwxr-xr-x   1 root     root       164216 Nov 14 12:13 bin/run-init
...

The Horse Pill rootkit replaces one file in the initrd, and thereby gains control over the system at boot time. Because the initrd is dynamically generated, and there are no integrity checks on it, it is unlikely that such a change will be noticed through casual observation.

Container-based Deception

When the Horse Pill rootkit gains execution via the initrd infection described above, it places the whole system into a newly created container (strictly speaking, “set of Linux kernel namespaces”). Let’s call that the “contained system”. It also launches a backdoor process, outside that container. Everything else on the system, including the systemd init process, and all the regularly expected system services and applications, are running inside the contained system.  Horse Pill’s run-init process also creates processes and renames them to appear to be kernel threads. The users and administrators who are “victims” of the rootkit can’t see the rootkit and backdoor processes running outside the contained system. The system appears pretty much normal from within it.

The remote network access backdoor used by the Horse Pill rootkit is a tool called dnscat2. To interact with the victim system, we run a dnscat2 server on another system. Via the dnscat2 server we can get a shell on the victim system which is running outside the contained system. Here’s a comparison of the view from within and without.

 is used for processes that make up the contained system, which is a process tree (corresponding to the contained system’s PID namespace) rooted at the systemd init process that is PID 1 within the contained system and PID 192 in the host environment.

 is used for fake kernel threads.

 is used for real kernel threads.

 is used for rootkit and backdoor processes.

From INSIDE the contained system, with the pstree command one sees all the processes and services running in the contained system, plus somewhat-realistic-looking (but actually fake) kernel threads:

From OUTSIDE (i.e. in a shell obtained via the dnscat2 server), one sees rootkit and backdoor process, the contained system, and the fake kernel threads:

But guess what… pstree does not normally list kernel threads!  On a normal, uninfected system one would not see kernel threads with pstree as we do in both the inside and outside views above.

With the ps command, from INSIDE one sees the contained system and somewhat-realistic-looking fake kernel threads:

From OUTSIDE with ps one sees real kernel threads, fake kernel threads, rootkit and backdoor processes, and the contained system:

Container technology hides the remote access backdoor (the dnscat process and its children) from users and processes within the contained system. In the “PID namespace” of the container, the systemd init process appears as PID 1, as normal, whereas in fact Horse Pill’s modified run-init is the actual PID 1 of the system and system init has PID 192. The backdoor process, because it is not part of the contained system’s PID namespace, is not visible there.

To attempt to maintain the illusion of normality, Horse Pill presents kernel threads inside the container, using a more old-fashioned kind of deception: process renaming.  The fake kernel threads are actually user-mode processes, set up to appear like kernel threads. This deception is inevitably imperfect. Aside from unexpectedly appearing in pstree output as mentioned above, note the “VSZ” and “RSS” (virtual and physical memory usage) of PID 2 (the kthreadd kernel thread) in the inside and outside views. In the inside view, where PID 2 is a fake kernel thread, these are non-zero.  Real kernel threads always have zero VSZ and RSS, since real kernel threads don’t have an “mm_struct” that defines the virtual address space for a user-mode process.

Would Horse Pill be better off not trying to fake the kernel threads? Is it more likely that someone would notice them missing, or that someone would notice that they don’t look quite right? Or is there some other better way to get them to appear normal-looking from within the contained system? Or is it just the right amount of stealth and deception in the absence of automated detection capabilities?

Aside from the anomalies one may notice regarding the faked kernel threads, as we have illustrated above, other avenues for detection of Horse Pill are discussed in the Black Hat presentation, including checking ns (“namespace”) links in the proc filesystem, and via the Linux audit subsystem.

Horse Pill vs. Forcepoint Threat Protection for Linux

Forcepoint Threat Protection for Linux (Second Look) is a security product that performs enterprise-scale Linux memory forensics. It checks the integrity of the running kernel, processes, and executable code cached in memory on Linux systems. It alerts users to rootkits, malware, and other unknown or unauthorized software in memory. Due to its integrity-based approach, it is not reliant on signatures or prior knowledge of specific threats. And due to being based on memory forensics, it gets what is essentially the “outside” perspective discussed above: it can see both the contained system and the rootkit and backdoor processes in memory.

When analyzing a system infected with the Horse Pill rootkit, we produce four different types of alerts: kernel_thread_masquerade, unlinked_executable, unverified_elf, and unverified_executable_mapping. All four alert types are visible in the screenshot below, of a “task details” page from the product’s web interface. It is perhaps worth noting that all this alerting was produced even with versions of the product that were available prior to release of the Horse Pill rootkit – we didn’t need to make any additions or adjustments to the product in order to detect even this novel rootkit, as shown here.

The kernel_thread_masquerade alert is produced when we detect a user-mode process that may be attempting to pretend to be a kernel thread. This is a technique long used by attackers to attempt to blend in on Linux systems, under the assumption that most administrators glance right over any kernel threads, whose names appear in square brackets in ps output. For example, the “mimic” tool for covert execution on Linux uses a default of “[kworker/0:0]” to hide root processes (see https://github.com/emptymonkey/mimic).

The unlinked_executable alert is produced when a process is running from a deleted executable file. This is most often an indication that a program has been updated (overwritten with a new version), while the old version remains running.  But it can also be caused by an attacker removing a tool after he runs it. In this case, the alert is produced because this process is running from a binary in the initrd, which is no longer accessible now that the system has booted. 

The unverified_elf alert is produced when the product detects an ELF binary (executable or shared object) cached in memory, that is not known to it (i.e. one that is not present in the reference data set that the product uses to verify code in memory). In this case, the ELF in question is the one the Horse Pill rootkit injected into the initrd. That is of course not present in the cloud-hosted reference data repository that tracks stock software from Linux distribution vendor packages, nor should it be in any user’s reference data for custom and third-party software deployed in their environment.

Finally, the unverified_executable_mapping alert is produced when an unverified ELF is found mapped into the address space of a running process. In this case that is particularly alerting because this process is ostensibly a kernel thread and would therefore not be expected to have any mappings of its own.

Not only do we alert on Horse Pill’s fake kernel threads, but also on the rootkit and backdoor processes. For example, below we show the “cached file details” page for the dnscat binary that was embedded as a payload in the infected binary in the initrd.

Blog contributors: Andrew Tappert, Tom O'Connor

References

Links to more information on the Horse Pill rootkit:

Project web site: http://www.pill.horse/

Project code on Github: https://github.com/r00tkillah/HORSEPILL

Black Hat conference presentation (PDF): https://www.blackhat.com/docs/us-16/materials/us-16-Leibowitz-Horse-Pill-A-New-Type-Of-Linux-Rootkit.pdf

Black Hat conference presentation (video): https://www.youtube.com/watch?v=wgbwmjtxZ5I