November 16, 2021

Zero-Day Protection by Catching Shellcode Encoders

Jenny Heino Principal Security Researcher
Antti Levomäki
Christian Jalio

The news is periodically being filled with headlines of another zero-day vulnerability in a common software component being utilized by attackers. As the name indicates, the vulnerability has been known, even to the security research community, for exactly zero days, which means that there's not much you can do to prevent an exploit - or is there?

An exploit

Let's start by taking a short look into what happens during a typical exploit scenario. Everything begins with a vulnerability in a software component on the victim machine, which the attacker is aware of. The attacker prepares an attack vector, be it a special kind of file, a certain pattern in network traffic, or something else, and somehow tricks the victim to execute this attack vector.

When the victim executes the attack vector, the vulnerability is triggered. In some cases, such as in a Denial of Service (DoS) attack, this may be enough: the system might become unresponsive or exponentially slow down. But in some cases, the vulnerability may enable the attacker to inject something additional on the victim's system that will be executed.


Shellcode is the general term used for the small pieces of code that are commonly injected to the victim system's memory during a successful exploit. The name originates from the historical fact that the initial shellcodes used to pop up a shell interface for the attacker. Nowadays, shellcode can do many other things as well, such as install a C&C beacon or a reverse backdoor on the system. But the term shellcode has established its place in common terminology.

In an attack framework, such as Metasploit, you typically first select the initial attack vector (i.e. the "exploit"). After selecting the exploit, you choose the payload you want to execute after the successful exploit. This payload is the shellcode.

Shellcode encoders

Most vulnerabilities do not allow injecting truly arbitrary bytes to memory. To reach the exploitable code, exploit payload may need to be formatted in a specific way - for example, it can contain no NULL bytes or only ASCII characters are allowed. To work around this issue, shellcode encoders are used.

A shellcode encoder encodes the shellcode into an acceptable form, such as full ASCII. The encoders themselves often do not attempt to evade detection by security vendors, such as AV's or IPS's. Their main functionality is in modifying the code into a format which can successfully be injected into the memory. These kinds of encoders are easy to identify even with simple string matching.

There are, however, some encoders that have been created specifically to be difficult to detect by security vendors. Polymorphic encoders are especially powerful. The shellcode that a polymorphic encoder produces will be different each time the encoder is run, which makes it impossible to identify by security systems that need an initial string match to work. An excellent example of a powerful polymorphic shellcode encoder is the legendary Shikata-Ga-Nai, which is a polymorphic XOR additive feedback encoder. Shikata-Ga-Nai has a reputation as a notoriously difficult encoder to detect by modern security products.

Shellcode stagers

Sometimes, the vulnerability that is going to be exploited only permits for a very short amount of code to be injected into the memory. In this case, the attacker cannot fit the complete shellcode into the attack payload. Instead, the attacker can inject something called a stager.

A stager is a very short piece of code which will fetch a larger piece of code to be run on the victim system. This makes it possible to get a larger shellcode into the exploited system. Exploit frameworks often implement different kinds of stagers as well.

Zero-Day detection

As mentioned previously, detecting a zero-day attack is a difficult task. But, in addition to the initial attack vector which utilizes the zero-day vulnerability, an exploit also usually consists of the encoded shellcode payload. Even though the initial exploit can vary, and even though the finally executed shellcode payload can vary, one part remains: the shellcode encoder.

This means that you do not necessarily need to identify the initial attack vector. You don't even need to know what the payload would do if it got executed. All you need to do is identify the different shellcode encoders. By identifying the shellcode encoders, you have a solid defense against zero-day attacks.

Detection capabilities in the Forcepoint Next-Generation Firewall

The in-house built, DFA-based fingerprinting in the Forcepoint Next-Generation Firewall enables fast, regular expression-based fingerprinting in any network stream. Because of this, it is possible to identify even the more complex shellcode encoders from any network traffic. This is noteworthy, as many security products are only able to perform this level of detection on files - not on the entire network level.

Detection for polymorphic encoders, such as Shikata-Ga-Nai, have been in the Forcepoint Next-Generation Firewall since 2010. Based on Forcepoint X-Labs research in 2021, a much wider coverage for shellcode encoders and stagers is now available for all types of network protocols in the Forcepoint Next-Generation Firewall. The X-Labs research team continues to monitor, improve, and add new identifications for the different shellcode payloads, shellcode encoders and shellcode stagers via the dynamic update packages.

To learn more about Forcepoint X-Labs, which includes the Forcepoint Next-Generation Firewall research team, click here.

About Forcepoint

Forcepoint is the leading user and data protection cybersecurity company, entrusted to safeguard organizations while driving digital transformation and growth. Our solutions adapt in real-time to how people interact with data, providing secure access while enabling employees to create value.

Jenny Heino

Principal Security Researcher

Jenny Heino is a Principal Security Researcher at Forcepoint, specializing in network security and based in Finland. Although she completed her master's studies in the field of Mathematical Logic, she quickly found her passion in vulnerability research and joined the research team for the...

Read more articles by Jenny Heino

Antti Levomäki

Antti Levomäki is a Senior Principal Security Researcher at Forcepoint, where his work focuses on researching network evasion techniques and writing testing tools. He holds a Master of Computer Science...

Read more articles by Antti Levomäki

Christian Jalio

Christian Jalio is the vulnerability research team lead for the Forcepoint Next Generation Firewall. An industry veteran cited in many CVE's, he recently transitioned from a position of senior principal to managing the NGFW vulnerability research team. Jalio holds a...

Read more articles by Christian Jalio