Case Study: Responding to an Attack in AWS
This is the second of a three-part blog series written by Cado and Invictus Incident Response, where we are investigating an incident that was discovered during an account audit of an Amazon environment.
Background
In part one, we walked through the analysis of AWS logs for account DevOps1-bpvbp
- which had been used to create an unknown account DevOps3
on 26 Oct 2022 at 21:29:03 UTC. Additionally the DevOps1-bpvbp
account had initiated a StartSession which provided a connection to EC2 instance i-08e56d6b6c0439daf
using Session Manager, at 21:29:53 UTC.
Investigative Questions
From here, we will continue the investigation analysis to answer the following questions:
- What actions were performed on the host?
- Are there other traces of suspicious activity?
Approach
As mentioned in the previous blog, we are focusing on the phases Acquisition and Processing & Analysis steps. Using the Cado platform, those phases look more like Acquisition & Processing and Analysis.
Acquisition & Processing
To start off using the Cado platform, investigations are compartmentalized as Projects, using the UI for the platform we created a project and acquired EC2 instance i-08e56d6b6c0439daf
.
A disk image is created from a snapshot of the volume attached to the instance i-08e56d6b6c0439daf
and automatically processed giving the analyst access to a super timeline and file content.
Analysis
From previous analysis, we know that the AWS Session Manager initiated session took place at 2022-10-26 21:29:03, so that will be the pivot point.
As this is a Linux system I will start with the secure
log file, the file shows that between Oct 26 21:29:53 and Oct 26 21:36:03 there was activity for the users ssm-user
and ec2-user. ssm-user
is the user account associated with SSM Session Manager and ec2-user
is the default user for Amazon Linux systems and the time of the account being accessed lines up with the log analysis.
secure
log shows session start and session close timestamps:
Oct 26 21:29:53 ip-10-0-2-54 useradd[32611]: new group: name=ssm-user, GID=1001
Oct 26 21:29:53 ip-10-0-2-54 useradd[32611]: new user: name=ssm-user, UID=1001, GID=1001, home=/home/ssm-user, shell=/bin/bash
Oct 26 21:30:08 ip-10-0-2-54 sudo: ssm-user : TTY=pts/0 ; PWD=/usr/bin ; USER=ec2-user ; COMMAND=/bin/bash
Oct 26 21:30:08 ip-10-0-2-54 sudo: pam_unix(sudo-i:session): session opened for user ec2-user by (uid=0)
Oct 26 21:36:03 ip-10-0-2-54 sudo: pam_unix(sudo-i:session): session closed for user ec2-user
Now we have a time window of interest.
Using the search function in the Cado platform, we want to see what activity took place on i-08e56d6b6c0439daf
between Oct 26 21:29:53 and Oct 26 21:36:03 so let’s add a timestamp range 2022-10-26 21:28:00 to 2022-10-26 21:38:00 to the search:
AWS SSM generates quite a lot of log events which results in a lot of noise so we removed those entries from the timeline search, reducing the number of events from 227 to 57:
Along with the secure
log, the audit
log tracks user logon activity, here we can see the ssm-user
is linked to audit entry USER_ACCT
which is triggered when a user-space user account is modified.
Our Investigative Questions are:
- What actions were performed on the host?
- Are there other traces of suspicious activity?
Timeline Analysis and file content can help answer these questions.
Between 21:29:53 and 21:36:04 we have events that show an AWS SSM interactive session had started; the associated ssm-user
switched user to the default Amazon Linux2 account ec2-user
which initiated a new bash
session. All of this takes place under the process ID 32627
directly linking the ssm-user
and ec2-user
accounts to the attacker activity. The ec2-user known-hosts
file was accessed and a file named staff.txt
was created in the ec2-user’s default directory.
Next step is to review the bash_history
files for ssm-user
and ec2-user
to identify any commands the attacker has executed.
The screenshot above is from using the Cado platform to view the contents of ssm-user
’s bash_history
file. The command seen within the file doesn’t have an associated timestamp, however, in our timeline there is log evidence that indicates that sudo
command was executed to switch user to ec2-user
during the attacker’s session.
The screenshot above shows the contents of ec2-user
’s bash_history
file. Again, the commands seen within the file don’t have timestamps, however in our timeline we did see the ec2-user
’s known_hosts
file being accessed, so there is a good indication the ssh
commands are linked to the attacker’s session. Also in our timeline we can see the file staff.txt
was created during the attacker’s session which is likely to be the result of the scp
command.
The other commands seen at the beginning and end of of the bash_history
file are also interesting:
macid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/) && vpcid=$(curl -sS http://169.254.169.254/latest/meta-data/network/interfaces/macs/${macid}/vpc-id)
aws ec2 describe-security-groups --region eu-west-1 --filters Name=vpc-id,Values=${vpcid} Name=egress.ip-permission.to-port,Values='22' --output text
The first command in the code box above, shows the Instance MetaData Service (IMDS)
being used to extract information regarding the Virtual Private Cloud (VPC) Id
. The VPC Id
is then used in conjunction with the aws
command line tool to provide detailed information of the security group that the instance belongs to, in particular looking for port 22 entries, port 22 is commonly associated with SSH. This could be seen as a method to perform network discovery to find other hosts to laterally move to.
The last two commands in the bash_history
file show that the aws
command line to create a S3 bucket and the file staff.txt
was copied to it. This could be seen as staging data for exfiltration:
aws s3 mb s3://eu-west-1-prod-data --region eu-west-1
aws s3 cp staff.txt s3://eu-west-1-prod-data --region eu-west-1
Our analysis of i-08e56d6b6c0439daf
has found that the scope of our investigation has extended to another system within the network, 10.0.8.25
, using the AWS console we identified that within the Security Group that IP address corresponds to an EC2 instance with ID i-020e722274233c7f2
and our Investigative Questions carry over to that host:
- What actions were performed on the host?
- Are there other traces of suspicious activity?Again we start the analysis again with the secure log file:
Again we start the analysis with the secure
log file:
Oct 26 21:31:55 ip-10-0-1-82 sshd[6867]: Accepted publickey for ec2-user from 10.0.2.54 port 45300 ssh2: RSA SHA256:C/JXEdpi4ui1exVUpnkKYCLDX8Xs9WYXdV1E+Mt7aAw
Oct 26 21:31:55 ip-10-0-1-82 sshd[6867]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
We can see the timings for the ssh
connection line up with the findings from our analysis of i-08e56d6b6c0439daf
.
Using the same time window and a filter to show LOGIN and USER from the audit
log along with the rest of the system and user events gives 53 events for further analysis.Due to the configuration of this system, user executed commands are saved to the audit
log, the commands are HEX encoded and the Cado platform automatically decodes them to the attacker executed the following commands:
-bash –login -c psql
psql -d prod_data_01 -c SELECT * FROM staff;
These commands show the attacker was very much interested in information from the staff
table in the postgres database. Returning to the timeline we can see that after the last command was executed /tmp/staff.txt
was created:
We now have two systems that have references to the file staff.txt
, as we are working with a disk image we can view the content of the file staff.txt
from both systems.
From i-020e722274233c7f2
:
and from i-08e56d6b6c0439daf
:
We can see that the file SHA256
hash and content are the same for both files on both systems.
The last section section of timeline analysis for i-020e722274233c7f2
shows the attacker’s ssh
session terminating followed by another ssh
session starting and terminating within the same minute:
We know from the ec2-user bash_history
file there was an scp
command executed after the SSH session which ties in with the timeline.
Let’s look at the investigative questions and where we stand:
What actions were performed on the host?
To recap, we’ve performed analysis of i-08e56d6b6c0439daf
and i-020e722274233c7f2
and identified that during the attacker’s AWS Session Manager session:
IMDS
andaws
command line tool were used to perform network discovery to identify other hosts for lateral movement;- connected to another host
i-020e722274233c7f2
for additional suspicious activity.
Are there other traces of suspicious activity?
- a database table was dumped to a file named
staff.txt
; - copied the database dump file
staff.txt
file fromi-020e722274233c7f2
toi-08e56d6b6c0439daf
; and - created a new s3 bucket and copied the database dump file
staff.txt
to it.
What happened to the database dump file staff.txt
and how did the attacker get access to the account DevOps1-bpvbp
? All will be revealed in the final part of this blog series.
Ready to perform your own cloud forensics investigation? Check out the 14-day free trial of the Cado platform.
More from the blog
View All PostsCase Study Continued: Responding to an Attack in AWS
February 10, 2023A New Perspective on Resource-Level Cloud Forensics
October 30, 2023Enhance Incident Response in GCP: Introducing Cado’s GCP Incident Response Playbook
December 21, 2023Subscribe to Our Blog
To stay up to date on the latest from Cado Security, subscribe to our blog today.