AWS EKS Incident Response

Organizations have increasingly adopted a container-based and serverless approach to allow for greater efficiency, agility, and cost savings. However, the ephemeral and dynamic nature of such resources makes investigations extremely challenging. 

This blog covers best practices and useful resources when responding to incidents in AWS EKS. 

If you’ve identified a potentially compromised container in EKS, there are two potential ways forward: 

• If the container is running on an underlying EC2, refer to the suggested steps for AWS EC2 incident response.
• If the container is running on Fargate, then collect any data required for later analysis before subsequently suspending it.

Community Resources

kube-forensics allows a cluster administrator to dump the current state of a running pod and all its containers so that security professionals can perform off-line forensic analysis.

Official AWS Resources

AWS provides advice on incident response and forensics in their EKS Best Practices documentation on Github. They also recently released new GuardDuty detections for EKS.

Cado Security Resources

At Cado, we published a playbook that covers best practices for investigating compromises in EKS environments. The Ultimate Guide to Docker and Kubernetes Incident Response explores how attackers are compromising containerized systems and best practices for conducting forensics and incident response of containerized applications including how to:

• Acquire Amazon EKS systems
• Export disks from Kubernetes containers on Windows with Hyper-V
• Conduct Kubernetes memory forensics
• And more…

In addition, to help security teams gain a better understanding of the types of data sources that can be captured in AWS, we published this GitHub repository that includes sample data taken from a compromised EKS system. We also published an associated talk which covers our own analysis.

Cado Platform

The Cado platform expedites incident response in the cloud by automating data capture, processing and analysis of cloud container-based and serverless resources. The Cado platform analyzes compromised EKS systems by automatically capturing:

• Logs in S3 from the AWS side (e.g. kubernetes authentication).
• A full copy of a container from within the container itself.
• A full copy of the underlying EC2 that the container is running on, which also includes a number of logs from the host operating system as well as versioned files from the container due to how the overlay2 filesystem works.

Interested in performing your own AWS EKS investigation using the Cado platform? Check out our 14-day free trial.