Friday, November 29, 2019

Changes to the AWS EC2 Instance Metadata Service (IMDS) around the recent Capital One hack

Captial One Bank (1) and 30 different organizations were hacked around end of July, I have written a blog (1) around the same time on how to recreate the hack in your own AWS account and also a few mitigations around the same. Now, AWS has made a few changes to the AWS EC2 Instance Metadata Service (IMDS) around the same (1, 2). Here (Security best practices for the Amazon EC2 instance metadata service) is the AWS re:Invent 2019 session around the same.

The old/existing approach is called IMDSv1 and the new one IMDSv2. Although IMDSv1 solves a few problems like not storing the access keys on the EC2, it bought its own headaches which lead to the hacks. Earlier the access keys for a IAM Role can be got using the below command. But, with the new IMDSv2 the same command would lead to `401 - Unauthorized` error after enabling IMDSv2. This would block SSRF and other such attacks.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod

Lets try to see it in action with the sequence of steps.

Step 1: Create an Ubuntu EC2 (t2.micro) and login to it via Putty.

Step 2: Create a role (Role4EC2-MetaDataMod) with the below JSON Policy (MetaDataModPolicy). Attach the role to the EC2.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:ModifyInstanceMetadataOptions",
            "Resource": "*"
        }
    ]
}

Step 3: Execute the below commands on the EC2 instance to configure the AWS CLI.

sudo apt-get update; sudo apt-get install -y python3-pip
pip3 install awscli --upgrade
export PATH="$PATH:/home/ubuntu/.local/bin/"

Configure the AWS CLI, make sure to provide only the region (us-east-1 or some other) and rest of them as blank.

aws configure

Step 4: Execute the below command on the EC2 instance to get the access keys associated with the IAM Role. The access keys would be displayed in the console.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod

Step 5: Turn off IMDSv1 executing the below command on the EC2 instance, by default both (IMDSv1 and IMDSv2) of them are turned on. Make sure to replace the EC2 instance-id in the below command. With this IMDSv1 is disabled and IMDSv2 is enabled. If IMDSv1 and IMDSv2 both should be enabled for the sake of compatibility with the existing applications the http-token can be set to optional. More on the command syntax and the documentation here.

aws ec2 modify-instance-metadata-options --instance-id i-053cb17f19ca95067 --profile default --http-endpoint enabled --http-token required

Step 6: Now the same command from Step 4 leads to `401 - Unauthorized` error message. as shown in the below screen.


Step 7: With the IMDSv2 a session token has to be got via the HTTP PUT method and then the token used to retrieve the access keys or in-fact with any of the EC2 Instance Metadata Service.

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
echo $TOKEN
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod -H "X-aws-ec2-metadata-token: $TOKEN"

Now the access keys can be got as shown in the below screen.


How IMDSv2 would prevent the hack from happening?


IMDSv2 would not prevent hacks like the Capital One hack from happening, but it addresses many misconfigurations and application bugs to some extent. There might be still a probability that a WAF is not configured properly to block the HTTP PUT requests with a bug in the application code to get the IMDS token and access the EC2 Metadata Service form outside the EC2.

For those who are hosting applications on AWS, would recommend enabling IMDSv2 and disabling IMDSv1 as mentioned here (1, 2, 3). But, before making the changes make sure the applications on AWS EC2 are compatible with IMDSv2. There won't be any changes required for any applications using the AWS SDK as this internally gets the token and then accesses the EC2 Instance Metadata Service using the token. But, if your application accesses the EC2 Instance Metadata Service using the HTTP endpoint as in the case of Step 4, it would require changes to the code to get the token.

This is a nice step from AWS, but it took more than 100 days for them to come up with a solution to block more such attacks. GCP has also the instance metadata service (1), not really sure if the same vulnerability is in GCP also and how GCP handles it. If you are familiar with how GCP tackles it, please let me know in the comments section and I will update this blog.

Along, the same lines AWS also introduced managed WAF rules (1) to avoid similar attacks.

1 comment:

  1. The response packet from PUT request has a default TTL of 1. So that packet won't ever leave the node.

    ReplyDelete