AWS Loopholes with Temporary Credentials

Netskope

If you don’t already have a plan for managing temporary credentials, or tokens in your AWS environment, you need one ASAP. Temporary tokens can be a security nightmare because:

  • They provide nearly all the access of permanent access keys
  • Their creation and usage are much harder to track versus permanent access keys
  • They are hard to revoke if compromised
  • You can’t prevent them from being created

Temporary credentials are access keys with a time duration (15m up to 36 hours). Touted as a security feature in order to reduce management credential overhead by users, temporary credentials (or tokens) have been used to implement several common AWS features such as:

  • Assuming roles, including the passing of roles to services
  • Federated identities (i.e. single sign-on) 
  • Authentication of IoT devices.

However, there are serious drawbacks in the design and implementation of temporary credentials, namely that they:

  • Are unmanaged, providing little to no ability to track active temporary credentials
  • Have minimal logging, creating obfuscation and confusion during incidents
  • Include few options for revoking/remediating compromised temporary credentials

Temporary Credentials

Temporary credentials can be implicitly generated by AWS in the case of IoT device authentication or the passing/assuming of roles. Temporary credentials can also be explicitly created with the STS GetSessionToken API call (aws sts get-session-token in the CLI), e.g.,

aws sts get-session-token --duration-seconds 900
{
    "Credentials": {
        "SecretAccessKey": "rv1xIuMa6Frz1h5ojNW8BsguSGPiqkT7GNUEZZoL",
        "SessionToken": "FQoGZXIvYXdzEOz//////////wEaDFONA2EY59z3Fe83tiKrAdJBHPpt5dcAOEYIv0XjtOOMfPDDTgjRu8PVtMQrGswr3AqmIlPx2q9H8p6W67e4ypves23jBpHwC1SYDqZI6o0T+B4RILFAsNQfd8oOUkzsNZX5bJN0zPUXxwRQmW33c+Ysu01tBsNX+GeNpF+jVEzDbC8TiQ7N7/YvzlKYd1hYBsOlmCYS0NDn6s3+oH0QvoGfip71C5maxZNfNCVKf0I4jRC39Hm3JwF8tyitr6zwBQ==",
        "Expiration": "2019-08-10T18:40:49Z",
        "AccessKeyId": "ASIAXWLG7NXMYTB2ON7Z"
    }
}

In addition to the access key ID and access secret associated with any access key, a temporary access key includes an expiration date and a session token, which must be included in any API calls (along with the access key and secret).

An Attack Scenario

To highlight the challenges when temporary credentials are compromised and/or used within an attack, let’s look at a simplified attack scenario:

The Attacker:

  1. Gains initial access to the environment with stolen credentials, access key A (e.g. finding embedded keys in source code or on pastebin)
  2. Generates temporary credentials, temp token B, using the stolen credentials (access key A) with a call to STS GetSessionToken, for later use.
  3. Uses access key A to escalate privileges by assuming a role, MyBktRole, that has access to an S3 Bucket.
  4. Accesses sensitive data on the S3 Bucket
  5. Exfiltrates data from the S3 Bucket (e.g. S3 sync to another S3 bucket)

Defender Viewpoint

Let’s play the role of the Defender. Assume we have decent security controls and best practices in place with good people on staff in our Sec Ops and IR/IH groups.

  1. Detection: We’ve configured CloudTrail logging, CloudWatch alerts, and have detected suspicious behavior with the access and data exfiltration of the S3 bucket. This might be straight-forward event alarms based on our expected policy or we may have some correlation or anomaly detection in place.
  2. Investigate: We quickly trace through the logs and determine that access key A was used to assume a role in order to access the S3 Bucket. Looking at the assumed role, MyBktRole, we determine it was valid and not overprivileged, as it was created for certain authorized users (including the user who owns access key A) to allow controlled access to the S3 Bucket. After interviewing the user, we determine that the problem was a compromised service key.
  3. Mitigate/Remediate: We proceed to delete the user’s compromised key, regenerate new keys, ensure key rotation is enabled/checked, change their console password, review their settings and configuration, and require the user to go to security training.
  4. Additional Investigation: During the initial investigation, we notice the following CloudTrail event that was logged during step 2 of the Attack (generation of the temporary token). Should we be worried about this event? GetSessionToken and the returned access keys are immediately troubling. What should we do about it? Time to tell our manager the bad news…possible continued access/compromise and we’re researching.

Figure 1: CloudTrail Event for Creation of Temporary Token

More on Temporary Tokens:

Researching temporary tokens confirms that they essentially have the same exposure as permanent access keys, but with a few minor differences:

  • Expiration: Temporary tokens have an expiration, ranging from 15 minutes to 36 hours. Remember that CloudTrail latency (from API call to logging on S3) is at least 20 minutes, so overall we have a delayed response and could have a fairly long timeframe of secondary access. Temporary tokens generated by AWS in the case of passing roles to services like EC2 usually have shorter time frames such as an hour but they’re automatically refreshed, so an attacker who’s gained control of an EC2 instance only needs to refresh their tokens every hour.
  • API Access: Sessions using temporary tokens cannot create more temporary tokens. In fact, within the STS service, they can only invoke AssumeRole. They can use any service that the original user has privileges for. Given that the organization was compromised with a privilege escalation attack (AssumeRole), and that there are a plethora of techniques for escalation of privilege, this is not a large barrier to an attacker.

Defensive Techniques

Assessment: Our first thought may be to find out all information about the temporary token and possibly any other temporary tokens that have been created. Unfortunately, the Console does not show temporary tokens. The CLI/API also does not have any command to show any current or active temporary tokens. This may seem a bit like going to a gunfight blindfolded and without a knife.

Detection: For assessment of detection, we can look in the CloudTrail logs for events such as GetSessionToken. We should immediately update the CloudWatch configuration to alert on GetSessionToken. We can also search historical logs for GetSessionToken or any access key starting with “ASIA,” which is the naming convention to distinguish temporary tokens from permanent tokens (AKIA). [1] 

Figure 2: CloudTrail Event for Use of Temporary Token

Mitigation: There is no way to revoke temporary tokens. The console doesn’t even list them and the CLI/API do not have an STS action to list or revoke temporary tokens.

AWS recommends: 

  1. Changing/restricting the privileges (policies) for the user or role that created the token since the temporary token creates a session with the current privileges of the user/role at the time of use (not creation). [2]
  2. Deleting the user. [3]

We need to update our remediation playbook for compromised credentials, since it may not even be addressed. Analysis is needed to determine how the different options work in terms of affecting users or applications. Addressing one compromised user is not difficult, but temporary credentials exist for every AssumeRole, PassRole, and IoT device that authenticates.

There is a capability to revoke active sessions associated with a Role — it’s an additional tab in the console under roles. It sets a condition in the policy by checking timestamps of the generation time of the token [4]. That’ll help some. In the Attack scenario, we could revoke active sessions for MyBktRole.

Figure 3: AWS Console Tab for Revoke Active Sessions for a Role

  1. Prevention: However, we need to get out of the reactive fire drill mindset, and instead look to prevent and lockdown temporary credentials in the first place. Unfortunately, by design, we cannot restrict or prevent calls to GetSessionToken, as explained in the IAM User Guide [5]:

IAM users can also call GetSessionToken to create temporary security credentials. No permissions are required for a user to call GetSessionToken. The purpose of this operation is to authenticate the user using MFA. You cannot use policies to control authentication. This means that you cannot prevent IAM users from callingGetSessionToken to create temporary credentials.

For similar reasons, MFA cannot prevent calls to GetSessionToken. And even if we could, MFA would never be acceptable for non-human authentication e.g. IoT device authentication. The last thing we need is to require MFA before our vacuum cleaner cleans the house.

Since we can’t prevent the creation of temporary credentials, the revoking of active sessions for a role seems more like a band aid, since new temporary credentials can be regenerated as long as the Attacker has continued access. 

Although there is no simple prevention strategy, such as completely locking down the use of temporary credentials, we can look to practice least privilege when defining roles and policies, as well as carefully restricting which roles can be assumed or passed. These techniques will help prevent abuse even when temporary credentials are created or compromised.

Summary

You need to plan immediately for how to manage and respond to temporary token creation, use, and compromise. Consider the following areas when revising or creating your own plan:

  1. Prevention: You cannot prevent creation of temporary tokens, but you can:
    1. Ensure your access key policies are tightened (do not allow permanent access keys unless needed, if needed, ensure key rotation) in order to reduce chances of compromised credentials.
    2. Isolate temporary token usage in separate accounts, since cross-account access can be locked down and restricted
    3. Review use of roles in AssumeRole or PassRole and ensure they are have minimum privileges in order to reduce impact from compromised accounts
  2. Detection:
    1. Generate alarms for GetSessionToken in CloudWatch or your SIEM
    2. Generate alarms for the use of any temporary token (beginning with “ASIA”) in any situation where not expected.
    3. Ensure CloudTrail logging and alarm service (CloudWatch) is properly configured to catch all events and maintain the integrity of events (Object Lock, Versioning, LogFileValidation, restricted permissions)
  3. Remediation/Mitigation:
    1. Review/revise the remediation playbook for compromised temporary credentials to ensure effective mitigation of compromised tokens, using some combination of restricting roles or user policies, revoking active sessions for roles, or deleting users.
    2. Beyond mitigation, recovery from use of compromised temporary tokens is paramount. Treat this as seriously as a section of your business continuity planning, including testing. 
  4. Provisioning/Inventory
    1. If there are valid uses for temporary tokens, consider enforcing a process that records every temporary token into a database, so that authorized temporary tokens (and therefore, unauthorized tokens) are always known. If you have a custom application creating temporary tokens, it’s best to save tokens during provisioning by having wrapper code around GetSessionToken.
    2. However, for tracking AWS generated temporary tokens (IoT, AssumeRole), you have no choice but to parse CloudTrail logs and attempt to identify/persist authorized tokens after the fact. Both of these approaches are messy and require coding and maintenance.

Netskope provides support in preventing abuse with continuous security assessment checks, detection with breach detection rules, and custom auto-remediation commands.

This blog post is based on a detailed talk given at Defcon 27 on August 10, 2019. A link to the slide deck can be found here.

[1] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_access-keys-audit

[2] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_disable-perms.html

[3] https://aws.amazon.com/premiumsupport/knowledge-center/potential-account-compromise/

[4] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html
[5] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_enable-create.html