Imagine you’ve protected your production Google Cloud environment from compromised credentials, using MFA and a hardware security key. However, you find that your GCP environment has been breached through the hijacking of OAuth session tokens cached by gcloud access. Tokens were exfiltrated and used to invoke API calls from another host. The tokens were refreshed by the attacker and did not require MFA. Detecting the breach via Stackdriver was confusing, slowing incident response. And there are multiple confusing options to revoking the active OAuth sessions and most do not work, causing further delays in remediation.
In OAuth Token Hijacking in Google Cloud (GCP)—Part 1, we demonstrated the ease of several attack scenarios using hijacked OAuth tokens. In Part 2 of this blog, we will focus on concrete steps that you can take to reduce risk around detection, remediation, and prevention of OAuth token abuse. Let’s continue the discussion of the attack in Part 1 from the viewpoint of the defender.
Prevention
Two of the most effective measures at preventing or mitigating the abuse of compromised credentials are:
- Setting the expiration time for Google Cloud SDK sessions, which shrinks the window for compromise.
- Enforcing IP allow lists using access policies and VPC service controls, which mitigates the usage if session tokens have been compromised. This also helps improve detection.
Session Expiration
In the attack scenario in Part 1, the production environment did not have an expiration period set at all. This effectively set an infinite window for potential endpoint compromise. Let’s look at what can be configured in G Suite Admin to expire sessions
- Session duration: The Google cloud session duration in G Suite Admin > Security > Google Cloud session control should be set. It defaults to Session never expires and should instead be set to a time period between 1 and 24 hours. It is recommended that production environments be set to 1 hour and non-production environments between 1 and 8 hours.
- Re-authentication method: On the same screen, one specifies the re-authentication method after the session expires. You specify one of two choices: password or security key. This should not be set to a lower authentication strength than your primary authentication. Specifically, if you have a security key in use, this should be set to match (not Password!). Otherwise, you will weaken security for re-authentication flows.
IP Allow Lists
VPC Service Controls
IP allow lists should be implemented with VPC Service Controls found in Google Cloud Console > Security > VPC Service Controls, using an access level based on IP address defined in Google Cloud Console > Security > Access Context Manager. An IP allow list does not prevent token hijacking, but is a mitigating control if tokens are hijacked. It also allows us to improve our detection as we’ll see later when we discuss Detection.
When implemented, users attempting to call the specified APIs from a non-authorized source IP will get an access denied error:
Although the attacker can still utilize the OAuth tokens on the compromised host, this is still an effective way to mitigate the compromised tokens being exfiltrated and used on a different host.
Compute Instances/Metadata Proxy
A custom solution for IP allow lists Compute Instance tokens would be to automatically update allow lists as Compute VMs are launched. The Netflix security team describes several techniques including a proxy for the AWS metadata service, and something similar could be implemented on GCP. This will reduce risk from Compute instance compromise of OAuth tokens.
Auditing/Enforcement
It is a good idea to regularly audit and check that IP allow lists are implemented. Checking an IP allow list policy manually with the CLI might involve:
In Netskope’s Continuous Security Assessment product, a custom rule could be written similar to this (assuming naming is like the examples in this blog):
MFA
- MFA should be set and enforced in G Suite Admin > Security > 2-Step Verification for any Google Cloud admins. As we saw in the attack scenarios, it will not prevent reuse of hijacked OAuth tokens, and in fact, is not required unless the session expires. But when configured along with session expiration and re-authentication, it is an effective means to mitigating compromised passwords.
- Note: if your MFA is a software authenticator app, the re-authentication method after expiration will have to be Password, which is less secure, but is better than nothing.
- When using security hardware keys, make sure the re-authentication method matches and is set to security key.
Detection
Assuming we have OAuth token hijacking, as in Part 1, would we have detected anything suspicious? Unlikely. Detection of compromised credential use is difficult since the logged activity may appear to be normal user activity.
However, if we can implement IP allow lists, this will help improve detection when compromised credentials are used because we can detect attempts to use credentials from unapproved IP addresses.
To make this all clear, let’s take a closer look at Stackdriver logging, our centralized event logging service, as well as G Suite Audit logs for any recorded events based on API calls.
Improved Detection (IP Allow Lists)
If we use better prevention measures like IP allow lists, monitoring of logs can be done to check for authorization failures, which can help detect potential compromised token/credential attacks. False positives may occur if GCP administrators forget and attempt to use credentials from non-approved IPs, but this can still be an effective means to detecting that credentials (tokens) have been compromised.
Here is an example event from Stackdriver, showing a failed authorization attempt because the source IP did not match the allowed IPs specified in the VPC service control policy:
You should alert on any similar permission denied events for your VPC service control that implements your IP allow list as it may indicate a compromised credential has been exfiltrated and is being used on another host.
General logging
To complete our discussion, let’s review what is logged by Stackdriver for normal, successful access operations. Here’s a log entry related to the bucket access done in Part 1:
Note that we see a bucket access operation on bucket-foo-dev-mfa by user [email protected]. There is no indication of what is happening at the OAuth token level, and even if there were, it would be hard to determine if it was suspicious activity.
There are also Token Audit Logs in G Suite Admin, but they have the same challenge–any activity there does not necessarily identify suspicious activity.
Remediation
Multiple remediation options exist but are confusing because of the number of options and the differences between browser vs. API sessions and Google Cloud vs G Suite apps.
If you suspect an account has been compromised and that session tokens are being used, remediation needs to include two parts:
- We need to prevent future logins i.e. account access by the attacker
- We need to revoke/disable any current access by the attacker i.e. revoke any current OAuth session access and refresh tokens
Although there are many potential settings in the Google Cloud and G Suite Admin Consoles and APIs, here are the effective ones that work in all scenarios:
User accounts
Prevent future account access via password reset
User accounts can be disabled by resetting the password in G Suite Admin > Users:
Resetting the password will not revoke currently active sessions, which is a second, separate step described below.
How about deleting the user account? That would also disable future access by the attack, as well as revoke currently active sessions all in one action, but is not recommended as it’s a drastic measure, requiring a high-level of reprovisioning and reconfiguration.
Revoke current sessions via delete OAuth connected applications
The best (only) way to revoke current Google Cloud OAuth sessions is to:
- Delete the connected Google Cloud SDK OAuth application in G Suite Admin Console or via the corresponding G Suite SDK API call.
- The Console action and API call will immediately revoke all access and refresh tokens for that user.
In G Suite Admin > Users > user > Security > Connected applications:
Or via the G Suite Directory API:
Since these actions are on the G Suite Console/APIs, ensure access is granted to G Suite for Google Cloud admins.
Service accounts
Prevent future account access via deleting/recreating service accounts
Service accounts can be deleted in Google Cloud Console > IAM & Admin > Service Accounts
You might be wondering whether the drastic action of deleting the service account is required vs. disabling or deleting the API key itself. It is needed, because it is the only way to revoke the current session tokens. Deleting or disabling the API key under the service account will not revoke current session tokens.
Revoke current sessions via deleting/recreating service accounts
Deleting a service account will revoke current OAuth sessions, which is why it is the recommended action.
Note: Since the recommended action involves deleting and recreating a service account, this has large implications for recovery/downtime during an incident. All uses of the service account must be reconfigured, including compute instance VMs. This requires stopping the instance at a minimum, which would be recommended anyway in order to ensure current shell access to the VM by the attacker is revoked. This does account provisioning and compute instance configuration should be reviewed and automated with CLI/API scripts as much as possible to minimize downtime and reduce errors.
Other Remediation Actions
These remediation options do not work well in all cases and should be avoided. They are mentioned to ensure that their drawbacks are clearly understood since the number of options available can be confusing.
User accounts
Service accounts
Conclusion
Because the implementation of OAuth includes caching of session tokens, we’ve seen how easy it is for attackers to exploit this as another attack vector, once access to the GCP administrator’s machine is attained. Credential caches can be easily copied and used elsewhere, and will not require MFA even if configured.
Furthermore, we’ve seen how confusing it can be to prevent, detect, and remediate compromised credential situations due to the numerous options available in both G Suite and Google Cloud.
Despite these considerations, there are concrete measures that can help prevent abuse in the case of compromise:
