Imagine you’ve protected your production Google Cloud environment from compromised credentials, using MFA and a hardware security key. However, you find that your GCP environment has been breached through the hijacking of OAuth session tokens cached by gcloud access. Tokens were exfiltrated and used to invoke API calls from another host. The tokens were refreshed by the attacker and did not require MFA. Detecting the breach via Stackdriver was confusing, slowing incident response. And there are multiple confusing options to revoking the active OAuth sessions and most do not work, causing further delays in remediation.
In OAuth Token Hijacking in Google Cloud (GCP)—Part 1, we demonstrated the ease of several attack scenarios using hijacked OAuth tokens. In Part 2 of this blog, we will focus on concrete steps that you can take to reduce risk around detection, remediation, and prevention of OAuth token abuse. Let’s continue the discussion of the attack in Part 1 from the viewpoint of the defender.
Prevention
Two of the most effective measures at preventing or mitigating the abuse of compromised credentials are:
- Setting the expiration time for Google Cloud SDK sessions, which shrinks the window for compromise.
- Enforcing IP allow lists using access policies and VPC service controls, which mitigates the usage if session tokens have been compromised. This also helps improve detection.
Session Expiration
In the attack scenario in Part 1, the production environment did not have an expiration period set at all. This effectively set an infinite window for potential endpoint compromise. Let’s look at what can be configured in G Suite Admin to expire sessions
- Session duration: The Google cloud session duration in G Suite Admin > Security > Google Cloud session control should be set. It defaults to Session never expires and should instead be set to a time period between 1 and 24 hours. It is recommended that production environments be set to 1 hour and non-production environments between 1 and 8 hours.
- Re-authentication method: On the same screen, one specifies the re-authentication method after the session expires. You specify one of two choices: password or security key. This should not be set to a lower authentication strength than your primary authentication. Specifically, if you have a security key in use, this should be set to match (not Password!). Otherwise, you will weaken security for re-authentication flows.
IP Allow Lists
VPC Service Controls
IP allow lists should be implemented with VPC Service Controls found in Google Cloud Console > Security > VPC Service Controls, using an access level based on IP address defined in Google Cloud Console > Security > Access Context Manager. An IP allow list does not prevent token hijacking, but is a mitigating control if tokens are hijacked. It also allows us to improve our detection as we’ll see later when we discuss Detection.
When implemented, users attempting to call the specified APIs from a non-authorized source IP will get an access denied error:
Although the attacker can still utilize the OAuth tokens on the compromised host, this is still an effective way to mitigate the compromised tokens being exfiltrated and used on a different host.
Compute Instances/Metadata Proxy
A custom solution for IP allow lists Compute Instance tokens would be to automatically update allow lists as Compute VMs are launched. The Netflix security team describes several techniques including a proxy for the AWS metadata service, and something similar could be implemented on GCP. This will reduce risk from Compute instance compromise of OAuth tokens.
Auditing/Enforcement
It is a good idea to regularly audit and check that IP allow lists are implemented. Checking an IP allow list policy manually with the CLI might involve:
In Netskope’s Continuous Security Assessment product, a custom rule could be written similar to this (assuming naming is like the examples in this blog):
MFA
- MFA should be set and enforced in G Suite Admin > Security > 2-Step Verification for any Google Cloud admins. As we saw in the attack scenarios, it will not prevent reuse of hijacked OAuth tokens, and in fact, is not required unless the session expires. But when configured along with session expiration and re-authentication, it is an effective means to mitigating compromised passwords.