If an attacker compromises a Google Cloud Platform (GCP) user’s device, he can easily steal and abuse cached credentials, even if MFA is enabled.
In this blog post, we will demonstrate an attack in real Google Cloud environments, involving:
- Hijacking cached OAuth tokens stored on a GCP administrator’s client machine
- Reusing existing gcloud CLI sessions to gain access to multiple GCP environments,
- Showing that MFA does not apply to OAuth token refreshes for cached credentials (only the initial login)
- Discussing broader implications for service account keys
We will use realistically configured Google Cloud environments, as well as client machines where the initial compromise would happen. To demonstrate the attack, as well as defensive measures, we will alternate among the Google Cloud and G Suite Admin Consoles, the Google Cloud SDK command-line tools (gcloud and gsutil), and Stackdriver log events to demonstrate commands in the attack as well as administrative tasks for defensive measures.
This blog is from the attacker’s viewpoint, and later, in OAuth Token Hijacking in Google Cloud (GCP), Part 2, we will discuss what users can do to detect with Stackdriver Logging or G Suite Auditing Logs, remediate compromised tokens/access, and prevent such an attack in the first place.
OAuth
All authentication in Google Cloud uses the OAuth protocol underneath, regardless of whether you log on interactively via the browser or programmatically access GCP via the SDK. Here is a simplified, high-level view of the OAuth flow for programmatic access to GCP from an external GCP administrator’s machine (e.g. laptop):
- Access is requested (OAuth access token request). A GCP user typically sees this step when initially authenticating with the CLI, and a browser is launched to authenticate you, and you approve access. Part of requesting a token is to specify what scopes of permissions you are requesting–this is the prompt asking for your approval for access in the browser that is launched.
- An OAuth session access token and refresh token are created and returned. The session tokens expire after an hour and can be refreshed/regenerated by using the refresh token. These session and refresh tokens are cached.
- The access token is used for subsequent authentication for all API calls.
Token Hijacking for CLI (Bulk Credential Copy)
If we gain initial access to a laptop of a GCP administrator with normal user privileges, we can immediately access the user’s current gcloud sessions that include the cached OAuth access tokens:
The account, [email protected], has MFA enabled with a hardware security key. Let’s see what happens when we switch to that account.
We’ve switched accounts without trouble, but let’s see if the account works i.e. the credentials (tokens) are up-to-date and determine what we can access.
So, we were able to switch to the production account prod-mfa-hw.com and access a production bucket sensitive-bucket using the cached gcloud credentials (note: gsutil and gcloud share cached credentials). There was no prompt to reauthenticate when switching to the production account. In addition, MFA is enabled on this production account, but it has no effect on reauthentication.
The actual cached credentials are OAuth access and refresh tokens generated during the initial authentication (gcloud auth login). On Linux/macos, they are stored in ~/.config/gcloud, while on Windows they are stored in C:\Users\<username>\AppData\Roaming\gcloud.
The .db files are sqlite database files with a legacy directory containing text files per account. We’ll look at these files in more detail in the next scenario.
For now, let’s see how easy it is to copy these credentials off-machine and use them. Let’s just tar up the files, copy to another machine, and see what happens.
Let’s switch to the other machine my-attack-host-12345.com and check if the copied credentials work with gcloud.
It worked. So, all context/credentials have been transferred over to another machine by simply copying over all files in ~/.config/gcloud. Bucket access via gsutil also works on the attacker’s machine. The cached OAuth tokens are still valid. No reauthentication or MFA prompt is required from the new host.
Token Hijacking for API Calls
We just showed how we can easily copy the cached credentials en masse and access the user’s GCP environments. We can also pull out the OAuth tokens from the cache and use them directly to execute API calls instead of the CLI.
Let’s look back at the sqlite database files in ~/.config/gcloud. The file, access_tokens.db, contains the current OAuth access token, while credentials.db contains the refresh token, the OAuth client id/secret, scopes, and other information.
As you can see, the files are unencrypted and are easy to query. OAuth access tokens normally expire in 3600 seconds, after which, the refresh token must be used to obtain another access token. The credential.id_token.exp field indicates when the initial OAuth token was set to expire:
Since the default token duration is one hour, we know that the production environment prod-mfa-hw.com was first accessed from this machine back on June 17 at 09:12:03am PDT. And more than a month later, these cached credentials (OAuth refresh and access tokens) have not expired, are still valid, and can be used by an attacker. In other words, the time window for accessing the production environment from this host has been open for many weeks/months.
The refresh token will continue to be valid except under certain conditions (e.g. expiration is set, tokens explicitly revoked, or max limits hit) — these scenarios will be discussed later in Part 2 of this blog series.
So, how do we make use of these OAuth tokens directly?
We take the client id, secret, and refresh token from credentials.db as shown in the above query, and then generate a new, valid access token with an API call (part of the OAuth flow):
The response from the API call is a new OAuth access token, which can be used in any API call. Let’s execute a bucket listing call on the production bucket.
Other Token Hijacking Risk Areas
Service Accounts
Since OAuth is used for all Google Cloud authentication, when you add a service account key file locally during gcloud account configuration, gcloud will obtain OAuth tokens and store the tokens in the local access_tokens.db and credentials.db cache.
On a client machine outside of the GCP environment, stealing OAuth tokens for service accounts may not seem useful since the service account key file is likely stored under the user’s account anyways , and is more general and valuable for an attacker to compromise–the key file allows permanent access/reauthentication as it contains the private key secret.
However, there is an advantage to stealing the OAuth tokens generated for service accounts. Depending upon the remediation step taken by the victim, service account OAuth tokens may not be revoked, thereby granting up to an additional hour of access/persistence for the attacker:
- Deleting the API keys generated under the service account, will not revoke current OAuth session tokens, and the attacker can still execute the CLI/API calls until the token expires (up to one hour).
- Disabling the service account works and causes subsequent API calls using the OAuth token to fail. However, the token is not actually revoked, it still exists, so if the service account is re-enabled, the last OAuth access token will work again.
- Deleting the service account revokes the OAuth access token.
These remediation steps are discussed in more detail in Part 2.
Compute Instances
If an attacker gains shell access to compute instances and if the user has installed gcloud (Google Cloud SDK), then all of the above regarding token compromise applies.
Service accounts and their associated OAuth tokens on compute instances are another common attack vector. Compute instances can run as a service account identity, and in order to make it easier and more secure for users to run their compute application code and perform CLI tasks as that service account, a metadata service is provided that fetches a valid OAuth access token.
This is useful as the service account credentials (a key file) are not normally stored on a compute instance–the metadata service was created to avoid storing key files locally. But as we can see, once access is gained to the compute instance, the metadata service is easily queried to obtain a valid OAuth token. The expiration time for the access token is still a max of one hour. After the access token expires, no refresh token is needed to obtain a new access token, one just requires the metadata service. The metadata service is run locally on the compute instance and must be queried locally.
Cloud Shell
A special compute instance use case is the Google Cloud shell, which provides access to a Google-managed compute environment that includes the Google Cloud SDK pre-installed. Google Cloud shell has recently had root compromises as well as backdoor vulnerabilities. Once access is obtained to a Cloud shell and the underlying compute instance, both the ~/.config/glcoud credential cache and the metadata service can be exploited to hijack OAuth tokens.
Conclusion
In this post, we discussed 3 scenarios for hijacking OAuth tokens directly for subsequent use in the gcloud/gsutil CLI or in REST API calls:
- Bulk copy of the gcloud sqlite database files used by the CLIs
- Reuse of the OAuth tokens stored in the sqlite database, for use in API calls
- Stealing of the OAuth tokens returned from a compute instance’s metadata service for use in API calls
The scenarios rely upon several design or configuration aspects of OAuth tokens:
Accessibility
- Open sqlite database holding cached credentials/tokens
- OAuth tokens are cached and unencrypted, allowing easy access once the client endpoint has been exploited.
- Ease of copying unencrypted cached tokens to another host for exploitation
Persistence
- Tokens can have long or no expiration, allowing potentially long time windows for compromise.
- The attacker can easily refresh tokens, allowing persistence.
- Token refresh does not require MFA making it easy to maintain persistence, creating a false sense of security when MFA is enabled.
In our next blog post, OAuth Token Hijacking in Google Cloud (GCP), Part 2, we’ll look at the challenges in detecting, remediating, and preventing abuse from hijacked OAuth tokens.
