Netskope named a Leader in the 2022 Gartner® Magic Quadrant™ for Security Service Edge. Get the Report.

  • Security Service Edge Products

    Protect against advanced and cloud-enabled threats and safeguard data across all vectors.

  • Borderless SD-WAN

    Confidently provide secure, high-performance access to every remote user, device, site, and cloud.

  • Platform

    Unrivaled visibility and real-time data and threat protection on the world's largest security private cloud.

Netskope Named a Leader in the 2022 Gartner Magic Quadrant™ for SSE Report

Get the report Go to Products Overview
Netskope gartner mq 2022 sse leader

Gartner® Quick Answer: How Does Netskope’s Acquisition of Infiot Impact SD-WAN, SASE, and SSE Projects?

Get the report
Quick Answer: How Does Netskope’s Acquisition of Infiot Impact SD-WAN, SASE and SSE Projects?

Netskope delivers a modern cloud security stack, with unified capabilities for data and threat protection, plus secure private access.

Explore our platform
Birds eye view metropolitan city

Make the move to market-leading cloud security services with minimal latency and high reliability.

Learn more
Lighted highway through mountainside switchbacks

Prevent threats that often evade other security solutions using a single-pass SSE framework.

Learn more
Lighting storm over metropolitan area

Zero trust solutions for SSE and SASE deployments

Learn more
Boat driving through open sea

Netskope enables a safe, cloud-smart, and fast journey to adopt cloud services, apps, and public cloud infrastructure.

Learn more
Wind turbines along cliffside
  • Our Customers

    Netskope serves more than 2,000 customers worldwide including more than 25 of the Fortune 100

  • Customer Solutions

    We are here for you and with you every step of the way, ensuring your success with Netskope.

  • Training and Certification

    Netskope training will help you become a cloud security expert.

We help our customers to be Ready for Anything

See our Customers
Woman smiling with glasses looking out window

Netskope’s talented and experienced Professional Services team provides a prescriptive approach to your successful implementation.

Learn more
Netskope Professional Services

Secure your digital transformation journey and make the most of your cloud, web, and private applications with Netskope training.

Learn more
Group of young professionals working
  • Resources

    Learn more about how Netskope can help you secure your journey to the cloud.

  • Blog

    Learn how Netskope enables security and networking transformation through security service edge (SSE).

  • Events & Workshops

    Stay ahead of the latest security trends and connect with your peers.

  • Security Defined

    Everything you need to know in our cybersecurity encyclopedia.

Security Visionaries Podcast

Episode 15: Building Permanent Security Awareness

Play the podcast
Black man sitting in conference meeting

Read the latest on how Netskope can enable the Zero Trust and SASE journey through security service edge (SSE) capabilities.

Read the blog
Sunrise and cloudy sky

SASE Week

Netskope is positioned to help you begin your journey and discover where Security, Networking, and Zero Trust fit in the SASE world.

Learn more
SASE Week

What is Security Service Edge?

Explore the security side of SASE, the future of network and protection in the cloud.

Learn more
Four-way roundabout
  • Company

    We help you stay ahead of cloud, data, and network security challenges.

  • Why Netskope

    Cloud transformation and work from anywhere have changed how security needs to work.

  • Leadership

    Our leadership team is fiercely committed to doing everything it takes to make our customers successful.

  • Partners

    We partner with security leaders to help you secure your journey to the cloud.

Netskope enables the future of work.

Find out more
Curvy road through wooded area

Netskope is redefining cloud, data, and network security to help organizations apply Zero Trust principles to protect data.

Learn more
Switchback road atop a cliffside

Thinkers, builders, dreamers, innovators. Together, we deliver cutting-edge cloud security solutions to help our customers protect their data and people.

Meet our team
Group of hikers scaling a snowy mountain

Netskope’s partner-centric go-to-market strategy enables our partners to maximize their growth and profitability while transforming enterprise security.

Learn more
Group of diverse young professionals smiling

GCP OAuth Token Hijacking in Google Cloud—Part 2

Aug 25 2020

Imagine you’ve protected your production Google Cloud environment from compromised credentials, using MFA and a hardware security key. However, you find that your GCP environment has been breached through the hijacking of OAuth session tokens cached by gcloud access. Tokens were exfiltrated and used to invoke API calls from another host. The tokens were refreshed by the attacker and did not require MFA. Detecting the breach via Stackdriver was confusing, slowing incident response. And there are multiple confusing options to revoking the active OAuth sessions and most do not work, causing further delays in remediation.

In OAuth Token Hijacking in Google Cloud (GCP)—Part 1, we demonstrated the ease of several attack scenarios using hijacked OAuth tokens. In Part 2 of this blog, we will focus on concrete steps that you can take to reduce risk around detection, remediation, and prevention of OAuth token abuse. Let’s continue the discussion of the attack in Part 1 from the viewpoint of the defender.

Prevention

Two of the most effective measures at preventing or mitigating the abuse of compromised credentials are:

  • Setting the expiration time for Google Cloud SDK sessions, which shrinks the window for compromise.
  • Enforcing IP allow lists using access policies and VPC service controls, which mitigates the usage if session tokens have been compromised. This also helps improve detection.

Session Expiration

In the attack scenario in Part 1, the production environment did not have an expiration period set at all. This effectively set an infinite window for potential endpoint compromise. Let’s look at what can be configured in G Suite Admin to expire sessions

  • Session duration: The Google cloud session duration in G Suite Admin > Security > Google Cloud session control should be set. It defaults to Session never expires and should instead be set to a time period between 1 and 24 hours. It is recommended that production environments be set to 1 hour and non-production environments between 1 and 8 hours.
Screenshot showing Google Cloud session control
  • Re-authentication method: On the same screen, one specifies the re-authentication method after the session expires. You specify one of two choices: password or security key. This should not be set to a lower authentication strength than your primary authentication. Specifically, if you have a security key in use, this should be set to match (not Password!). Otherwise, you will weaken security for re-authentication flows.

IP Allow Lists

VPC Service Controls

IP allow lists should be implemented with VPC Service Controls found in Google Cloud Console > Security > VPC Service Controls, using an access level based on IP address defined in Google Cloud Console > Security > Access Context Manager. An IP allow list does not prevent token hijacking, but is a mitigating control if tokens are hijacked. It also allows us to improve our detection as we’ll see later when we discuss Detection.

Screenshot of Access Context Manager
Screenshot of VPC Service Controls

When implemented, users attempting to call the specified APIs from a non-authorized source IP will get an access denied error:

Example of access denied error

Although the attacker can still utilize the OAuth tokens on the compromised host, this is still an effective way to mitigate the compromised tokens being exfiltrated and used on a different host. 

Compute Instances/Metadata Proxy

A custom solution for IP allow lists Compute Instance tokens would be to automatically update allow lists as Compute VMs are launched. The Netflix security team describes several techniques including a proxy for the AWS metadata service, and something similar could be implemented on GCP. This will reduce risk from Compute instance compromise of OAuth tokens.

Auditing/Enforcement

It is a good idea to regularly audit and check that IP allow lists are implemented. Checking an IP allow list policy manually with the CLI might involve:

Example of checking an IP allow list policy manually

In Netskope’s Continuous Security Assessment product, a custom rule could be written similar to this (assuming naming is like the examples in this blog):

Example of a custom rule written for Netskope's Continuous Security Assessment

MFA

  • MFA should be set and enforced in G Suite Admin > Security > 2-Step Verification for any Google Cloud admins. As we saw in the attack scenarios, it will not prevent reuse of hijacked OAuth tokens, and in fact, is not required unless the session expires. But when configured along with session expiration and re-authentication, it is an effective means to mitigating compromised passwords.
Screenshot showing 2-factor verification
  • Note: if your MFA is a software authenticator app, the re-authentication method after expiration will have to be Password, which is less secure, but is better than nothing.
  • When using security hardware keys, make sure the re-authentication method matches and is set to security key.

Detection

Assuming we have OAuth token hijacking, as in Part 1, would we have detected anything suspicious? Unlikely. Detection of compromised credential use is difficult since the logged activity may appear to be normal user activity. 

However, if we can implement IP allow lists, this will help improve detection when compromised credentials are used because we can detect attempts to use credentials from unapproved IP addresses.

To make this all clear, let’s take a closer look at Stackdriver logging, our centralized event logging service, as well as G Suite Audit logs for any recorded events based on API calls.

Improved Detection (IP Allow Lists)

If we use better prevention measures like IP allow lists, monitoring of logs can be done to check for authorization failures, which can help detect potential compromised token/credential attacks. False positives may occur if GCP administrators forget and attempt to use credentials from non-approved IPs, but this can still be an effective means to detecting that credentials (tokens) have been compromised.

Here is an example event from Stackdriver, showing a failed authorization attempt because the source IP did not match the allowed IPs specified in the VPC service control policy:

Example event from Stackdriver showing a failed authorization attempt because the source IP allow list did not match the access level in the policy

You should alert on any similar permission denied events for your VPC service control that implements your IP allow list as it may indicate a compromised credential has been exfiltrated and is being used on another host.

General logging

To complete our discussion, let’s review what is logged by Stackdriver for normal, successful access operations. Here’s a log entry related to the bucket access done in Part 1:

Example log entry related to the bucket access done in Part 1.

Note that we see a bucket access operation on bucket-foo-dev-mfa by user [email protected]. There is no indication of what is happening at the OAuth token level, and even if there were, it would be hard to determine if it was suspicious activity.

There are also Token Audit Logs in G Suite Admin, but they have the same challenge–any activity there does not necessarily identify suspicious activity.

Remediation

Multiple remediation options exist but are confusing because of the number of options and the differences between browser vs. API sessions and Google Cloud vs G Suite apps.

If you suspect an account has been compromised and that session tokens are being used, remediation needs to include two parts:

  1. We need to prevent future logins i.e. account access by the attacker
  2. We need to revoke/disable any current access by the attacker i.e. revoke any current OAuth session access and refresh tokens

Although there are many potential settings in the Google Cloud and G Suite Admin Consoles and APIs, here are the effective ones that work in all scenarios:

Table of effective remediation settings for User and Service accounts

User accounts

Prevent future account access via password reset

User accounts can be disabled by resetting the password in G Suite Admin > Users:

Screenshot of password reset in G Suite Admin.

Resetting the password will not revoke currently active sessions, which is a second, separate step described below.

How about deleting the user account? That would also disable future access by the attack, as well as revoke currently active sessions all in one action, but is not recommended as it’s a drastic measure, requiring a high-level of reprovisioning and reconfiguration.

Revoke current sessions via delete OAuth connected applications

The best (only) way to revoke current Google Cloud OAuth sessions is to:

  • Delete the connected Google Cloud SDK OAuth application in G Suite Admin Console or via the corresponding G Suite SDK API call.
  • The Console action and API call will immediately revoke all access and refresh tokens for that user.

In G Suite Admin > Users > user > Security > Connected applications:

Screenshot of connected applications in G Suite Admin

Or via the G Suite Directory API:

Screenshot of password rest in G Suite directory API

Since these actions are on the G Suite Console/APIs, ensure access is granted to G Suite for Google Cloud admins.

Service accounts

Prevent future account access via deleting/recreating service accounts

Service accounts can be deleted in Google Cloud Console > IAM & Admin > Service Accounts

Screenshot showing how to delete service accounts in Google Cloud Console

You might be wondering whether the drastic action of deleting the service account is required vs. disabling or deleting the API key itself. It is needed, because it is the only way to revoke the current session tokens. Deleting or disabling the API key under the service account will not revoke current session tokens.

Revoke current sessions via deleting/recreating service accounts

Deleting a service account will revoke current OAuth sessions, which is why it is the recommended action.

Note: Since the recommended action involves deleting and recreating a service account, this has large implications for recovery/downtime during an incident. All uses of the service account must be reconfigured, including compute instance VMs. This requires stopping the instance at a minimum, which would be recommended anyway in order to ensure current shell access to the VM by the attacker is revoked. This does account provisioning and compute instance configuration should be reviewed and automated with CLI/API scripts as much as possible to minimize downtime and reduce errors.

Other Remediation Actions

These remediation options do not work well in all cases and should be avoided. They are mentioned to ensure that their drawbacks are clearly understood since the number of options available can be confusing.

User accounts

Table showing remediation options for user accounts

Service accounts

Table showing remediation options for service accounts

Conclusion

Because the implementation of OAuth includes caching of session tokens, we’ve seen how easy it is for attackers to exploit this as another attack vector, once access to the GCP administrator’s machine is attained. Credential caches can be easily copied and used elsewhere, and will not require MFA even if configured.

Furthermore, we’ve seen how confusing it can be to prevent, detect, and remediate compromised credential situations due to the numerous options available in both G Suite and Google Cloud.

Despite these considerations, there are concrete measures that can help prevent abuse in the case of compromise:

author image
Jenko Hwong
Jenko has 15+ years of experience in research, product management, and engineering in cloud security, AV/AS, routers/appliances, threat intel, Windows security, vulnerability scanning and compliance. At Netskope, he researches new cloud attacks.