Introduction
In this blog, we’ve analyzed data from Netskope customers that include security settings of over 1 million entities in 156,737 Google Cloud (GCP) projects across hundreds of organizations (see Dataset and Methodology for more details on the dataset).
We will specifically look at the configuration of service accounts, see what’s commonly occurring in the real world, and analyze how multiple security misconfigurations can lead to escalation of privileges and lateral movement. In this Netskope dataset, we observed:
- Service Accounts with user-managed keys: Over 38% (4,160) of service accounts use user-managed keys, which are often not needed, require work to keep secure (e.g. key rotation), and create a larger attack surface for compromised credentials. Service accounts with user-managed keys are common, existing in 82% of the customers in this dataset.
- Service Accounts with old user-managed keys: Of the 4,160 service accounts with user-managed keys, 89% of the service accounts have keys older than 90 days, and over 71% have keys older than a year with the oldest key at 8 years, 1 month. All of this shows the challenges in keeping up with best practices when self-managing keys, which then poses an even greater risk of compromised credentials attacks.
- Over-privileged services accounts with old, user-managed keys: Of the 4,160 service accounts with user-managed keys, 26% (1,102) not only have old keys but are also over-privileged with project-level administrator privileges, such as the project owner/editor roles. This greatly increases the impact after compromise, as it allows wider lateral movement and escalation of privileges.
- Over-privileged services accounts with old, user-managed keys and access to multiple projects: Of these 1,102 high-risk service accounts that are over-privileged with old user-managed keys, 10% (110) have access privileges to multiple projects, providing more opportunities for lateral movement.
- Service account access risk from users: Access to over-privileged service accounts (item #3) can be gained not just from service account key compromise (items #1 and #2), but also from compromise of user accounts. 92 (1.3%) users have the iam.serviceAccountUser or iam.serviceAccountTokenCreator roles at the project level, allowing them to access or impersonate ALL service accounts within a project.
As we can see, there is a cumulative, interrelated risk from the above misconfigurations around service accounts. We’ll be looking closer at:
- Service account design in GCP
- The interdependencies of these four controls
- How misconfigurations in the four controls can be chained by attackers to gain broader access to your GCP environment
Service Accounts
Before we dive into the data and risk analysis, we need to be clear on what service accounts are and how they’re used within GCP. Service accounts are security principals, but created for use by user scripts, applications, or Google services such as virtual machines—the service accounts have roles/permissions that govern the access of the scripts/applications/services using them. Services such as virtual machines can have service accounts attached to them so they run as that particular service account. They differ from user accounts in that:
- They do not have passwords
- Instead, they use RSA public/private key pairs which are used for authentication for Google API access
- Users can impersonate or “run as” a service account
- Finally, they are resources so user access to service accounts can be protected by IAM policies
Types
There are two types of service accounts:
- User-managed. These service accounts include default service accounts, for example, a default service account will be created when you enable the Compute Engine API in a project. However, most user-managed service accounts are created by users. The user is responsible for all management including the creation and rotation of keys.
- GCP-managed (and created). An example of a GCP-managed service account is the Google APIs Service Agent, with an email address that has a format like: [email protected]. This service account runs internal Google processes on behalf of users. It is not listed in the Service Account section in the Console and is not modifiable or accessible by the user.
Keys
User-managed keys: User-managed service accounts can have optional, user-created (and user-managed) keys, which are commonly used for applications that run outside of GCP to authenticate as the service account and access your GCP environment. The keys are a public/private RSA key pair that serve as the credential for authenticating as the service account and the key file is typically a downloadable .json file, which is also the risk for compromise of a user-managed service account. User-managed keys create risk in two areas:
- They involve a key file (credentials) that is often stored in an insecure manner on a less-secure endpoint running the code that needs it, and
- The management of the keys relies on the user. Key rotation, as one example, is a management task that is usually not done by the user, as we see in our dataset.
Google-managed keys: Google creates and manages keys for GCP-managed service accounts as well as user-managed service accounts, and these are used under the hood for the service attachment and impersonation of service accounts. The difference is that when Google manages the keys, it is more secure because private keys are not shipped around and the management (e.g. key rotation) of the keys is more likely to be done.
External vs. Internal
Service accounts that run inside GCP have more secure methods that do not involve downloadable key files containing private keys. Services that run within GCP, such as Compute Engine VMs, can be configured and attached to service accounts and the services will effectively run as that service account. No explicit user keys are needed in this case. Service account keys are described by Google in depth.
However, for scenarios where applications must run outside of GCP and do not have human interaction/authentication, the commonly used option is to have a user-managed key that results in a key file that is collocated or accessible by the application.
Considerations
When using service accounts, consider:
- Whether service accounts are really required or needed in the first place?
- Is the application or service inside or outside of the GCP environment?
- Relatedly, can you have GCP manage the service account keys, rather than have user-managed keys?
- Whether you are using Kubernetes workloads (GKE)?
Google has good guidance in its best practices for using and managing service accounts.
Generally, much of the risk with service accounts is the abuse of them if there are user-managed/created keys, which have key files containing private keys that are downloaded, typically outside of the GCP environment, and used by client application code. It’s a big risk for compromise of those credentials, and multiple CIS IAM controls aim to secure user-managed service account keys in order to reduce the chances of abuse should they be compromised. We’ll discuss this next.
CIS IAM Controls
The backdrop of the service account configurations highlighted in the introduction is the broader IAM controls in the CIS Foundations Benchmark for GCP v1.2. We analyzed 10,783 service accounts, 207,538 policies, and all generated encryption keys across 156,737 projects in several hundred organizations. Some of the key controls in the IAM section and the violations from our dataset are shown in the following table:
Controls 1.4 through 1.7 capture several recommended best practices regarding service accounts:
- Control 1.4: Use GCP-managed service account keys whenever possible, since GCP will handle key rotation automatically and local key files will not be created.
- Control 1.5: Ensure that service accounts do not run with broad admin privileges, so that it is less likely that compromised service accounts can be used for lateral movement and escalation of privileges.
- Control 1.6: Ensure that users do not have access to all service accounts at the project level, as this almost always is giving users overly-broad privileges.
- Control 1.7: If user-managed service account keys are used, ensure that the keys are rotated every 90 days or less, which reduces the attack surface for compromised credentials
As we can also see from the table, some practices such as KMS key security and use of personal login credentials (e.g. @gmail.com) within policies have low numbers of violations and overall expose less risk.
By contrast, service account practices (1.4-1.7) are lagging and introduce risk in a significant (30+%) number of the 10,783 service accounts. This risk comes not only from the higher numbers of violations but also the cumulative risk from violations among these interdependent controls, as we’ll discuss below.
Service Account Risk
What’s of particular interest is how service accounts pose a common attack surface for access to your GCP environment, how various configuration options can increase the attack surface, and how multiple security issues can be chained together and exploited by an attacker to gain elevated privileges and access to all resources in a project.
Let’s go back and analyze the controls and risks associated with service accounts in our dataset.
1. User-managed keys create service account attack surface
An attacker will focus on service accounts with user-managed keys, because the keys pose a higher probability for compromising and gaining access to the service account. User-managed keys will have a json key file containing the private key which is often downloaded and stored on disk local to the external application or script that needs access to the GCP environment. This attack surface is similar to any credential file stored locally on endpoints like laptops or work computers, outside the protections of your GCP environment’s IAM controls.
In our dataset, we found 38% (4,160) service accounts out of 10,783 total service accounts ha