For any security operations team, a key question whenever a new alert pops up is inevitably the same: Who did it?
Exposing which entities were involved in a threat is a crucial first step in any investigation. Once responders know who triggered the alert, they can move on to deeper questions like:
● Is this a typical behavior for this entity?
● Were any other actions performed by this entity around the same time?
● What other resources could potentially be at risk?
These questions allow responders to start building the context they need, relying on understanding the original “whodunnit.” However, when investigating an alert in AWS using CloudTrail, the process of identifying the source entity behind an alert can become difficult due to the complexities introduced by role assumption chains. Our experience at Gem has also shown us that many customers haven’t implemented enhanced session traceability features provided by AWS, such as SourceIdentity, which can remove this complexity.
Gem's research team has uncovered a new technique used by attackers, which we have named Console Conceal. Attackers can take advantage of a quirk of the default AWS configuration (without SourceIdentity configured) to potentially make detecting and attributing their actions more difficult.
For clarity, it is important to note that in order to leverage these techniques the attacker must first gain unauthorized access to the AWS security credentials of a user with permissions to perform assume role operations.
In this post, we’ll explain ‘Console Conceal’ as well as the session traceability features provided by AWS that can mitigate it. We will show how to put SourceIdentity into practice in your own organization and provide a methodology to deal with ‘Console Conceal’ when you find yourself in a situation without the benefits of SourceIdentity.
The Console Conceal Quirk
Before exploring the technique, let's begin by understanding the non-obvious behavior central to 'Console Conceal'.
Actions in AWS are performed using access keys. There are two types of access keys:
Permanent access keys, which are associated with specific IAM Users.
Temporary access keys, generated through role assumption.
Temporary access keys are expected to appear in two types of CloudTrail records:
In the userIdentity element of the CloudTrail record corresponding to any action made with the access key.
In the 'responseElements' field of the CloudTrail record representing the AssumeRole event responsible for generating the temporary access key.
When using the command line interface or one of AWS’ SDKs, the combination of these two records provides the desired traceability back to the original IAM principal. Surprisingly, this does not extend to actions performed through AWS Console. When a console session is initiated under an assumed role, every action executed through the console will be logged in CloudTrail with a temporary access key that’s different from the one in the corresponding AssumeRole event because the key is derived by AWS from the original request.
But why does this matter? To comprehend how attackers can exploit the Console Conceal quirk to their advantage, let's delve into the story of an attack through the eyes of the fictional attacker, Eve. As we follow Eve's narrative, we will gradually uncover the workings of the Console Conceal technique. This case study offers a simplified version of similar incidents encountered by our team.
Eve the Attacker
By some means, Eve successfully obtained the permanent access key of an IAM User named Alice. With these credentials in hand, Eve's objective is to maintain prolonged operations and persistence. To achieve her goal, Eve wants to conceal the fact that she is leveraging Alice's access key, as it serves as a sole foothold in the AWS account.
Unfortunately for Eve, each action performed leveraging Alice’s access key will be logged by CloudTrail, where each record will contain the following userIdentity element:
As shown in Figure 1, any actions carried out by Eve using the access key are immediately linked to Alice's stolen access key. This simple connection allows security teams to pinpoint the source of suspicious actions, presumably leading to quick remediation of Eve’s only foothold in the AWS environment.
So what can Eve do?
Shifting suspicion to Bob with session name manipulation
Eve makes a significant discovery: using Alice's access key, she can use Alice’s permission to assume the 'DevelopersRole,' which also has the permissions needed to carry out her attack. Additionally, she understands what she hopes the defenders might not: that when doing an assume role call, the role session name identifier is just a free form command line parameter, not trusted or verified metadata about the role session.
Putting this together, before performing malicious actions in the AWS account, Eve assumes the 'DevelopersRole' while naming her session ‘bobB@techwave.com’, in a way that she hopes will slow down and throw off defenders:
aws sts assume-role --role-arnarn:aws:iam::121175308212:role/DevelopersRole --role-session-name bobB@techwave.comAny further actions performed by Eve under this assumed role will be recorded by CloudTrail with the following 'userIdentity' element:
From Eve's perspective, this scenario seems highly favorable. As seen in Figure 2, an investigator examining the event that prompted the investigation would find no indication of Alice's stolen access key. Instead, the userIdentity element contains a temporary access key (prefixed with ‘ASIA’) and Bob’s name.
Fortunately for investigators and for Bob, Eve did leave a trace - the AssumeRole event which initiated the session. Eve also left the perfect breadcrumb - the temporary access key. Let's examine the CloudTrail log of the AssumeRole event:
The AssumeRole event in Figure 3 reveals the entire story. By analyzing it, investigators can simply link the malicious session to Alice's access key. To identify it, a responder needs only to search for an AssumeRole event in which the access key id in the 'responseElements' field is identical to the one found in the suspicious entry. This crucial connection leads the responder directly to the compromised user and their associated security credentials, uncovering the attempt to conceal it by implicating poor Bob.
Leveraging Console Conceal
Eve has one more trick up her sleeve. Following the assumption of the 'DevelopersRole' and before taking any further action, Eve proceeds to create a federated console session. A federated console session is a long outdated (10+ years) but still valid means of accessing the AWS console that doesn’t change an IAM principal’s permissions, but simplistically converts a CLI session into a console session.
However, due to the Console Conceal quirk, every action carried out by Eve through the AWS Console will not be logged with the temporary access key of her role session, but with an access key ID that isn’t the same as the one that appears in the AssumeRole event. This means investigators will need a new method to identify the corresponding AssumeRole event linked to Alice's compromised key. By leveraging Console Conceal, Eve was able to eliminate the best breadcrumb which may well slow down those investigators in remediating her malicious foothold.
Console Conceal Countermeasures
We can’t let Eve get away with it! Let’s discuss both how Console Conceal can be prevented through the use of SourceIdentity and how to maintain traceability even if you haven’t yet configured SourceIdentity.
SourceIdentity
In short, source identity is an immutable identifier that carries over role assumption events. It also gives administrators control over the initial value minted into the identifier. For federated access into an IAM role with SAML or OIDC, the SourceIdentity value comes from the Identity Provider directly.
In the case we’ve been discussing here, an IAM user assuming an IAM role, the policy is flexible, but most organizations should just require that the source identity be set to the IAM username of the user assuming the role. Here’s an example of an IAM role trust policy that does just that:
With this policy in place, the traceability complications go away – Eve can’t successfully assume the role unless she mints Alice’s username into the source identity parameter. While she can still put what she wants in for role session name, the discrepancy between the two will likely draw more attention, not less:
aws sts assume-role --role-arn arn:aws:iam::121175308212:role/DevelopersRole --role-session-name bobB@techwave.com --source-identity aliceA@techwave.comMost importantly, source identity also appears in CloudTrail. If we look at the same event we saw before (Figure 2), we see that the sourceIdentity now appears within the UserIdentity structure, providing direct traceability for both the AssumeRole call and all subsequent actions.
These benefits for traceability are pretty clear, and we recommend that you take the time to deploy SourceIdentity for your organization. You can learn more about SourceIdentity by checking out the AWS documentation.
Sourcing Identities without SourceIdentity
But what about if you find yourself in this situation without the benefit of SourceIdentity?
The key to fighting Console Conceal is locating the AssumeRole event corresponding to Eve’s console activity - but how can we find it without an indicative access key telling us where to look?
The first approach which may come to mind is correlating malicious console events to corresponding AssumeRole events through source IP addresses. While tempting, this approach will often be hindered by the fact that many console events will have internal AWS IP addresses as their sources. Another approach would be to comb through all role assumptions in the timeframe before Eve’s malicious operations. Depending on the organization’s usage of IAM roles, this might be a very tedious effort, if not impossible.
At Gem, we've devised a novel methodology for automating the search for the corresponding AssumeRole, utilizing multiple fields as pointers (see Figures 6 and 7):
principalId: The second part of the principalId corresponds to the roleSessionName in the requestParameters of a logged AssumeRole event.
userName: This field represents the role's name and correlates to the roleName in the requestParameters of a previous AssumeRole event.
sessionContext_attributes_creationDate: This field can be matched against the eventTime of a previous AssumeRole event. However, it's important to note that the eventTime in an AssumeRole event might be slightly earlier (by as much as 1 second).
Summary
Finding out the relevant identity behind an alert is central for security operations in the cloud. Despite this importance, in AWS, role assumption chains can obscure the source entity behind an alert. Enhanced session traceability features provided by AWS, such as SourceIdentity, can remove this complexity, but many customers do not have this functionality enabled by default. The Console Conceal technique takes advantage of this default configuration to obscure source identities in CloudTrail logs, making detecting and responding to these threats more difficult.
The technique illustrates the importance of leveraging AWS features like SourceIdentity to enable efficient threat investigation and response. If these features are not enabled, though, Gem’s Cloud Detection and Response platform can still recover source identities, providing effective protection against the Console Conceal technique.