At Gem, our threat research team is constantly bringing the attacker’s perspective to discover new methods and exploits that can be used to gain access to cloud environments. A few weeks ago, we discovered a technique that could have allowed an attacker to bypass AWS GuardDuty’s threat detection service. Using these methods, threat actors in possession of IAM active credentials that had the power to update S3 bucket policies could have bypassed GuardDuty’s S3 detections and silently updated permissions for S3 resources, resulting in a bucket configuration that allowed anonymous data access.
This technique takes advantage of an issue that is described within GuardDuty's documentation, but probably wasn’t known to AWS users. Although this behavior was documented, we realized it could be used maliciously under a narrow set of conditions to bypass GuardDuty alerts that normally fire when access is changed to allow public or anonymous access to an S3 bucket. This gap in GuardDuty’s alert coverage occurred only when S3’s Block Public Access was not enabled on the account or the bucket, and when KMS-based server-side bucket encryption was not in use. Gem discovered this technique in a real-world scenario, and reported it to AWS. Our report led the GuardDuty team to make changes to its behavior to cover the reported gap.
GuardDuty Background
AWS GuardDuty is Amazon’s premier cloud threat detection engine. GuardDuty monitors customers’ AWS accounts, scanning activity on S3 buckets, databases, containers, EC2 instances, and more to detect emerging threats in the environment using anomaly detection.
GuardDuty is extremely widely-used: at Re:Invent 2022, AWS CEO Adam Selipsky noted in his keynote address that GuardDuty was used by tens of thousands of customers, including 85% of the largest 2,000 customers at AWS. This is a critical security service upon which some of the largest corporations in the world rely.
How Attackers Could Have Changed Bucket Permissions Without Triggering a High-Severity Alert
Changing bucket permissions is always a sensitive action in cloud environments. S3 buckets often hold sensitive data, and bucket permissions are the primary way that access to that data is controlled. As such, attackers who have somehow obtained powerful IAM credentials and are preparing to exfiltrate data may first change bucket permissions to allow them to do so later without such credentials.
A robust cloud detection program should track changes to S3 bucket permissions and trigger alerts when permissions are changed in unexpected ways. Indeed, GuardDuty has two built-in alerts that trigger when bucket permissions are changed. The first alert triggers when a bucket’s permissions are changed to allow public access (i.e., anyone with valid IAM credentials from any AWS account, regardless of their principal-based permissions can download data from the bucket). The second alert triggers when a bucket’s permissions are changed to allow anonymous access (which enables access for anyone with an internet connection and knowledge of the bucket name). The first type of access (public) involves a Sigv4-signed request, and the second type (anonymous) involves an unsigned request.
Researchers at Gem discovered a way attackers with access to administrative S3 credentials could have exposed potentially sensitive S3 data in the bucket without triggering the high-severity alerts on GuardDuty mentioned above.
The method that Gem discovered can bypass GuardDuty alerts for buckets for which S3’s Block Public Access (BPA) feature was disabled. If BPA was enabled either at the account or the bucket level, then to make a protected bucket public you must first disable that feature. GuardDuty always alerts in that case with low severity alerts, and so for customers following best practices and using BPA, there was no way to bypass some level of GuardDuty alert.
The Issue Discovered by Gem
The following policy contains one policy statement that enables anonymous access to a bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::<bucket-name>",
"arn:aws:s3:::<bucket-name>/*"
}
]
}Attaching this policy to any bucket in an environment monitored by GuardDuty should trigger an alert. Indeed, we observe GuardDuty alerts when this and similar policies are applied to S3 buckets. However, consider a new policy with an additional statement added:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:GetBucketPublicAccessBlock",
"Resource": "arn:aws:s3:::<bucket-name>"
},
{
"Sid": "Statement2",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::<bucket-name>",
"arn:aws:s3:::<bucket-name>/*"
]
}
]
}Notice that the second statement in this policy is identical to the first, allowing public access to the bucket. The first statement appears somewhat unrelated - it simply disallows read access to the bucket’s public access configuration settings. However, when these statements were applied together in the same policy update, no high-severity GuardDuty alert was triggered.
Documented Behavior Versus Expected Behavior
In the documentation for the two high-severity GuardDuty findings, the following note appeared:
"If a bucket's ACLs or bucket policies are configured to explicitly deny or to deny all, this finding cannot be generated for that bucket.’"
It is our understanding that in order to trigger the two high-severity alerts, GuardDuty invoked two api calls: GetBucketPublicAccessBlock and GetBucketPolicyStatus. Blocking these specific api calls, essentially blocked GuardDuty’s ability to trigger the alerts. As a result, customers that did not adhere to the best practices regarding S3 policies (i.e. - configuring an S3 bucket’s policy to explicitly deny or to deny all) might have inadvertently granted public or anonymous access to their buckets without receiving a GuardDuty alert.
With Gem’s high fidelity detection technology, our system was able to detect the change from private to public even in the case where access to an important configuration API was explicitly denied and thus no GuardDuty alert was generated.
We reported this gap to AWS Security and suggested that documentation of the gap alone wasn’t good enough. We recommended that GuardDuty alert on its loss of visibility as to the BPA status of the bucket, rather than allowing the lack of visibility to prevent an alert. After analyzing our report, the AWS Security and GuardDuty teams agreed that such a change was worthwhile. As of April 28th, GuardDuty will alert on activity where a dual-purpose policy such as the one above -- or any similar policy that both allows data access but seeks to deny access to configuration information -- is applied. This change ensures customers and environments are well covered from this subtle attack path.
Better Together
The cloud brings a new age of security needs: the rapid change of both the cloud and the ability of attackers to navigate through it, require a new operationalized security program that brings in not only the speed of the cloud, but the context - an assume breach approach to the cloud.
By using the attacker's perspective, Gem brings the ability for enterprises to eliminate visibility blindspots, and through high fidelity cloud alerts, triage to root cause analysis with built in context.
Gem works closely together with AWS GuardDuty and other threat detection services, building tools for security operations teams to visualize and simplify triage of their alerts, as well as ours.
With the attacker’s perspective in mind, our threat research team constantly discovers new methods and exploits that can be used to gain an advantage in cloud environments. Through such close collaboration, immediately upon discovery, Gem reported the issue to GuardDuty, leading to fast resolution.
About the Authors
Itay Harel is a Senior Software Engineer and a key member of Gem’s core team. He specializes in cloud threat research, with a primary focus on identity research in multi-cloud environments, leveraging his expertise to drive the development of Gem’s cloud-native threat detection, investigation, and response platform.
Ran Amos is a Software Engineer and a core member of Gem's R&D team. Ran specializes in cloud threat detection, developing new technology to bring best-in-class detection coverage to customers. Prior to Gem, Ran spent over seven years leading research and development teams in Unit 8200, Israel's elite cyber and intelligence unit.