top of page

Lessons from Cloudflare’s Response to the Okta Breach: How to Detect & Respond to Cloud Identity Compromise

Cloudflare disclosed late last week that its cloud infrastructure had been compromised by an identity-based attack, likely conducted by a nation-state adversary.


Not all organizations are targeted by nation-states, but history has shown that advanced techniques quickly trickle down to financially-motivated cybercriminal groups that target a broader range of organizations.


This blog analyzes the timeline of the attack with recommended detections and response actions to defend against similar identity-based threats in the future.



What Happened

  • On October 18, 2023, attackers breached Okta’s support ticket system using a compromised service account. From there the attackers stole HAR files uploaded by Okta’s customers, which contained customers’ credentials.

  • Cloudflare failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise, because they mistakenly believed they were unused.

  • The attackers began probing and performing reconnaissance on Cloudflare’s cloud applications on November 14, looking for a way to use the credentials and discover what systems were accessible.

  • They attempted to log into the Okta instance and were denied access. They attempted access to the Cloudflare Dashboard and were denied access. Additionally, the threat actor accessed an AWS environment used to power the Cloudflare Apps marketplace.

  • They established initial access to Atlassian Jira and Confluence on November 15 using a Moveworks service token to authenticate through the gateway, and then used the Smartsheet service account to gain access to the Atlassian suite.

  • On November 16, the threat actor used the Smartsheet credential to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment should the Smartsheet service account be removed.

  • On November 22, they established persistence by installing the Sliver Adversary Emulation Framework on the Atlassian server.

  • The next day, the threat actor downloaded 76 source code repositories which were primarily related to how identity works at Cloudflare, how backups work, how the global network is configured, remote access, and how Terraform and Kubernetes are used.

  • Cloudflare’s security team was alerted to the threat actor’s presence at 16:00 and deactivated the Smartsheet service account and Atlassian user account within an hour. By the next day, Cloudflare had removed Sliver and all threat actor access was terminated.

  • The company initiated a “Code Red” remediation and hardening effort (staffed by a large part of the Cloudflare technical staff, inside and outside the security team) that lasted just over a month, to understand what the threat actor got access to and ensure that any potentially compromised credentials were rotated. They also hired CrowdStrike to perform an assessment of the scope and extent of the threat actor’s activity.



How Gem’s Cloud Detection & Response (CDR) Platform Can Help in Similar Attacks


  • Unusual Access to Okta and AWS: In the initial phase of the attack, the threat actor attempted to log into the Okta instance and was denied access; and accessed an AWS environment housing the Cloudflare Apps marketplace. Gem would alert on:

    • Unusual or suspicious Okta activity

    • Any initial connections to AWS with unusual patterns of activity

    • Additionally, Gem would automatically correlate all of these related events into a single investigation timeline and blast radius graph

  • Stale Access Keys: The attackers used credentials stolen from the Okta breach that had not been used in a long time.

    • Gem would alert on the use of stale AWS access keys and unused tokens.

  • Unusual Access to SaaS applications (Atlassian Jira and Confluence)

    • Gem doesn’t monitor Atlassian application logs, but would alert on unusual access by continuously analyzing the load balancer logs and VPC Flow Logs (for organizations self-hosting Atlassian in a public cloud infrastructure).

  • C2 Communication (Adversary Framework)

    • Gem’s detection for flow logs is a classic use case for C2 servers. This communication would have been detected by examining flow logs for any unusual outbound communication, including communication with a suspicious IP address.

    • In particular, the attacker downloaded 76 repos from the organization‘s Bitbucket. If they were exfiltrated, this would have been detected as unusual mass downlink communication with an external or suspicious IP.


Summary

Cloudflare's response to the Okta incident was immediate and thorough – the breach was effectively contained, and CrowdStrike confirmed that no customer data or systems were compromised. Clouflare’s transparency about the incident deserves recognition and gratitude from the entire community. 


At Gem, we’re strong believers in an “assume-breach” defensive strategy, and we use every published attack to conduct a thorough assessment of our defense-in-depth capabilities and how they can be continuously improved. 


Our cloud security research team is now using this incident as a learning opportunity to introduce new detections to our cloud UEBA, and we’d be happy to share specific detections that would have been helpful in providing detection across the different stages of the attack, 


Learn more:

Comments


bottom of page