On October 20th, Okta announced that it had discovered unauthorized activity in its support systems. The announcement followed disclosures to Okta from security vendors 1Password, BeyondTrust, and Cloudflare, who had all discovered malicious activity in their environments that they had traced back to Okta.
The Okta breach has spawned lots of discussion about the importance of identity (and identity providers) in security today, in particular in the cloud. Indeed, the disclosures from Okta and from the affected customers who have come forward illustrate key patterns in today’s enterprise environments:
Continuous activity monitoring, with rapid detection and response, is critical to defending against identity-based attacks. Relying only on traditional security hygiene and posture management measures, like configuration management for machine identities or MFA for human identities, doesn’t help the organization when an attacker inevitably breaks through with compromised credentials.
Response time (MTTR) matters. All three affected organizations who published their own responses to the breach emphasized the importance of fast response, and their ability to contain the threat before it impacted customers was critical to stopping the attack
Here, we discuss and analyze what happened at both Okta and the affected customers.
What Happened?
What We Know
When Okta customer support agents are debugging issues, they often require detailed technical information about the Okta session in a customer’s browser. As such, Okta support often asks customers to record their browser sessions and upload the recordings as HTTP Archive (HAR) files. HAR files record everything in a browser session, including sensitive credentials. These credentials embedded in the HAR files were the attacker’s targets.
The initial discovery of the breach came not from Okta, but from Okta customers 1Password, BeyondTrust, and Cloudflare, who noticed suspicious activity associated with valid credentials that had previously been linked to HAR files shared with Okta support (more on this below).
Over the course of its initial investigation, Okta did not identify any suspicious HAR file downloads associated with these support cases. The tipoff came from BeyondTrust, who had noticed an attempt to access their Okta admin console from an IP address in Malaysia linked to anonymizing proxy/VPN services. After BeyondTrust provided Okta with this suspicious IP address, Okta was able to identify a compromised service account that had accessed the HAR files through a non-standard path that did not appear in the access logs the company normally monitored.
The initial access vector remains unclear, but according to Okta, an employee had saved corporate credentials for the compromised service account in their personal Google profile on an Okta-managed laptop. Okta believes that compromise of either the Google profile or the laptop itself is the most likely explanation for initial access.
Key Learnings from the Breach
Knowing the right logs to monitor is hard. In their published response, Okta noted that the attacker evaded detection for 14 days because they did not directly access customer support cases in the Okta system. Instead, they accessed the uploaded HAR files directly. These two types of actions generated different logging events, and though Okta closely monitored customer support access events (control plane), access to uploaded files themselves was not monitored (data plane). This problem generalizes beyond just the Okta incident. For example, in AWS, we often see that CloudTrail management events (control plane) are closely monitored, but data events like S3 or RDS access (data plane) are not. This leaves organizations blind to exactly the type of exfiltration seen here. Understanding what you need to monitor is a critical first step in any program for detection and response, but understanding exactly what telemetry is needed to detect each type of attack is not easy.
Activity monitoring is key to detecting identity-based threats. As is common in cloud attacks, the initial access vector here was likely a compromised identity. This is particularly important for machine identities and service accounts, which are essential for providing application access but are often poorly configured and lack the ability to enforce secure access with multi-factor authentication. Detecting identity-based threats requires understanding suspicious behavioral patterns. Some of these patterns are widely used - i.e., impossible travel or working from unusual locations. Others are specific to the application in question: in this case, a highly-specialized detection might enumerate over Okta services to understand the scopes and the permissions normally used by a given identity and alert when unusual scopes are used.
Lessons Learned from Compromised Okta Users
The breach at Okta itself is interesting, but what does it mean for Okta customers? Okta has identified 134 customers who were affected. The published responses from 1Password, BeyondTrust, and Cloudflare give us some insight into how customers can detect and respond to these types of supply chain compromises.
Monitoring Okta System Logs is Necessary
As an Okta customer, monitoring for suspicious activity in Okta system logs is critical to detecting malicious activity within your tenant. In the 1Password case, the initial tipoff that something was wrong came not from an automated detection system, but rather a team member who received a report from Okta that they didn’t remember asking for. Nothing was “wrong” in their environment, but due to the employee’s diligence, the company was able to detect and stop the breach. Here, activity monitoring was critical: if the company were reliant on only security posture and preventative approaches, the alert may have been missed.
So what should SecOps teams be looking for in Okta logs? Okta is a very sophisticated application, but simple rules like impossible travel still apply. Organizations should make sure they have activity monitoring in place to detect these types of tactics.
But teams should also develop the ability to detect Okta-specific malicious activity. For example, at Gem, we protect against Session Hijacking by alerting if a user agent changes within a specific Okta session. In the context of this breach, this detection would trigger an alert if, for example, an analyst uploaded a HAR file created on Windows and an attacker used the session key embedded within that HAR file to continue the session on Linux.
Response Time is Critical
But detection is only half the battle. Significantly, all three published responses from 1Password, BeyondTrust, and Cloudflare represent fairly mature security organizations. In all three cases, the affected teams emphasized the importance of timely response.
Effective response to these kinds of threats requires organizations to take several different actions quickly. First, teams need to disable the affected Okta account. This is a clear first step in containing the attack and eliminating the attacker’s access to the environment.
But this isn’t the only action that’s required. Teams need to be able to understand what else happened while the attacker had access: were any other assets compromised? Has the attacker established any other foothold in the environment? Was sensitive data exfiltrated?
In the case of BeyondTrust, the attackers used their Okta access to create a new service account called “svc_network_backup” in the BeyondTrust environment. Luckily, BeyondTrust detected this malicious account and quickly took action. But this illustrates the importance of quickly understanding the investigation timeline and blast radius of an attack to effectively contain the incident. Had BeyondTrust missed this, the attacker may have been able to persist in the environment.
At a minimum, teams need to be able to construct a blast radius map of all affected assets, isolate compromised resources from the main environment, and ensure they have the ability to forensically analyze the affected resources to ensure they understand the root cause of the breach. The faster organizations can do this, the lower their risk of a severe incident.
Summary & Conclusions
Identity plays a critical role in securing today’s IT environment. The Okta breach demonstrates the importance of real-time activity monitoring for identity-based threats: configuration and entitlement management is important, but being able to detect compromised identities requires analysis of behavior. When organizations are impacted by identity-based attacks, ensuring that teams are ready for fast and effective response is key to minimizing the impact of the breach.
At Gem, identifying and responding to suspicious activities in Okta, Azure AD, and Google Workspace is built into our Cloud Detection and Response (CDR) platform. If you’re interested in learning more, please don’t hesitate to reach out.
For additional information, be sure to check out our on-demand webinar, in which Yotam Meitar, our Director of Cloud Incident Response, discusses the attack and provides actionable advice on how to respond. The recording is available here.