I commend CircleCI for sharing with its customers and SC Magazine the investigation results of their disclosed breach of Jan 4, 2023. In today’s world of cyberattacks, it is imperative that information about attack tactics, techniques, and procedures (TTPs) be shared so others may better defend themselves. The CircleCI executive team shows true leadership here, and it is appreciated.

 

What are the lessons learned from each step of this attack and breach?

 

Initial attack and compromise

The initial attack is believed to have occurred around Dec 16, 2022, when the laptop of a company engineer was compromised, and a set of privileged, two-factor authentication-backed credentials were stolen. The employee whose machine was compromised just happened to generate production access tokens as part of their job, giving the attacker lateral carte blanche to expand the attack.

 

Lessons

1) Endpoint security failed again. There’s a vast overreliance across organizations on EDR or AV software. Yes, covering the endpoint is a core part of a cybersecurity defense strategy, but it is not the only part, and the simple truth is that it can always be defeated by a determined adversary. The lesson learned is that if you rely on endpoint protection as the core of your defensive strategy, you are susceptible to attack.

2) For many employees in an organization, broader access to systems required for their role makes them a high-value target. Engineers, Finance and Accounting, Research, etc., are all high-value targets. For a software company like CircleCI, the engineering/software development team with their access to dev and production environments means they are targets. In this case, with the employee having privileges to create access tokens to production environments, the attacker scored an easy win. The lesson learned is that access control (think zero trust architecture) is a must for all parts of the organization.

3) Once the machine was compromised, the attacker immediately targeted two-factor authentication to expand the attack. The attacker bypassed all of the strong authentication controls by compromising the token manager. One wonders if this attack was targeted or if that attacker just got really lucky. The lesson learned here is to ensure employees and their machines, no matter how much they complain, are locked down and monitored. The keys to the kingdom can be found in many different places. Make sure you know where they are and have protections built around them. Another lesson learned is that strong authentication, another core to a competent cyber defense strategy, like endpoint, can be defeated and alone, does not protect your organization.

 

Expansion, Reconnaissance and Data Theft

The attacker, once in, elevated privileges with stolen credentials and, with those credentials, introduced malware to impersonate the employee (via session cookie theft) and expand access to different production systems. It is interesting to note that “remote location” was important to the attack but not illuminated. With broad access, the attacker stole data that included customer environment variables, tokens, and keys from backend systems. The attacker, still undetected, shifted to broad reconnaissance activities across the backend and cloud and then to the company’s partners and supply chain. The attacker also managed to extract the encryption keys, bypassing all of the encryption controls in place.

 

Lessons

4) Network monitoring remains a significant gap in most organizations’ cyber security programs. In this case, the lack of continuous monitoring of internal traffic left CircleCI blind to the attack unfolding across its network for close to a month. Where strong auth, firewalls, and endpoint defenses all failed, Continuous network monitoring with a Network Detection and Response (NDR) system would have detected the large amounts of anomalous internal traffic involved in this attack and alerted the IT team to the threat in a matter of hours. Even their SIEM missed the indicators of compromise unfolding through the attack, most likely because all the sessions looked valid. If one of CircleCI’s clients had not notified them, they would not have known about the compromise for months. Lesson – continuous visibility to the risk and threats across and inside your network remains a critical component of a complete cybersecurity program. With new, lower-cost cloud-native NDR and continuous monitoring solutions available, review your program to see if this level of critical protection can be added.

5) As noted above, “remote location” was mentioned in the investigation report, but no details as to why it was important to the attack. Based on what we at CyGlass have seen in the field, many organizations cannot deploy consistent levels of security controls across their geographic locations due to cost or operations complexity. This is especially true when hardware deployments are involved. The lesson is that your weakest link in the chain is where you will be targeted. Do not deploy programs and tools that leave gaps and weak links when deciding on technologies and control environments. Spend time evaluating your programs, determining where risks exist, and how they can be mitigated. Select defense tools that ease both deployment and operational costs, while meeting project goals.

 

Expanding Across the Supply Chain and partners

Once the attacker was able to decrypt data and generate new credentials, the attack expanded across CircleCI’s ecosystem. Like Okta, Github, and Solar Winds before them, the attacker quickly moved on to bigger fish. Currently, four customers of CircleCI have reported unauthorized access following the breach. Supply chain/partner ecosystem attacks exploit the weak points in one company’s cyber defenses to easily exploit the defenses of a more critical, better-protected partner. Once again, a customer alerted CircleCI to the attack when they detected risky GitHub OAuth activity, significantly reducing the attack’s dwell time.

The hybrid network world of today means that cloud environments are also targets in the CircleCI attack. Since the CircleCI platform stores authentication credentials, threat actors can use them to move from network to cloud environments, including GitHub, AWS, Google Cloud, and Microsoft Azure.

 

Lessons

6) Almost any organization that has moved into the digital world of business or has adopted cloud technologies ( and since the start of the COVID pandemic, that is pretty much everyone) has to expand its definition of its cybersecurity program to include its partner and cloud ecosystem. Defining where systems connect, how authentication and access are managed, and where the risks are in that process is now core to any effective defense plan.

7) Hybrid networks are changing the definition of network defense and response. Legacy NDR tools that utilize on-premise hardware and compute often cannot correlate network and cloud threats creating defensive gaps. XDR tools built around EDR systems are often weak in network-to-endpoint and cloud threat correlation. As you replace or deploy new cybersecurity tools, look to systems that cover and correlate the greatest number of threat surfaces to ensure effective defenses.

 

https://www.scmagazine.com/analysis/breach/compromise-of-employee-device-credentials-led-to-circleci-breach

https://circleci.com/blog/jan-4-2023-incident-report/

 

Learn more about how 100% cloud-native CyGlass solutions will deliver continuous visibility to your network and cloud systems.

Learn more about protecting your supply chain.

Bill Munroe

VP Marketing, CyGlass