CrowdStrike and Microsoft Outage

On July 19th CrowdStrike deployed a faulty patch/configuration update for its Falcon sensor software. The effected devices were those PC’s and servers running Windows operating systems. The outage, which affected systems worldwide, including Windows virtual machines and the MS Azure platform, began rebooting and/or crashing at approximately 10:48 AM Eastern. (The affected systems also included those running Windows 10 and 11 that were running CrowdStrike Falcon). Machines running macOS and Linux were NOT affected. It was noted in a number of sources that there was a similar issue for devices running Linux in April 2024.

 At 9:27 AM Eastern, CrowdStrike deployed updated content. Devices that booted with this later content were not affected. 

NATURE OF THE ISSUE:

CrowdStrike’s CEO, George Kurtz, confirmed the issue was due to a faulty kernel configuration level file and NOT the result of a Cyber Attack. Given the AT&T data breach just 10 days ago on July 12th, and the fact that there have been 10 major Cyber Attacks or Data Breaches so far in 2024, it was not surprising for the general public, as well as all of the affected all types and sizes of businesses ranging from airlines to hospitals to federal agencies and retail stores to immediately think that another Cyber Incident was the cause of the ensuing outages caused by the configuration issue. Thankfully this was not a cyber-attack, however, this issue does point out just how vulnerable the organizations we critically reliant on are. 

THE FIX:

Should any organizations still be unable to fix their issues, there are a number of organizations, including CyberSecOp, that are able to help you with the relatively direct but painstaking task fix for the ‘outage’.

• Affected Machines can be restored by booting into safe-mode or the Windows Recovery Environment and deleting any .sys files beginning with C-00000291- and with timestamp 0409 UTC in the %windir%\System32\drivers\CrowdStrike\ directory. 

• This process must be done locally on each individual device.

• Someone will have to reboot the affected computers individually with manual intervention on each system.

• NOTE: Some Azure customers have had success by rebooting the affected virtual machines numerous times (10,12,15 times was not unheard of) while connected to Ethernet.
  NOTE: Microsoft has also recommended restoring from back-up from before July 18th. 

HOW AND WHY THIS HAPPENED

Though at this time we cannot be 100% certain as to what caused this issue, we can expect that one of the primary culprits was the lack of testing and validation of the configuration update prior to its release. As noted in last week’s posting from CyberSecOp, it is absolutely critical for organizations of all sizes, across all market segments to properly plan and establish policies for their use, deployment and on-going updating of their technology ecosystem. In this case, whether or not CrowdStrike has the proper plans and controls in place to ensure they have screened and tested their upgrades, patches and releases is not in question – what is being questioned, and far more importantly is - have they been followed. Further, are those organizations and their managed services providers; effected by this latest outage properly testing and validating ANY changes to their environments before deploying them in to production environments? Is the CMDB in place? 

THE MORAL OF THE STORY  …

Organizations must develop, iterate on, and adhere to robust policies and procedures to enhance their change management processes. From a risk management standpoint, organizations should reconsider their operational strategies to ensure that dependency on a single vendor does not impact all of their operations. Diversifying vendors and creating competitive hedges can be crucial in times of disaster. Questions organizations should consider asking their managed services providers include:

• Was this a planned update?

• What testing was conducted in non-production environments, and what were the results?

• What CMDB policies and procedures were overlooked?

Ensure Cyber Resilience by Developing Your Cyber Resilience Plan

Cybersecurity risks are common in the digital age, but we should not allow them to limit our business goals. Cybersecurity consulting and security breaches have gradually become critical parts of the IT department. We cannot tackle cyber threats without the expertise of cybersecurity consultants or cybersecurity breach experts.

Whether starting from scratch or looking to improve your existing cybersecurity resources, hiring a qualified cybersecurity consultant or security breach expert is always a good idea first.

Cybersecurity Vulnerabilities, Exploits, and Threats

Cybersecurity vulnerabilities are more likely to occur when old software versions are in use. Cybercriminals may exploit any software bugs they find. They may even write malware to specifically target older applications, which are less secure and easier to hack than their newer counterparts.

If you want to protect your company from cyber attacks, here are a few ways:

• Use the latest version of any application that is essential for your business (e.g., antivirus software);

• Have an expert conduct regular audits and penetration tests;

• Install operating system patches as soon as possible;

• Train employees on cybersecurity best practices;

• Strengthen network defenses by having an active firewall, two-factor authentication, and strong passwords policies in place;

• Let employees know what they can do to keep their devices safe at home and work.

• Implement ransomware backup best practices

Cyber Risk Mitigation Solution

When considering an organization's cyber security approach, the initial needle-moving goal should not be to avoid all risks, but instead, the strategy should contemplate how much security and what type of risk level is appropriate for different vulnerabilities.

The vulnerability analysis techniques present an opportunity for organizations to address the weaknesses in their current system and security framework. The flaw-based techniques help in assessing risks, hypothesizing breach points as well as benefits and disadvantages produced as a result of these risks. The damage potential (DP) technique offers this kind of precise information about vulnerable areas within cyber-systems that are commonplace features of computers and mobile devices.

Companies are getting hit by ransomware attacks, but there are things that both companies and employees can do to protect themselves by hiring an organization like CyberSecOp to implement a robust security program.

Security breaches can have disastrous effects as they potentially expose sensitive data to hackers. Companies are advised to plan their security measures in a preventive way and commit to these methods. The response should also be planned in order to restore the state prior to or minimize the damage caused by a security breach effectively.

Governmental agencies are investing in cybersecurity.

Cybersecurity has become a major issue. Governmental agencies are investing in the cybersecurity domain and focusing on security advancements. For example, Cybersecurity Strategy 2020 and expanding U.S. Cyber Task Force under DHS relate to this area of security and address the broad scope of cyber threats to federal networks, systems, or communications infrastructure identify priorities for cyber risk management, ensure persistent attention to counterterrorism with respect to cyberspace policymaking, execution baseline resilience against digital interference, reduce interstate conflict over jurisdictional responsibilities related to cybersecurity matters by improving key information-sharing practices among relevant federal entities in order to promote operational coordination on cybersecurity matters.