The recent widespread technical issues, being reported as a Microsoft issue, have caused significant disruptions for many companies globally. We understand the frustration and challenges this has brought to your business operations. As your trusted partner, we want to provide you with information, support at this time and in the future create strategies to build resilience into your IT infrastructure, ensuring you are better prepared for future incidents.
Understanding the incident
As you may be aware, there has been an ongoing technical issue, being reported as a Microsoft incident, affecting many companies worldwide. From our analysis and validation of available information, it appears that the root cause is an update deployed by the cybersecurity company CrowdStrike. The update affected Microsoft Windows systems running their Falcon agent, causing them to crash with the infamous Blue Screen of Death (BSOD).
CrowdStrike has since rolled back the update and provided workaround information to help recover from the issue.
How could an outage of this scale have happened?
Based on the information available, it appears that CrowdStrike’s software updates are controlled centrally by CrowdStrike, meaning that customers have no control over how the updates are deployed. Standard procedure for updating software typically involves installing updates on a small set of test or pre-stage systems to confirm stability before a wider rollout.
The fix for this particular issue is not particularly difficult or complicated; however, it is challenging to automate. Given the number of affected systems, it may take some companies a long time to get all services back up and running.
What does this outage mean for the future of IT systems?
This outage highlights the risks associated with not fully understanding how deployed software works, especially when the vendor has the ability to deploy changes without customer awareness. While this capability can save costs and reduce management overheads, it comes with risks, as we have seen in this incident.
The dangers of centralised control and the need for diversification
While it is always a good idea to diversify to some degree, having single tools providing the same capability for all IT systems has numerous advantages for visibility, ease of administration, and lower costs. More importantly, fully understanding the technology you are deploying ensures it aligns with your business processes and needs, thereby minimising risk.
Why has this outage caused such chaos and how long can we expect it to last?
CrowdStrike is a leading provider in the endpoint protection software market, so its customer base is extremely large. Their approach to deploying an update to all systems at once resulted in widespread effects, compounding the issue. While recovering from the problem is fairly easy, depending on the number of affected systems, it may take some time to get everything back up and running.
Lessons learned and moving forward
This outage has underscored the need for robust disaster recovery (DR) and backup solutions. Having a well-planned DR strategy ensures that your business can quickly recover from disruptions, minimising downtime and data loss.
By embracing a multi-cloud strategy, which involves using services from multiple cloud providers, you can enhance your redundancy and minimise the risk of total service failure. By distributing workloads across various platforms, businesses can ensure higher availability and reliability of their services.
Make sure you are regularly backing up data to a secure, separate location through BaaS which provides you with an additional layer of protection. DRaaS ensures that you have a comprehensive disaster recovery plan in place, allowing for quick failover to maintain business continuity.
Creating resilience with a multi-cloud strategy
Adopting a multi-cloud strategy can significantly enhance your IT resilience. Here’s why it’s beneficial.
- Increased resilience: Distributing services across multiple clouds reduces the risk of complete service failure.
- Flexibility and scalability: Multi-cloud environments allow businesses to scale resources up or down based on demand, optimising performance and cost.
- Enhanced security: By diversifying cloud providers, businesses can implement more robust security measures, reducing the risk of cyberattacks affecting all systems simultaneously.
How Redcentric can help to build out your cloud resilience
Where downtime is not an option, ensure your business remains operational even when the public cloud fails. At Redcentric we can complement your cloud offering, creating a multi-cloud solution where our IaaS platform can seamlessly take over, providing robust business continuity and scaling up computing power when needed allowing your organisation to benefit from:
- Scalable computing power: Easily scale your computing resources using Redcentric IaaS to handle peak loads and increased demand.
- Enhanced reliability: Depend on Redcentric’s high-availability infrastructure to keep critical applications and data secure and accessible.
- Seamless failover: Experience a smooth transition with automated failover mechanisms that activate Redcentric IaaS without any disruption.
The recent Microsoft outage serves as a crucial reminder of the importance of resilience in IT operations. By adopting a multi-cloud strategy and implementing robust disaster recovery and backup solutions, businesses can better prepare for unexpected disruptions. We are committed to helping you navigate these challenges and build a stronger, more resilient future for your business.
We know that downtime and disruptions can be incredibly stressful. If you need assistance or advice on how to implement these strategies, we are here to help. Our team is ready to support you in building a resilient IT infrastructure that can withstand future challenges. We are here to help you every step of the way.
