Andy Jorgensen
As we saw July 19, when a security software update caused perhaps the biggest computer outage in human history, it is increasingly important to properly manage your organization’s relationships with service providers.
What happened?
CrowdStrike, a major computer security software and services vendor whose customers include major corporations and government agencies around the world, released an update to its Falcon XDR sensor agent that caused some Windows computers to fail to boot. This relates more to its role as a software vendor than as a managed service provider. However, many managed service providers sell CrowdStrike’s solutions under their own brand names.
Why is management of MSPs needed?
The metrics tracked in a ticketing system only tell part of the story. Outages and failures will occur. Organizations’ technology needs and strategies can vary widely. Friendly regional MSPs can be acquired by larger corporations causing disruptive changes. I have personally experienced two such acquisitions as a customer. In both cases, service fell off a cliff while they simultaneously raised their prices.
The managed service provider business model works on the principle of economies of scale, managing your wait times, maximizing resource utilization, and customer marketing. This is important to understand this because these are the forces that can potentially be at cross-purpose with your organizations’ interests.
Standardization vs. flexibility
A good MSP will accommodate your organization’s unique needs and priorities. However, an incentive exists for them to standardize their client base on the same stack of solutions. This reduces their overhead.
Stability vs. agility
Security services are perhaps the most valuable sector of the managed services business right now. The Falcon XDR platform at the center of the historic global IT outage, was apparently brought down by a rush to deploy an update that was ostensibly intended to increase the security of the systems that failed.
George Kurtz, the founder and CEO of CrowdStrike, said Friday on the Today Show, “Well, when you look at software it is a very complex world and there’s a lot of interactions and always staying ahead of the adversary is certainly, you know, a tall task. These sort of things, obviously, you try to understand and mitigate them in some cases you have a weird interaction. It didn’t seem like it happened on every Windows system. There are different versions and flavors and patch levels and we’re just trying to sort out where that negative interaction was.”
What he doesn’t mention is that testing updates against a wide variety of operating system versions and patch permutations not only takes time, but it also costs a lot of money.
One function that MSPs perform is making sure operating systems are current on their security updates and testing new updates before they are deployed in production environments. An event like this will hopefully cause organizations to scrutinize their operating system update management practices. Keeping things current is easy enough and MSPs often provide reports on the security updates deployed in client organizations. Less information tends to be provided on update testing procedures and results.
More information about exactly what happened will come out in the coming days. What is becoming clear is that this security software update did not work as intended on certain Windows operating system configurations. In my estimation, this could have been caught in testing by CrowdStrike.
Is your MSP using CrowdStrike? Are they in a position to potentially switch to another vendor? Just as companies can become locked-in to a vendor through complex interactions and integrations, so can MSPs. Make sure you have the internal capacity to evaluate, monitor, and change providers when necessary.
NCET is a member-supported nonprofit organization that produces educational and networking events to help people explore business and technology.
Andy Jorgensen manages IT for Catholic Charities of Northern Nevada and is NCET’S VP of Creative Services.
