Ransomware: Even backups may not save you

When the Colonial Pipeline ransomware attack became public in the first half of 2021, many were surprised that the company paid a $4.4 million ransom to recover its business systems. After all, it’s unthinkable that a company so large wouldn’t have backups in place. According to an article in the Wall Street Journal, however, Colonial’s CEO "authorized the ransom payment of $4.4 million because executives were unsure how badly the cyberattack had breached its systems, and consequently, how long it would take to bring the pipeline back."

Colonial didn’t say much more than this about why they decided to pay the ransom, but, assuming the company had backups systems in place, there are two possible scenarios. The first is that the backups, themselves were hit by the ransomware malware. This seems unlikely, as a large organization like Colonial likely follows the 3-2-1 maxim for backup: three copies of data on two different forms of storage media with one offsite. Certainly, it’s possible that there was still a logical connection between all backups and the production network, which would have allowed the attack to access backups. But typically, offsite backups are well protected against this kind of attack and are often stored in a read-only format that cannot be encrypted or overwritten by malware.

The importance of recovery time

It’s much more likely that recovery time was the issue. Unfortunately, while nearly every organization has some sort of backup system in place, far too many don’t consider how long it will take to recover. That’s  especially true in the case of a large-scale ransomware attack or disaster that destroys a significant portion of company data.

For a company like Colonial Pipeline, getting the business back up and running as soon as possible was the No. 1 priority. Not only was the company losing millions of dollars of revenue daily, but closing the pipeline sparked a panic that led to gas shortages throughout the Southeastern United States. There’s no way to know exactly what Colonial’s recovery time objectives (RTOs) were for the data it absolutely had to have in order to conduct business, But in my experience, I’ve seen systems at even very large organizations that would require several weeks to restore enough data so that the business could function.

So, when faced with the choice of paying ransom to recover data faster than an organization could accomplish with its backup systems, the cost of additional downtime amounts to a much larger loss. In that situation, financially,  it’s a no-brainer financial decision to pay the ransom. Though, paying a ransom dies raise a number of legal and ethical questions that companies should discuss with their counsel and ethics committees.

Different technologies for different workloads

To avoid having to pay ransom, organizations need to think about backup and disaster recovery (DR) from the point of view of recovery. While everyone wants to be back online as fast as possible, it’s important to categorize data and apps by their importance; it’s far too expensive to require an RTO of minutes or hours for every scrap of data in the organization. So it is critical to understand the company’s downtime tolerance for each workload, and then match those tolerances to the appropriate solution.

Options include:

• Immediate total recovery: This will require a synchronous hot site, which is by far the costliest approach. But if ransomware strikes, the company will experience, at most, a minor hiccup in operations. A good example here would be a case in which a museum’s HVAC control systems being offline would result in the destruction of priceless works of art.
• Continuous data protection: Backup vendors and backup-as-a-service (BaaS) providers have solutions that deliver RTOs of seconds to minutes. They’re less expensive than a synchronous hot site but more pricey than a typical backup system. For this category, think about a global just-in-time supply chain system. If it’s offline, a  truck or ship could wait a few minutes before leaving with a load without causing significant harm.
• Backup systems: There’s an enormous range of RTOs and pricing, depending on how the solution is architected and whether one is working with a managed service provider. They can range from less than an hour to days or weeks, if it’s not well designed. It’s critical to test the system to make sure it will deliver the RTO required for each tier of workload. A line-of-business application that does weekly batch processing of information could stand to be offline for hours, even days without any significant impact.
• Tape: This is the cheapest route but recovery can take hours for a relatively small amount of data, and if backups are stored offsite, it could be days. While not useful for large-scale recovery, tape is good for archiving data for compliance and legal matters. Many businesses have data they are required by law to retain for seven, 10, or even 20+ years where the only time objective for recovery is "within a reasonable time."

Finally, as organizations revisit their backup and DR strategies to ensure that they can meet the RTOs, they need to continue functioning in the face of ransomware or other disasters -- so they must not neglect regular testing. IT infrastructure is always changing. Companies need to make sure they have accounted for IT dependencies so workloads are recovered in the proper order and everything functions properly. And they need to ensure that new applications and data are protected with the RTOs they require.

Doing this in-house will require DR experts who can recommend the right balance of protection for each workload and corresponding expense. That’s a lot to expect from an IT staff that deals with disaster issues only on occasion. Additionally, when you factor in the time that goes into ongoing management, organizations might find the DIY approach can be pricier than using a managed service provider (MSP).

However an organization approaches backup and DR, they can’t take the approach that, if a backup is out there, we’re covered. Backups that can’t be recovered in a timely fashion are only slightly better than having no backup at all.

Image credit: AndreyPopov/depositphotos.com

Bret Piatt is CEO of OffSiteDataSync, a global provider of highly available and secure cloud data protection solutions, including Infrastructure (IaaS), Disaster Recovery (DRaaS) and Backup (BaaS). The company delivers best in class data protection and availability solutions built on market leading technology from Veeam, Zerto, VMware and Cisco.