Mission Impossible: Complete Disaster Recovery for Google Workspace

I'm a frequent user of Google Workspace and even accept the switch from free to paid for my family domain. One topic has always been on my to-do list: Proper backups to support disaster recovery after a major problem.

It turns out that Google Workspace has a significant flaw: It is technically impossible to create a full backup of all data and to restore that! Google simply doesn't offer any API for that. As a result, all backup vendors are forced to work with the regular APIs. As a result, not everything in Google Workspace can be stored in a backup, e.g. Google Sites (new, not classic). Some content, like Google Drawings and others, can only be backed up as a static file (e.g. PDF) and not restored into a new Google Drawing.

Google itself doesn't offer much on backup and disaster recovery:

The white-paper doesn't even mention backup and disaster recovery as a recommended solution, it only lists standard recommendation to increase user account security.

The built-in solutions from Google only allow recovering deleted Drive content and Gmail content, if less than 25 days passed since deleting it. They don't allow a point-in-time recovery or help with large-scale file changes, e.g. after a ransomware attack. And there are no centrally managed solutions for other Google Workspace content.

Restore vs. Disaster Recovery

When thinking about the problem I like to discuss disaster recovery rather than backup and restore. The reason is the difference in the problem space:

  • Restore is required if some or all user data becomes unusable and needs to be restored into the same infrastructure or platform
  • Disaster recovery is required if the infrastructure or platform becomes unusable and all user data needs to be recovered either somewhere else or new infrastructure.

Any solution to the disaster recovery problem will also be able to cover the restore problem, but a solution to the restore problem won't necessarily also cover the disaster recovery problem.

Data Ownership

An additional aspect unique to Cloud services is the problem of "having" my data. While I might legally own the data that is stored in Cloud services, I am actually at the mercy of the vendor to have access to the data. To take control of my own data I want to have a copy of the data that I legally and technically own and that nobody can take away from me with the push of a button.

As a Google Workspace domain admin I also owe my users that level of data sovereignty. I wouldn't be surprised, if there is also a legal or compliance requirement for companies to have such a "fully owned" copy of their data.

Tool Vendors

I did a little vendor survey for backup & disaster recovery solutions and by large they all told me the same: Restore happens to a different file with a different document ID, and restored Google Docs, Sheets, Slides lack all "smart" content like embedded charts, Apps Scripts code or Add-On settings.

The change in the document ID is in my opinion the biggest problem of all: With the Google Workspace ecosystem we only share links to documents. These links are based on the document ID and we assume that they never change. This is true till such a file has to be recovered from the backup. Then it will have a new document ID and also a new link. And every place where this link was used will be broken and require manual updating.

Some vendors, like Afi.ai (from Cloud Architecture Center) offer somewhat better features and can restore document IDs. However, this only works if the file is restored to an existing file, thereby replacing it. If the original file has been deleted more than 25 days ago then the document ID is lost and also Afi.ai cannot restore it.

My Recommendation: NAS Appliance

The surprising conclusion of my research is that a NAS appliance from QNAP or Synology would probably be the best solution, at least for small domains like my family. All the backup vendors work with the same Google APIs and support more or less the same feature set, especially for the Google Workspace core services: Mail, Drive, Contacts and Calendar. With the NAS appliances the software is free, I would only pay for buying the storage. Additional benefits are the fact that the data is usable offline and doesn't need to be downloaded from the Cloud. This approach gives me a lot of confidence to keep my data, even if all the Cloud services suffer a major outage.

I'm not able to say which of the two NAS vendors offers the better solution, I hope to be able to try them out and compare them in detail.

I talked about this at the Workspace Arena: Backup-Lösungen für Google Workspace event on April 6th, 2022, you can watch the video (in German) or flip through the slides:

plus3trainings in Hamburg refers to this article in their write-up of Backup-Lösungen für Google Workspace (backup solutions) and I can happily recommend them as a partner for Google Workspace in Germany.

Are you interested in learning more about backup and disaster recovery, not only for Google Workspace, in your company? Together with my colleagues from Tektit Consulting I'll be happy to support you personally!

Comments

Post a Comment

Like this content? You could send me something from my Amazon Wishlist. Need commercial support? Contact me for Consulting Services.

Popular posts from this blog

Overriding / Patching Linux System Serial Number

Image
I'm a big fan of test driven development  (TDD) for infrastructure components. I'm currently working on a hardware-related topic where we also use the system serial number as identifier. To create a proper integration test, we need to be able to start a system and set the serial number to a known value. This can easily be done with the help of virtual machines like in VMware or VirtualBox , but I couldn't find a way for changing the system serial number on hardware boxes, cloud VMs (e.g. on Alibaba Cloud) or other Linux system. Problem Analysis I was thinking: Linux is the operating system where I can potentially do everything . So how hard can this be? After some digging around I found out that there are those main sources for the serial number on Linux: /sys/firmware/dmi/tables/DMI contains a binary blob of Desktop Management Interface data provided by the kernel and the dmidecode utility is commonly used to decode...
Read more

A Login Security Architecture Without Passwords

Image
Following up on Lifting the Curse of Static Credentials and Eliminating the Password of Shared Accounts , I have many discussions about why we would benefit from removing password prompts for website logins. Let's dig deeper into the details and show why removing password prompts leads to a  safer security architecture . Update 11.03.2022: Added more details about business vs. consumer websites and additional security suggestions surrounding WebAuthn Problem Space For context, imagine a website that needs to identify online users via their email address. We assume that the website in question is not the primary email system of a user but some other website, e.g. an e-commerce shop system or a collaborative productivity tool. As a User As a user of that website I want to easily sign up for an account have an easy way to login into the account be sure that my account is protected from others or attacks be able to easily recover access to my acco...
Read more

Fixing Chrome Color Printing on Linux with HP Color LaserJet M880

Image
It seems like I can't get a printer that "just works". I recently decided to replace our HP X476 printer with something nicer and bigger, an HP Color LaserJet M880 ( background story & review in German ). And of course there is something that needs fixing: The Chrome browser on Linux wouldn't let us print in color, even though all other applications had no problem to print in color. Even with Chrome color printing was possible, if one used the system printing dialog instead of the built-in Chrome print preview. This strange behavior of course piqued my curiosity. After some digging around I found out that the Chrome browser needs to parse and understand  the printer driver PPD! Chrome tries to find out how to configure color or grayscale printing in order to offer the user the choice. If Chrome can't understand the printer driver then it simply doesn't offer the choice between color and grayscale — and then some printer driver default can ch...
Read more