GMail IMAP Backup With mbsync on Ubuntu

April 30, 2009 —

Well, it sure has been a while since my last post on here. So I thought I'd kick it off with a discussion of how I went about getting my email backed up.

First, a description of my situation. I run all my email through GMail. I enjoy the interface and the fact that it is a cloud service; I can access my email seamlessly on my phone, my home computer, my work computer, some other computer, etc. However, I don't want to lose all that information. Google is great, but who is to say that something terrible won't happen and some (or all) of my mail is lost? So I wanted to setup some sort of backup. And then once I got that setup, make it automated.

At home, I run an Ubuntu box, that I just upgraded to 9.04, Jaunty Jackalope. This machine primarily serves as a media box, hosting video that streams to my Tivo off the 1.5TB RAID 5 array. I also use it as a network mounted TimeMachine box as well. Since I have extra storage on it, I figured I'd get something to sync my mail over IMAP periodically, and then I have a nice little backup.

After some searching, I came across two sites that had instructions using the utility mbsync (formerly isync). I found that following the instructions worked pretty well, though I had to customize the patch provided to get it to work with the version provided by Ubuntu. And then I thought I'd detail my steps here for others to see.

First, enable IMAP in your GMail account.

Install the dependencies for mbsync:

sudo apt-get install libc6 libdb4.8 libdb-dev libdb4.8-dev libssl0.9.8 libssl-dev

Get the source for mbsync on Ubuntu with:
```
sudo apt-get source isync
```
This will download the source for mbsync and create a directory (in your current directory) named isync-1.0.4 (Yes, I know it is called isync. That is a legacy name).
Now the normal mbsync install works just fine. However, it doesn't support recursive directories. I use GMail labels setup to provide a hierarchy using forward slashes. It turns out that the fix to get mbsync to support directories like this is really easy to do and someone went through the trouble to get it to work. However, it doesn't quite work on the version of the code packaged with Ubuntu. So I had to modify the patch a bit. The patch I created can be found here. Download it and put it somewhere on your system. To apply the patch, change directories into the isync-1.0.4 directory that was just downloaded and run:
```
patch -p1 < /path/to/patch/file/recursive_imap_ubuntu.patch
```
Once you've patched the directory, configure (./configure), build (make) and install (sudo make install).
Now you should have an executable mbsyncin your path. So it is time to start preparing to do the initial sync. Choose a place to store your backups. I chose
```
/export/backups/mail/
```
We want to connect to Google securely which means you'll need the latest SSL certificates. To get those, use the openssl client on your machine. Run:
```
openssl s_client -connect imap.gmail.com:993 -showcerts
```
which should show two blocks of
```
-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
```
in the output. You'll want to take each block (including the BEGIN/END CERTIFICATE lines), and put each of them into their own file. I put the first one in a file gmail.crt and the second one in the file google.crt (since the first signs imap.gmail.com which is signed by Google Internet Authority, the second certificate).
The second certificate, the one for the Google Internet Authority, is signed by Equifax. So we'll need Equifax's certificate also. An as it turns out, Ubuntu has a copy of Equifax's certificate already sitting in the repositories. Just run
```
sudo apt-get install ca-certificates
```
to get the latest CA certificates. After installing the CAs, Equifax's CA sits at /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt, which we'll need in the configuration file in the next step.
Now we can write the configuration file we are going to use. Here is a copy of mine:
```
IMAPAccount gmail
Host imap.gmail.com
User [email protected]
UseIMAPS yes
CertificateFile /export/backups/mail/gmail.crt
CertificateFile /export/backups/mail/google.crt
CertificateFile /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt

IMAPStore gmail-remote
Account gmail

MaildirStore gmail-local
Path /export/backups/mail/gmail/
Inbox /export/backups/mail/gmail/Inbox

Channel gmail
Master :gmail-remote:
Slave :gmail-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
Create Slave
Sync Pull
SyncState *
```
Check out the Patterns line. That is where you would include or exclude various labels. All lables are stored at the root of the hierarchy, with the special directory [Gmail] having things like 'Sent Mail', 'Spam', 'Starred', etc in it. I wanted to exclude all the items in the [Gmail] directory except for the ones listed. The '*' at the beginning includes all other labels. You will also want to change the Path and Inbox lines to point to your mail location, as well as the first two CertificateFile lines. Also, be sure to enter your actual GMail login on the User line. Now save this file somewhere. Note: saving it as ~/.mbsyncrc will cause it to be automatically loaded when mbsync is run, meaning you don't need to specify which config file with the -c option.
Now go ahead and test it out by listing the labels in your account with the command mbsync -l -c /path/to/the/configfile.rc gmail. Running it will look like this and ask you for your password:
```
[streeter@scout]:~$ mbsync -l -c ~/.mbsyncrc gmail
Reading configuration file /home/streeter/.mbsyncrc
Resolving imap.gmail.com... ok
Connecting to 209.85.199.109:993... ok
Connection is now encrypted Logging in...
Password ([email protected]@imap.gmail.com):
Channel gmail [Gmail]/Starred [Gmail]/Sent Mail [Gmail]/All Mail INBOX
[streeter@scout]:~$
```
If you see something like this, then it worked! Now just go ahead and start your first mail download with mbsync -c /path/to/the/configfile.rc gmail And then you get to wait while it finishes which can take a while depending on how much mail you have.

You can go an extra step and save your password in the file. If you add the line Pass yourpasswordhere right after the User line, you won't be asked for your password and can then setup a cronjob to automatically keep mail locally on a schedule. However, your password for your Google Account then is stored in cleartext on your machine. So only do this is you absolutely know what you are doing and who can access the machine.

Hopefully this becomes useful for people.

Update 4/18/2011:

I just updated the directions slightly for Ubuntu 10.10 Maverick Meerkat. For Maverick, I had to also install libssl-dev to get it to work with recursive directories.