ChrisStreeter
SF Cable Car Route Llamas, Alpacas and Mutants Fall Colors Mardi Himal Trail Golden Gate Zabriskie Point Prayer Flags Sunset Chinchero at Sunset Annapurna South in the Early Morning Fish Market Cementerio de Trenes de Uyuni Night Cats Sunset From Moro Rock Very rare deserted street Half Dome at Sunset Machhapuchhre Sunset from Marshall Beach, SF Grafitti The Danube at Night Annapurna South Under the Stars Mardi Himal Trail Rolling Fog Prepping to FIght Boudhanath Stupa Sheepherders and Annapurna South Cumhuriyet Anıtı Phewa Boats Alamo Square Bluebird Skies Stars Over Annapurna South White Mountains Sunset Fog Rolling In Half Dome at Sunset Vocano Scarred Mountains The Golden Gate at Night Old Man in Siding Twilight Fog Market Flags Bodie Eastern Sierras Under Shadow Snow and Shadows Temple Statue Marin Headlands Sand Dunes Hong Kong from the Peak Bay Area Sunrise Yosemite Fall Leaves Monument Valley Green River Canyons at Sunrise The Blue Mosque Buildings and a Bridge Cropped Crosswalk Headfirst Pisac Hillside The Buttermilks Rooster Fights at Forest Camp Resting Zabriskie Point Photographers Annapurna South

GMail IMAP Backup With mbsync on Ubuntu

April 30, 2009 —

Well, it sure has been a while since my last post on here. So I thought I'd kick it off with a discussion of how I went about getting my email backed up.

First, a description of my situation. I run all my email through GMail. I enjoy the interface and the fact that it is a cloud service; I can access my email seamlessly on my phone, my home computer, my work computer, some other computer, etc. However, I don't want to lose all that information. Google is great, but who is to say that something terrible won't happen and some (or all) of my mail is lost? So I wanted to setup some sort of backup. And then once I got that setup, make it automated.

At home, I run an Ubuntu box, that I just upgraded to 9.04, Jaunty Jackalope. This machine primarily serves as a media box, hosting video that streams to my Tivo off the 1.5TB RAID 5 array. I also use it as a network mounted TimeMachine box as well. Since I have extra storage on it, I figured I'd get something to sync my mail over IMAP periodically, and then I have a nice little backup.

After some searching, I came across two sites that had instructions using the utility mbsync (formerly isync). I found that following the instructions worked pretty well, though I had to customize the patch provided to get it to work with the version provided by Ubuntu. And then I thought I'd detail my steps here for others to see.

  1. First, enable IMAP in your GMail account.

  2. Install the dependencies for mbsync:

    sudo apt-get install libc6 libdb4.8 libdb-dev libdb4.8-dev libssl0.9.8 libssl-dev

  3. Get the source for mbsync on Ubuntu with:

    sudo apt-get source isync

    This will download the source for mbsync and create a directory (in your current directory) named isync-1.0.4 (Yes, I know it is called isync. That is a legacy name).

  4. Now the normal mbsync install works just fine. However, it doesn't support recursive directories. I use GMail labels setup to provide a hierarchy using forward slashes. It turns out that the fix to get mbsync to support directories like this is really easy to do and someone went through the trouble to get it to work. However, it doesn't quite work on the version of the code packaged with Ubuntu. So I had to modify the patch a bit. The patch I created can be found here. Download it and put it somewhere on your system. To apply the patch, change directories into the isync-1.0.4 directory that was just downloaded and run:

    patch -p1 < /path/to/patch/file/recursive_imap_ubuntu.patch

    Once you've patched the directory, configure (./configure), build (make) and install (sudo make install).

  5. Now you should have an executable mbsyncin your path. So it is time to start preparing to do the initial sync. Choose a place to store your backups. I chose

    /export/backups/mail/

  6. We want to connect to Google securely which means you'll need the latest SSL certificates. To get those, use the openssl client on your machine. Run:

    openssl s_client -connect imap.gmail.com:993 -showcerts

    which should show two blocks of

    -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

    in the output. You'll want to take each block (including the BEGIN/END CERTIFICATE lines), and put each of them into their own file. I put the first one in a file gmail.crt and the second one in the file google.crt (since the first signs imap.gmail.com which is signed by Google Internet Authority, the second certificate).

  7. The second certificate, the one for the Google Internet Authority, is signed by Equifax. So we'll need Equifax's certificate also. An as it turns out, Ubuntu has a copy of Equifax's certificate already sitting in the repositories. Just run

    sudo apt-get install ca-certificates

    to get the latest CA certificates. After installing the CAs, Equifax's CA sits at /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt, which we'll need in the configuration file in the next step.

  8. Now we can write the configuration file we are going to use. Here is a copy of mine:

    IMAPAccount gmail
Host imap.gmail.com
User yourusername@gmail.com
UseIMAPS yes
CertificateFile /export/backups/mail/gmail.crt
CertificateFile /export/backups/mail/google.crt
CertificateFile /usr/share/ca-certificates/mozilla/Equifax_Secure_CA.crt

IMAPStore gmail-remote
Account gmail

MaildirStore gmail-local
Path /export/backups/mail/gmail/
Inbox /export/backups/mail/gmail/Inbox

Channel gmail
Master :gmail-remote:
Slave :gmail-local:
# Exclude everything under the internal [Gmail] folder, except the interesting folders
Patterns * ![Gmail]* "[Gmail]/Sent Mail" "[Gmail]/Starred" "[Gmail]/All Mail"
Create Slave
Sync Pull
SyncState *

    Check out the Patterns line. That is where you would include or exclude various labels. All lables are stored at the root of the hierarchy, with the special directory [Gmail] having things like 'Sent Mail', 'Spam', 'Starred', etc in it. I wanted to exclude all the items in the [Gmail] directory except for the ones listed. The '*' at the beginning includes all other labels. You will also want to change the Path and Inbox lines to point to your mail location, as well as the first two CertificateFile lines. Also, be sure to enter your actual GMail login on the User line. Now save this file somewhere. Note: saving it as ~/.mbsyncrc will cause it to be automatically loaded when mbsync is run, meaning you don't need to specify which config file with the -c option.

  9. Now go ahead and test it out by listing the labels in your account with the command mbsync -l -c /path/to/the/configfile.rc gmail. Running it will look like this and ask you for your password:

    [streeter@scout]:~$ mbsync -l -c ~/.mbsyncrc gmail
Reading configuration file /home/streeter/.mbsyncrc
Resolving imap.gmail.com... ok
Connecting to 209.85.199.109:993... ok
Connection is now encrypted Logging in...
Password (yourusername@gmail.com@imap.gmail.com):
Channel gmail [Gmail]/Starred [Gmail]/Sent Mail [Gmail]/All Mail INBOX
[streeter@scout]:~$

    If you see something like this, then it worked! Now just go ahead and start your first mail download with mbsync -c /path/to/the/configfile.rc gmail And then you get to wait while it finishes which can take a while depending on how much mail you have.

You can go an extra step and save your password in the file. If you add the line Pass yourpasswordhere right after the User line, you won't be asked for your password and can then setup a cronjob to automatically keep mail locally on a schedule. However, your password for your Google Account then is stored in cleartext on your machine. So only do this is you absolutely know what you are doing and who can access the machine.

Hopefully this becomes useful for people.

Update 4/18/2011:

I just updated the directions slightly for Ubuntu 10.10 Maverick Meerkat. For Maverick, I had to also install libssl-dev to get it to work with recursive directories.