Long Term Storage of Gmail/Email using Mozilla’s Thunderbird (on Linux)

Posted on by rcragun

I have email going back to 2003. There have been times when I have actually benefited from my email archive. Several times, I have gone back 5 or more years to look for a specific email and my archive has saved me. However, my current approach to backing up my email is, let’s say, a borderline violation of Google’s Terms of Service. Basically, I created a second email account that I use almost exclusively (not exclusively – I use it for a couple of other things) for storing the email from my primary Gmail account. However, Google has recently changed its storage policies for their email accounts, which has made me a little nervous. And, of course, I’m always wary about storing my data with Google or other large software companies.

Since I have already worked out a storage solution for old files that is quite reliable, moving my old email to that storage solution makes sense. (FYI, my storage solution is to run a local file server in my house with a RAID array so I have plenty of space and local backups. Certain folders on the file server are backed up in real-time to an online service so I also have a real-time offsite backup of all my important files. In other words, I have local and offsite redundancy for all my important files.)

I’m also a fan of Open Source Software (OSS) and would prefer an OSS solution to any software needs I have. Enter Mozilla’s Thunderbird email client. I have used Thunderbird for various tasks in the past and like its interface. I was wondering if there was a way to have Mozilla archive my email in a way that I can easily retrieve the email should I need to. Easily might be a bit of a stretch here, but it is retrievable using this solution. And, it’s free, doesn’t violate any terms of service, and works with my existing data backup approach.

So, how does it work? And how did I set it up?

First, install Thunderbird and set up whatever online email account you want to backup. I’m not going to go through those steps as there are plenty of tutorials for both of them. I’m using a Gmail account for this.

Once you have your email account set up in Thunderbird, click on the All Mail folder (assuming it’s a Gmail account) and let Thunderbird take the time it needs to synchronize all of your email locally. With the over one hundred thousand emails I had in my online archive, it took the better part of a day to do all the synchronizing.

I had over 167,000 emails in my online archive account.

Once you’ve got your email synchronized, right-click on “Local Folders” and select “New Folder.” I called my new folder “Archive.”

Technically, you could store whatever emails you want to store in that folder. However, you’ll probably want to create a subfolder in that folder with a memorable name (e.g., “2015 Work”). Trust me, it will be beneficial later. I like to organize things by year. So, I created subfolders under the Archive folder for each year of emails I wanted to back up. You can see that I have a folder in the above image for the year 2003, which is the oldest email I have (that was back when I had a Hotmail address… I feel so dirty admitting that!).

The next step is weird, but it’s the only way I’ve been able to get Thunderbird to play “nice” with Gmail. Open a browser, log in to the Gmail account you’re using, and empty your trash. Trust me, you’ll want to have the trash empty for this next step.

Now, returning to Thunderbird, select the emails you want to transfer to your local folder and drag them over the “Trash” folder in your Gmail account in Thunderbird. This won’t delete them but it will assign them the “Trash” tag in Gmail. Once they have all been moved into the Trash, select them again (CTRL+A) and drag them into the folder you created to store your archived emails. In the screenshot below, I’m moving just a few emails from 2003 to the Trash to test this process:

Once the transfer is complete, click on the Local Folder where you transferred the emails to make sure they are there:

And there are the emails. Just where I want them. This also means that you have a local copy of all your emails in a folder exactly where you want them. At this point, you have technically made a backup of all the email you wanted to backup.

To remove them from your Gmail account, you need to do one additional thing. Go back to the browser where you have your Gmail account open, click on the Trash, and empty the Trash. When you do, the emails will no longer be on Gmail’s server. The only copy is on your local computer.

Now, the next tricky part to this (I didn’t say this was perfectly easy, but it’s pretty easy). Thunderbird doesn’t really store the Local Folder emails in an obvious location. But you can find the location easy enough. Right-click your Local Folder where you are archiving the emails and select “Properties.”

You’ll get this window:

Basically, in the Location box you’ll see the folder’s location. This is telling you where to find your Local Folder where your email is stored. On Linux, make sure that you have “view hidden files” turned on in your file manager (I’m using Dolphin), the location is your home folder, followed by your user folder, then it’s inside the hidden “.thunderbird” folder followed by a randomly generated folder that Thunderbird creates. Inside that folder, look for the “Mail/Local Folders” folder. Or, simply:

/home/ryan/.thunderbird/R$ND0M.default-release/Mail/Local Folders/
I have opened all the relevant folders on my computer in Dolphin see you can see the file structure.

Since I created an additional folder, there are two files in my Archive.sbd folder that contain all the emails I have put into that folder: “2003” and “2003.msf.” You can learn more about the contents of those files here, but, basically, the file with no file extension stores the actual contents of the emails. The .msf file is basically an index of the emails (I was able to open them both in Kate, a text editor, and read them fine). In short, you won’t have a bunch of files in your archive. You’ll have two files – one with the contents of the emails and one that serves as an index of the emails that Thunderbird can read. These are the two files that you’ll want to backup.

I’ll admit, this was the part that scared me. I needed to know that I could just take those two files, move them to wherever I wanted to ultimately store them and then, when needed, move them back into my Thunderbird profile and read them just fine. So, I tested it repeatedly.

Here’s what I did. First, I clicked on my backup folder in Thunderbird to make sure all of the emails were there:

I then went into the Local Folders directory and moved the files to where I want to back them up.

I then clicked on a different folder in Thunderbird and then hit F5 to make Thunderbird refresh the Local Folders. It took a couple of minutes for Thunderbird to refresh the folder, but eventually, it did. Then, I selected the 2003 Archive folder again and the emails were gone:

This is what I expected. The emails are in the “2003.msf” and “2003” files on my backup server. Now for the real test. I copied the two backed up files back to the Archive.sbd folder in my Thunderbird profile, selected the parent folder in Thunderbird, and hit F5 to refresh the folders again. It took a minute for the folder to refresh, but eventually, when I clicked on the 2003 folder and…

The emails are back!

It worked!!!

What this means is that you can put all of the email you want to back up into a folder; that folder is stored in your Thunderbird profile. You can then find the relevant .msf file and its corresponding data file, move them wherever you want for storage and, if needed, move them back to your Thunderbird profile (or, technically, any Thunderbird profile using the same folder structure) and you’ll still have all of your email.

This may not be the most elegant method for backing up your email but it’s free, it’s relatively simple and straightforward, and works reliably. Your email is not in a proprietary format but rather in an open source format that can actually be ready in a simple text editor. Of course, it’s easiest to read it in Thunderbird, but you have the files in a format that is open and secure.

BONUS:

If you don’t think you’re going to need to access your email on a regular basis, you can always compress the files before storing them. Given that most of email is text, you’ll likely realize a savings of close to 50% if space is at a premium (once I moved all of my 2003 email to storage, I compressed it and saw a savings of 60%). This will add a small amount of time to accessing the email as you’ll have to extract it from the compressed format, but it could mean pretty substantial space savings depending on how many emails you’re archiving.

EXTRA BONUS:

This is also a way to move emails between computers. I ended up using this approach to move my email archives to my laptop so I could go through my old email while I’m watching TV at night and delete all the emails I’ll never want in the future (you could of course do that with the email online before you archive it). I’m pretty good about deleting useless emails as I go, but I don’t catch them all. With the .msf file and its accompanying data file, I was able to transport my email archives to whichever computer I wanted and modify the file, then return it to my file server for long term storage.

 2,824 total views,  2 views today

transferring some email between Gmail accounts using Thunderbird

Posted on by rcragun

I have several Gmail accounts for various reasons.  Occasionally I have the need to transfer emails between the accounts.  The only way to transfer just some email between Gmail accounts is to use a desktop email client, like Thunderbird, Evolution, or KMail.  Unfortunately, the process isn’t all that straightforward and can be quite confusing at times.  I prefer using Thunderbird for transferring emails between Gmail accounts, so I’m going to show how to do this using Thunderbird.

To begin with, you need to enable IMAP support in your Gmail accounts.  You can see how to do that here.

Next, you need to set up at least two Gmail accounts in Thunderbird (I’m assuming you have it installed, if not, install it).  Thunderbird now makes this very straightforward.  When you first open Thunderbird, it will ask you to set up an account.  You can fill in your username and password at that time or do it later.  If you need to set it up later, you click on File -> New -> Existing Mail Account:

You’ll get this window.

Fill in your information and Thunderbird will take care of the rest.

Once you’ve got your accounts set up in Thunderbird, you need to let Thunderbird download the emails in the various accounts before you can start transferring.  To get this process started, click on “All Mail” in each of the accounts you are using and Thunderbird will start downloading the relevant data it needs to manage the emails.

Depending on how many emails you have, this may take a while.  Go get a snack or read a book.

Once Thunderbird has downloaded all the emails, you can begin transferring the emails between accounts.  However, here’s where things get tricky.  I’ve fiddled around with lots of different ways to make the transfer work, and the only one that seems to reliably work is weird.

First, open the Gmail account you are transferring files from [Gmail Account #1] in a browser (not in Thunderbird) and empty your trash.

Don’t try to empty your trash in Thunderbird as it doesn’t do anything. And you really want your trash empty for what we are about to do.

[NOTE: Scroll down to the section on the Large Email Problem before you continue.  No, really, go read that section right now as it will save you a lot of time later.]

Now, go back to Thunderbird. Select the emails you want to move in [Gmail Account #1] and, I know this is a little scary but, drag them to the trash in Thunderbird.

Once you drag them to the trash, they will no longer show up in All Mail, but they are not deleted, they are just in the trash (they have been labeled with the tag “trash”).  Once they are in the trash, select all the ones you want to move and drag them to the All Mail folder in the receiving account – [Gmail Account #2].

In the bottom bar in Thunderbird you’ll see a status update which will tell you how many files have transferred.

Once they have copied, you can check to see if they are in the receiving account [Gmail Account #2] by opening that account in a browser and using the search function to find those emails.  Once you’re positive that they have transferred, you can then go back to [Gmail Account #1] in your browser, open your trash, and empty your trash.  The emails are now in [Gmail Account #2] and are no longer in [Gmail Account #1].

Success!

Well, sort of.  There are a couple of issues you may run into.  First, it’s probably not a good idea to try transferring more than a few hundred to maybe a 1,000 emails at a time as the Gmail accounts have some limits on things like this.  Also, emails that are larger than about 5mb in size won’t transfer.  I don’t know why, but they won’t. other things you need to know before you try the above.

How to solve the Large Email Problem

There is a problem transferring large emails using this process.  The problem is attachments that are above 5mb in size won’t transfer between the accounts.  So, you can either forward those emails manually or delete the attachments.  Here’s how you find the attachments in Thunderbird:

Click on Tools -> Message Filters.

Once the Message Filters window comes up, click on New.

You’ll then get a new window, the Filter Rules window.  You need to name your Filter.  I called mine “large attachments.”  You’ll also need to indicate when you want the filter to run.  I set mine to manual only (see screenshot below).

Next are the filter rules.  What we’re looking for are attachments that are above 5mb in size.  Here’s how I set up my filter.  I wanted Thunderbird to first find all the emails that have attachments, so I added the following filter (note, this is probably unnecessary as the second filter will find these anyway):

Attachment Status = is = Has Attachments

I then wanted only emails that were above a certain size.  Since the offending emails are 5mb in size, I looked for all emails above 4000KB in size with the following filter:

Size (KB) = is greater than = 4000

 

Finally, I was interested in emails from just a specific date range, so I used a filter to only select emails before a certain date:

Date = is before = 01/01/2011

With the filter options in place, we need to tell the filter what to do with the emails that fit the criteria.  I tagged them with a tag I don’t use for anything else using the following in the “Perform these actions:” section of the Filter Rules window:

Tag Message = Later

 

Once you’ve got your filter set up, select OK.  You can then close the Message Filters window.  When you’re ready to run the filter, go back up to Tools -> Run Filters on Folder:

The filter will run and tag all the offending emails.  Then, from the top of Thunderbird, you can do a quick filter on the emails that you have in that folder by selecting on the “Tags” option and all the offending emails will filtered:

You can then decide if you want to delete the emails or forward them directly to the other gmail account (which means the date will be changed to the current date rather than keeping the original date, which bugs me).  Alternatively, you can just delete the attachment, which is done at the very bottom of Thunderbird:

NOTE: If you delete the attachment in Thunderbird, it doesn’t actually end up deleting the email.  It creates a copy without the attachment but leaves the email with the attachment.  You’ll then need to delete the email with the attachment before you transfer the files.

 

 

 1,682 total views,  2 views today