Tech
Mail, an Odyssey
08/03/06 20:44
A
longtime member of my list of
Things To Do has
been to move my home email server from my
UW-IMAP/mbox based server to a Courier-IMAP/Maildir
based system. Earlier this weekend, I came to the
realization that I am no longer interested in
maintaining a linux server. The bloom is off the
rose. I have a perfectly good (in fact, amazingly
good) hosting company in Dreamhost. I'll let the
pros, who I'm paying anyway, worry about server
uptime, security patching, backups and all the rest.
Dreamhost already has Courier set up so all I need to
do is get my existing mail onto their servers.
One small problem:
I have on the order of 10,000 emails going back six years, all in mbox format.
Attempt 1: Oh, I'll just run my home imap server and my dreamhost imap server side by side in Mail.app, and copy the emails over.
Er...
Boy that's slow.
And boy is Mail.app bogging badly.
Holy CRAP that's slow!
And now my dreamhost terminal connection is barely responding.
ABORT, ABORT, ABORT!
Deep breath. OK.
Attempt 2: So clearly IMAP to IMAP communication isn't going to work well. So, what else can we do? Well, I can just scp the mbox files to my local machine,
import them into Mail.app as local messages, and upload them from here to dreamhost...
Hmmm. Well, mbox import sucks in Mail.app. But I'll wait it out, because I'm sure it will be a smokin' upload.
taps fingers...
goes off to do other things...
And we're back. OK, everything is loaded into mail. Admittedly, the CPU is pegged and the machine is barely usable, but that's because Spotlight is indexing all this temporary content like a coked up rabbit.
So, kill Spotlight's various processes and hope that the machine doesn't hang. Well, it doesn't hang per se, but it has stopped accepting click events.
Expletive Removed..
OK, log out, and log back in. Miracles of miracles, the UI works and the email is locally available. Ready to try to upload. Here we go...
Err....
It's slower than the IMAP to IMAP copy?!?
I'm struggling with how this is even technically possible...
ABORT, ABORT, ABORT!
Deep breaths. Deep, Deep breaths. OK.
Attempt 3: OK, Mail.app clearly sucks at this sort of thing. Let's download Thunderbird, which makes my eyes bleed, but has a reputation for a pretty solid IMAP implementation.
Local mbox import is a dream, performance-wise, but pretty technically obscure. Always knew that CS masters would come in useful at some point.
Try uploading again.
After about 10 minutes, my head is in my hands, and I'm quietly sobbing. I hit the cancel button.
Attempt 4: Now, it's pretty clear that we've established that IMAP, whatever it's virtues, is not a protocol that sings when dealing with bulk operations. So why don't we use protocols that are. I scp my mbox files up to the dreamhost server.
Er... Now what? Part of the whole point of moving them over via IMAP copy was to convert from mbox to Maildir.
Well, there is an mb2md perl script for converting from mbox to Maildir. But first things first. If I'm going to have to deal with all this stuff locally, it's time to do some research on exactly how the Maildir file format works...
2 hours of googling
OK. Now I that I have a vague understanding of how it works, let's run the script.
nothing happens
Hmmm. Is it working? It's clearly doing something, since my shell is bogged down, and top says it isn't dormant. But I'm a funny sort of user: I like to have feedback.
ctrl-c
I dug around and found the inner loop in the perl script, and stuck a print statement in, which made me feel better.
The entire conversion took about 40 minutes, making me more sympathetic to the plight of IMAP, and causing me to pause and reflect on the confounding usefulness of the perl programming language.
So, now I've got my email in the right format, but not in the IMAP Maildir directory. So let's just copy them in.
Er...
Holy crap that copy is slow.
Is this stuff network mounted or something? shrug Well, whatever the case, it's not getting faster than this.
An hour later, the copy is complete, and I'm even more willing to let IMAP's problems slide.
OK, now that everything is in, its time to log in with Thunderbird and start reorganizing my imported email...
Select 100 mails, create a new folder, and move them over.
And wait.
And wait.
And... server time out.
G*DD**MIT!
OK, OK. Look at the folders. Well, the messages were moved (at least some of them), but they weren't deleted from the original folder.
Now I'm in a really terrifying state, where I don't know how inconsistent the data has become.
ABORT, ABORT, ABORT!
Attempt 5: The key realization hits me: mutt, my favorite term-based email client, speaks Maildir! Dreamhost has it installed, of course. I put the appropriate hieroglyphs in my .muttrc, and blam, I'm slicing and dicing my email locally, with only minimal bog.
Furthermore, mutt has built in functionality for removing redundant emails from email directories, so I can just organize as I wish, and not worry about creating duplicates since I can open the folders up and remove them easily. Beautiful.
Now if I had just realized all that back at attempt 1.
One small problem:
I have on the order of 10,000 emails going back six years, all in mbox format.
Attempt 1: Oh, I'll just run my home imap server and my dreamhost imap server side by side in Mail.app, and copy the emails over.
Er...
Boy that's slow.
And boy is Mail.app bogging badly.
Holy CRAP that's slow!
And now my dreamhost terminal connection is barely responding.
ABORT, ABORT, ABORT!
Deep breath. OK.
Attempt 2: So clearly IMAP to IMAP communication isn't going to work well. So, what else can we do? Well, I can just scp the mbox files to my local machine,
import them into Mail.app as local messages, and upload them from here to dreamhost...
Hmmm. Well, mbox import sucks in Mail.app. But I'll wait it out, because I'm sure it will be a smokin' upload.
taps fingers...
goes off to do other things...
And we're back. OK, everything is loaded into mail. Admittedly, the CPU is pegged and the machine is barely usable, but that's because Spotlight is indexing all this temporary content like a coked up rabbit.
So, kill Spotlight's various processes and hope that the machine doesn't hang. Well, it doesn't hang per se, but it has stopped accepting click events.
Expletive Removed..
OK, log out, and log back in. Miracles of miracles, the UI works and the email is locally available. Ready to try to upload. Here we go...
Err....
It's slower than the IMAP to IMAP copy?!?
I'm struggling with how this is even technically possible...
ABORT, ABORT, ABORT!
Deep breaths. Deep, Deep breaths. OK.
Attempt 3: OK, Mail.app clearly sucks at this sort of thing. Let's download Thunderbird, which makes my eyes bleed, but has a reputation for a pretty solid IMAP implementation.
Local mbox import is a dream, performance-wise, but pretty technically obscure. Always knew that CS masters would come in useful at some point.
Try uploading again.
After about 10 minutes, my head is in my hands, and I'm quietly sobbing. I hit the cancel button.
Attempt 4: Now, it's pretty clear that we've established that IMAP, whatever it's virtues, is not a protocol that sings when dealing with bulk operations. So why don't we use protocols that are. I scp my mbox files up to the dreamhost server.
Er... Now what? Part of the whole point of moving them over via IMAP copy was to convert from mbox to Maildir.
Well, there is an mb2md perl script for converting from mbox to Maildir. But first things first. If I'm going to have to deal with all this stuff locally, it's time to do some research on exactly how the Maildir file format works...
2 hours of googling
OK. Now I that I have a vague understanding of how it works, let's run the script.
nothing happens
Hmmm. Is it working? It's clearly doing something, since my shell is bogged down, and top says it isn't dormant. But I'm a funny sort of user: I like to have feedback.
ctrl-c
I dug around and found the inner loop in the perl script, and stuck a print statement in, which made me feel better.
The entire conversion took about 40 minutes, making me more sympathetic to the plight of IMAP, and causing me to pause and reflect on the confounding usefulness of the perl programming language.
So, now I've got my email in the right format, but not in the IMAP Maildir directory. So let's just copy them in.
Er...
Holy crap that copy is slow.
Is this stuff network mounted or something? shrug Well, whatever the case, it's not getting faster than this.
An hour later, the copy is complete, and I'm even more willing to let IMAP's problems slide.
OK, now that everything is in, its time to log in with Thunderbird and start reorganizing my imported email...
Select 100 mails, create a new folder, and move them over.
And wait.
And wait.
And... server time out.
G*DD**MIT!
OK, OK. Look at the folders. Well, the messages were moved (at least some of them), but they weren't deleted from the original folder.
Now I'm in a really terrifying state, where I don't know how inconsistent the data has become.
ABORT, ABORT, ABORT!
Attempt 5: The key realization hits me: mutt, my favorite term-based email client, speaks Maildir! Dreamhost has it installed, of course. I put the appropriate hieroglyphs in my .muttrc, and blam, I'm slicing and dicing my email locally, with only minimal bog.
Furthermore, mutt has built in functionality for removing redundant emails from email directories, so I can just organize as I wish, and not worry about creating duplicates since I can open the folders up and remove them easily. Beautiful.
Now if I had just realized all that back at attempt 1.
|