When GitHub locked me out in September, I lost nothing. Every repository lived on my own disk first, and a git clone is the whole thing: every commit, every branch, every tag. I pointed the remotes somewhere else and carried on. I wrote about that in Six Days Without GitHub.
Mail doesn't work like that. If your mail provider locked you out tomorrow, or folded, or doubled its price, what would you walk away with?
What "take your data" gets you
On paper, quite a lot. Most providers will hand you an export, and in Europe they have to: Article 20 of the GDPR gives you the right to your data "in a structured, commonly used and machine-readable format." So you click the button and get a pile of files. A mailbox file or two. A folder of contacts. A calendar per calendar. Every one of them is in an open format, and every one of them opens somewhere else.
Then you import them, the mail is all there, and you think you're done. On Monday morning you find out what you left behind:
- Your old Sent folder is now just a folder called Sent, sitting beside the new server's real one, and your mail app is filing replies in the empty one.
- The filter that kept mailing lists out of your inbox is gone. So is the one that flagged mail from your bank.
- New contacts are landing in an address book you didn't know you had.
- Your signature is gone, and so are the two addresses you send from that aren't your login.
- A folder called "v1.0 notes" is now "v1_0 notes", because something on the way couldn't hold a dot.
None of that is mail. All of it is the account. It's the part you notice.
The formats aren't the problem
I want to be fair to the formats, because they're good. Maildir has kept one message per file since the nineties, and Dovecot will serve it as it sits. vCard holds contacts. iCalendar holds calendars. Sieve holds filters. They're open, they're old, and they work.
What none of them holds is the glue. A vCard file has no idea it belonged to an address book, let alone the default one. Two Sieve scripts on disk don't say which one was running. Maildir uses dots to separate folder levels, so a folder with a dot in its name has to change on the way down, and the real name is gone unless something writes it down. And nothing writes down which address you reply from, or what you sign with.
A pile of open files meets the letter of a portability law. Whether your account survives the trip is somebody else's problem, and right now that somebody is you.
So I wrote it down
inbuxa-migrate, the tool I built to move accounts onto inbuxa, learned to write an account out as plain files this week. Then I published the layout as a specification, so it belongs to the format and not to my tool.
It's deliberately boring. Mail goes in a Maildir++ tree, contacts in vCard, calendars in iCalendar, filters in Sieve and files as files. Then there's one small JSON file, manifest.json, for everything those formats can't hold: real folder names and roles, the default address book and calendar, calendar colors and time zones, which script was active, the identities with their signatures, and the real names of files that had to be renamed on disk.
A few rules do most of the work:
- Every part stands on its own. You don't need the manifest to read the mail. Point Dovecot at it. Import the
.vcfinto anything. The manifest adds to the files, and nothing in them depends on it. - The manifest is written last. No manifest means the export didn't finish, and a reader has to refuse it. "Did it finish?" gets a yes or a no.
- Missing isn't empty. "I didn't export your calendars" and "you had no calendars" are different statements, and the manifest says them differently.
- No secrets go in. There's no place for a password, a token or a key, and a writer must not add one. The new server sets its own.
- The name is an address. The format is named by the specification's URL, so no tool owns it and it can't collide with anything else.
No registry, no working group, no network. It's a directory you'll still be able to open with ls in ten years.
A format with one writer is a diary
Here's the embarrassing part. When I published it, the only thing that could read the manifest back was me, squinting at it. inbuxa-migrate wrote it, and then its own importers threw it away and read the open formats on their own. A format that only ever gets written isn't a format. It's a diary.
So that's what I fixed next. inbuxa-migrate 27.1-rc.9 added import files, which reads the whole export back, manifest and all. 27.1-rc.10 closed the last gap I knew about by keeping the real names of files that couldn't survive a disk. A message filed in three folders is written three times, because Maildir has no other way to say it, and it comes back as one message in three folders. Every path in the manifest is treated as hostile, because someone else wrote it. Nothing that climbs out of the export gets read, no symbolic link is followed, and no entry can rename itself on top of another.
The test I care about is the round trip. Export an account with everything the manifest exists for, read it back, export it again, and compare. The two exports come out the same, and that test runs on every change.
Why not just use JMAP?
I'm a JMAP person. inbuxa speaks it, and when both servers are up and talking, JMAP is how you should move an account: live, complete, no files in the middle.
But a move doesn't always have both ends up. The old provider is closing. Your account has been suspended and everything answers 404, which, I can tell you, is not a great moment to start writing a migration script. The new server doesn't exist yet. Or you just want a copy that outlives both of them, on a disk you own.
That's what this is for. It isn't a replacement for JMAP. It's what you keep for when JMAP isn't an option.
Where it falls short
Some of the limits come with the formats. Maildir gives each folder 26 letters for custom keywords, and a folder that needs more loses the rest. The manifest counts what was dropped, so at least you'll know. A message in several folders takes disk space in each.
The big one is mine. So far there's one writer and one reader, and I wrote both. A format becomes real when someone else picks it up and it works for them, and that hasn't happened yet.
The longer version, with the design decisions and the tests, is a white paper on coffeylabs.org.
What I'd like from you
If you build a mail server, a client, an archiver or a migration tool, read the specification and tell me what's missing. Write an exporter for your thing. Break my reader with an export it should have handled. The address is specs@coffeylabs.org, and inbuxa-migrate's code is there under Apache-2.0 or MIT if you'd rather borrow than write.
My code survived a lockout because git made every copy complete. Your mail should be able to say the same.