Six Days Without GitHub: What inbuxa Became Anyway

On 20 September I published inbuxa Is Here, the long answer to why I forked a mail server I admire. The same day, GitHub suspended my account.

No notice. No reason. No email I could reply to. My profile, both organizations under it, and every repository in them started answering 404, and the container images stopped pulling. From the outside it looked as if the whole thing had been deleted, which was an interesting look on a launch day.

It hadn't been deleted, and nothing was lost. This is what happened next, and what got built while it was happening.

Why nothing was lost

Two habits turned out to matter more than I knew.

Every repository lived on my own disk first, with GitHub as the place it was published, not the place it was kept. And nothing in production pulled from GitHub: deploys push over SSH from a machine I control. So the suspension took away the front door, CI, release downloads and the image registry. It didn't touch the code, the history or anything running.

I've filed an appeal through the one form a suspended account can reach, and I'm not going to speculate about the cause here. I'd rather describe what I did about it.

Moving house in two days

For two days everything lived on a self-hosted GitLab, which worked, but was a lot of machine for what I needed. On 22 September it all moved to Gitea, running on hardware I rent and administer:

• every repository, with full history, at git.coffeylabs.org;
• CI on my own runners, container images in my own registry, and release binaries attached to tagged releases;
• ihasmail's GitHub issues, pull requests and discussions, rebuilt read-only as an archive, with the same numbers they had on GitHub, so old references still resolve;
• the links on this site, on inbuxa.org and in each project pointed at the new home.

If you had a clone from GitHub, change the remote to the matching path on git.coffeylabs.org and carry on.

The infrastructure, briefly

I've written about hosting before, so just the shape. There are three servers now, not two. They sit on a private WireGuard mesh, and each one backs up to the other two. The websites, this one included, are served from all three behind a load balancer, so losing a machine doesn't take a site down. Git, CI and the registry live on the same machines and are backed up the same way.

The more interesting part is mail, because mail is the product.

inbuxa, running itself

On 25 September production mail moved off a single server and onto a three-node inbuxa cluster: a shared PostgreSQL database with automatic failover, NATS to coordinate the nodes, and replicated object storage for message contents. All three nodes accept mail from the internet.

It was rehearsed first, on a copy of the real mail store, and the rehearsal found four bugs that would have hurt. Those were fixed before the real move, not after. On the night, inbound mail was refused for 43½ minutes while the store was copied over. Sending servers treat that as a temporary failure and retry, which is why a mail migration can have a window at all. In testing, a clean stop of the database leader failed over in about 6 seconds and a hard crash in about 30, with no acknowledged write lost.

That matters for one reason: I said in the launch piece that the only commitment device I trust is running my own mail on it. Scale-out storage, the headline feature that sits behind a paywall elsewhere, is now what my mail runs on.

What shipped since launch

Six days, eight dated releases. The highlights, grouped by what they do for you:

Clustering that holds up

• A dead node shows up in three minutes, not an hour. Nodes check in every minute, and the console's new Cluster Health card names any node that has gone quiet and for how long.
• A crashed node's work moves elsewhere within minutes, and a node shutting down cleanly hands its work back at once.
• A node joins the cluster as soon as its coordinator comes up, without a restart.
• Cluster roles decide which node does which kind of work, and changing a role takes effect without a restart.
• DMARC and TLS reports cover mail received on every node, not only the one that sends the reports.

A sturdier database layer

• A stalled database fails requests instead of hanging them, and the readiness check says so.
• On PostgreSQL, searching by address works as it always did on the built-in index, and search finds words inside links and file names.
• PostgreSQL search stays fast as mail arrives.
• Export and import keep everything: deleted items you can still restore, spam training data and the trained spam model all survive a move between databases.

Fewer ways for one mistake to stop everything

• One broken setting no longer blocks every other settings change, and a DNS hiccup can't fail a reload.
• One bad row can't stop the task queue.
• Logging and tracing changes apply without a restart.

AI spam filtering, local and per node

• The console has a Local AI page that sets the classifier up, guided or by hand. It's off until an administrator turns it on.
• The webmail shows the model's opinion on a message, labeled as one signal among several, never the verdict.
• As of today it runs on my own cluster, with one model on every node. Each node asks its own model, so a message's text never leaves the machine that received it.

The webmail on every node

• The webmail runs on all three machines, each copy talking to its own mail node, so losing one machine no longer takes webmail down. People who were on it sign in again.
• Settings › About now says which webmail node and which mail node you're on, and presents inbuxa as the whole suite it is: server, console and webmail.

Getting it installed

• The installer exists now. It surveys the machine, shows a dry run of every file, container, port and DNS record before touching anything, offers to install what's missing, and does real container installs. It's early, and it says so.
• Tagged releases publish multi-architecture container images and attach binaries for installs that aren't containers.

Staying an honest fork

• The fork's own identifiers are its own: protocol and configuration names now carry the inbuxa name, and it's written in lowercase everywhere people see it.
• A CI job opens an issue whenever upstream publishes a release not yet imported, and every import still passes through the strip tool. The latest upstream release is already in.
• CI fails if the upstream name creeps into a new user-visible string, and it checks that every changed file keeps its copyright notices.
• The docs have a page listing what changes name for anyone coming from upstream.

Where that leaves things

Losing GitHub on launch day cost me a front door, two days of plumbing and some sleep. It also made me finish the self-hosting I'd been putting off, which is the same argument inbuxa makes about mail. If a platform can switch you off without a word, it's worth knowing ahead of time exactly what still works when it does.

Everything still works.

• inbuxa: inbuxa.org, with the docs at docs.inbuxa.org
• The code: git.coffeylabs.org/inbuxa
• Me: @[email protected], or the feed