Why I left

On the 9th of September DigitalOcean joined the Omacom Foundation as a founding corporate patron: three million dollars, a million a year for three years, for the development and the spread of Omarchy.

Omarchy is DHH's opinionated Arch Linux setup. I have nothing against bringing Linux to the masses and I haven't checked if it's really bad work, but let us be clear about what three million dollars is buying here: a curated pile of configuration files, an installer and a theme. There are maintainers of libraries in everybody's dependency tree who have never seen three thousand.

The money does not go to a pile of configuration files, though. It buys a seat next to the man who writes them, and that is a man who spent the last year explaining that London has become unrecognisable because only a third of it is "native Brit", praising Stephen Yaxley-Lennon, who goes by "Tommy Robinson", and describing the crowd at a far-right London rally as "perfectly normal, peaceful Brits" worried about demographic change. Tekin Süleyman collected the record a year ago, with the receipts, and nothing since has walked any of it back. Call that what it is. I do: it is racism, and a company that writes an eight-figure cheque into that orbit has decided it is comfortable there.

Not sending racists money is an absolute minimum I did not expect to have to explain. I am not interested in the argument about whether sponsoring somebody's Linux distribution is an endorsement of his politics (it is), or in the difference between a bad opinion and a bad person, or in how good the product is. The product (DigitalOcean, not Omarchy) is good. That is the point of writing this down: the service was excellent and I left anyway, because paying a monthly invoice is funding, and I decide who I fund.

You can disagree with where I draw that line. I would rather be wrong about where the line sits than have no line, and I have noticed that people who insist the line is complicated usually just want to keep their setup.

There is one detail in that announcement I keep coming back to. DHH writes that the deal went from first contact on a Wednesday to a signed agreement by Sunday, and calls it "the speed of collaboration that's possible with X, agents, and boundless ambition". Four days to decide to put three million dollars behind this. It took me one to leave (plus waiting for DNS, of course).

Which is the part worth a technical post. Almost everybody I have talked to about this agrees with the principle in the abstract and then explains why moving is unrealistic for them: the servers are pets, the DNS records were clicked into a web console over eight years, and nobody remembers why the mail flow works. That is not a moral problem, it is an architecture problem, and it is the reason a provider can behave badly without losing customers.

And it's usually overkill for your private server.

What made it cheap

My server is one small virtual machine that hosts 26 sites, mostly static, a few PHP, some with a database. Hobby scale. The setup around it is deliberately larger than the thing it manages, and this week was the payoff.

Everything is infrastructure as code (IaC), split into two layers that do not know much about each other:

• OpenTofu (Terraform) owns the provider side: the machine, and every DNS zone and record for every domain. The zones are not a copy of what is in a web console somewhere, they are the source, and each site is a module call with a handful of variables (has_www, has_php, docroot, mx).

• NixOS plus morph owns everything inside the machine: Caddy and the vhosts, the bare git repositories, the users, the timers for backups and the newsletter. Secrets live encrypted in the repository with sops and age, and are uploaded at deploy time.

Backups run nightly into Google Cloud Storage, on purpose with a different provider than the one running the server, which turned out to matter less for this migration than for my general peace of mind, but it did give me a reference for what the new machine had to end up containing.

Two commands do everything. make apply runs tofu apply and then pushes the NixOS configuration to every host in the network with morph (I've run this on larger clusters, but also works for a single server, obvioulsy). sops secrets/secrets.json edits the secrets. There is no third place where state lives, which is the entire reason this was a one-day job: I did not have to find out what my server does, because the answer is in my repository.

DNS first, then the server

The DNS half: every site module emits its records to both providers at once. One tofu apply created 21 zones at Hetzner next to the DigitalOcean ones, with identical content, pointing at the same old address. A zone that nobody delegates to is inert, so nothing happened. Then I verified each zone against Hetzner's nameservers directly:

and switched the nameservers at the registrar, one domain at a time. The DigitalOcean zone kept serving identical answers throughout, so rolling a domain back was a second registrar change and nothing else.

The server half: Hetzner Cloud had no NixOS image when I checked a year ago (now it has, and I should've checked again) and no image upload API, so my old routine of building a qcow2 image by hand in a local virtual machine and uploading it does not carry over. The server boots stock Debian and rewrites itself into NixOS with nixos-infect from the user data. The bootstrap image barely matters, because it only has to produce a bootable NixOS that accepts my root key. Everything real arrives afterwards from make apply, which is the same command that deploys the old host.

Copying the data is a script that reads both addresses out of the terraform state and then runs rsync on the old droplet, pushing to the new server through a forwarded SSH agent, so the roughly 30 GB never travels through my laptop. Every phase is idempotent, which is what makes the schedule relaxed: a full pass days ahead, then a short delta at the cutover:

And then the cutover itself is one line. Every site module derives its address from a single local:

Changing which of those two lines is commented out re-points all 26 sites in one apply, and reverting is the same one-line change.

What did not come for free

It would be a nicer story without this section, so here it is.

Two provider models differ in ways the abstraction does not hide. DigitalOcean creates the apex A record implicitly as part of creating a domain and Hetzner does not, so the module has to create it explicitly. DigitalOcean was also hosting a subdomain of mine as a zone of its own next to its parent; self-delegating that at Hetzner would be pointless, so those five sites became records inside the parent zone instead. Both are small, and both are exactly the kind of thing that would have been an afternoon of confused clicking without a module to change in one place. Two Terraform providers are never the same, you might have stumbled across this.

The sharper problem was that for a few days I had two live servers. My timers are single-owner jobs: they upload to one fixed object path, send real mail, or dispatch a GitHub workflow. Running the backup on the empty new machine would have overwritten an archive that by design keeps no history, and the newsletter would have gone out to my subscribers twice. So one value in the configuration names the host that runs timers, the units stay installed everywhere so I can still trigger one by hand. That was the part I actually had to think about, and it is the part IaC did not solve for me.

A few smaller things: the git host keys have to move with the server or every clone url starts screaming, so that is its own phase in the script. The old droplet is powered off rather than destroyed, because its disk is the only place a write missed by the final sync could still be recovered from (and then I almost deleted it immediately and purged my account anyways, because I was pissed about DigitalOcean). And while reading through zones I had not looked at in years I found a Sender Policy Framework (SPF) record that included two domains which have no SPF record at all, making the whole thing a permanent error and, therefore, invalid. It had presumably been broken for a long time. Migrations are a good excuse to read your own configuration.

Summary

DigitalOcean gave me years of good service and I still left within one day of deciding to, because the servers are not pets and the DNS is not a web console full of forgotten clicks. That is worth saying plainly: the freedom to act on a principle is a technical property of your setup, bought in advance, usually for reasons that have nothing to do with ethics. I automated all of this because I am lazy and forgetful. It turned out to also be what let me act like somebody with a spine, on a Monday, while waiting for a surgery.