Our public status page shows the same numbers we page ourselves on. Outside-in probes, watchdogs for slow-motion failures, surviving cloud maintenance, and the nightly copy that lives hundreds of kilometres away.
Every provider says "reliability is our top priority", usually in the apology. This post is the non-apology version: what we actually run to keep email.eu up, what we've built for the days something breaks anyway, and why our public status page shows the same numbers we page ourselves on. Part 6 of the series.
The foundational mistake in monitoring is asking your servers how they feel. Servers are optimists. The process is running, the port is open, the disk has space, and meanwhile a customer can't send mail because of a certificate hostname or a stuck queue two layers up.
So our monitoring is built outside-in. Probes run against every pillar the way a user would reach it: through the public addresses, over the real protocols, checking the responses a real client needs. The mail TLS watchdog from part 3 is the clearest example: a certificate can be valid and trusted yet miss the one hostname a strict mail client insists on, and only a check that connects like that client will notice, so that's how ours connects. The pattern, "monitor the promise, not the process", applies across the platform: probes sign in where a user would sign in and fetch what a user would fetch.
The same probes feed status.email.eu. There's no marketing layer between our alerting and the public page; the uptime figure you see is computed from the checks that wake us up. A status page that's hand-edited during incidents is a press release. Ours is an instrument.
Outages get all the attention, but the failures that actually erode a platform are the slow ones: a disk filling over weeks, a certificate renewal that drops a hostname, a nightly job that silently stops. For each of these we run a small watchdog whose only job is professional pessimism. One watches the mail certificates daily, as above. One watches database disk headroom and complains long before "long before it matters" becomes urgent. The pattern is deliberately dumb: a tiny independent checker per known slow failure, because a subtle monitoring system is one more thing that can subtly fail. Every time we identify a new slow failure mode, it gets its own watchdog, and the collection only ever grows.
Our cloud provider periodically replaces the machines under the Kubernetes cluster. Kernel patches, hardware maintenance, routine and unannounced-ish. In a managed cloud the floor moves, and it's our job to make that a non-event. So everything critical runs at least two replicas, and, just as important, the scheduler is told explicitly which pairs must never be down at the same time, so maintenance drains machines in an order that keeps every service quorate. The bar for a maintenance night is not "we recovered quickly". It's "nothing observable happened".
Redundant servers protect against machines failing. They don't protect against the bigger, rarer categories: a datacenter-scale event, or an operational mistake that faithfully replicates itself to every replica. For that, core customer data is copied every night to a second OVH region hundreds of kilometres away, on separate infrastructure.
A backup nobody watches is a hope, not a backup, so the mirror has its own probe: the control plane checks the freshness of the off-region copy and treats a stale mirror as an incident in itself, not something to discover during a disaster. We hold ourselves to the same rule here as everywhere: don't claim more than runs in production. As we add new products, wiring their data into the mirror is part of shipping, and the freshness checks are the enforcement.
What we promise is structure: two of everything critical, blast radii kept deliberately small, watchdogs that surface problems before customers do, a status page that can't flatter us, and straight answers on it when something does break. Reliability isn't heroics. It's the steady removal of every reason to need any.
Last in the series, part 7: the sixty seconds after you click sign up, which is where every pillar in this series has to work at once.