Boring servers are a feature

The best compliment a client can pay our hosting is that they forgot about it. Nobody calls to say the server was up all month. They just notice, eventually, that nothing has gone wrong in a while.

That quiet is something we work for. It comes from a handful of plain decisions that we make on every project, and from saying no to a lot of clever ones.

One project, one server

Every app we run gets its own server. Not a slice of a shared cluster, not a corner of someone else’s database. Its own machine, sized for what it actually does.

It sounds wasteful. In practice it’s the opposite. When something misbehaves, there’s exactly one place to look. A noisy neighbor can’t slow you down, because there are no neighbors. And when a client wants to leave, we hand them the keys to a machine they can understand.

Fewer moving parts

The fastest way to make a server unreliable is to add things to it. Every extra service is one more thing to patch, monitor and explain at 2am.

So we start small and stay there as long as we can:

• One database. Postgres, until there’s a real reason for anything else.
• One app process type, plus a background worker if the app needs one.
• No orchestration layer for a project that fits on a single machine.
• Plain logs shipped somewhere searchable, instead of a dashboard nobody opens.

When a project does outgrow this, we know exactly why, because we’ve watched it grow. That’s a much better place to make a big decision from than a guess on day one.

Backups you have actually restored

Almost everyone has backups. Far fewer people have restored one recently. A backup you’ve never restored is a hope, not a plan.

Once a month we pull last night’s backup onto a fresh machine, boot the app against it and click around. It takes about twenty minutes. Twice in the last year it caught a problem: once a missing file store, once a permissions change that would have made the restore fail. Both were dull to fix on a Tuesday afternoon. Neither would have been dull during an outage.

Alerts that mean something

An alert that fires every day teaches people to ignore alerts. We keep ours few and specific:

• The site is down, or slow enough that it may as well be.
• The disk is filling faster than it should.
• A background job queue has stopped draining.
• A certificate expires within two weeks.

If an alert fires and the right response is “that’s fine,” we change or delete it the same week. The goal is that every page is worth waking up for, which means almost none are.

Patching on a calendar

Security updates go on every Monday morning, on a schedule, with the same small checklist each time. Kernel updates that need a reboot get a short, announced window. Nothing about it is exciting, which is the point.

The most reliable server is the one with the fewest surprises left in it.

What it adds up to

None of these habits is impressive on its own. Together they mean that the servers we run are dull, predictable and cheap to look after. Our clients get to spend their attention on their product instead of on their infrastructure.

If your production setup keeps you up at night, it usually doesn’t need more tools. It needs fewer, and a steadier hand on the ones that are left.