Annual health check across 37 live systems: three problems nearly every project has

The short version: between late October and mid-November we ran an annual health check on 37 systems still in production, covering backups, certificates and dependencies, accounts and permissions, capacity and logs, and emergency plans. The result was better than expected and worse than expected: no system was close to failing, but three problems turned up almost everywhere — and none of them fail immediately.

The trigger was mundane. On a project two years into production, the client's IT lead changed and nobody could say which services ran on the server or when the certificate expired. Nothing was broken; nobody was confident either. Handover documents, we realised, are rarely the current version two years later. Hence a full sweep.

37 live systems: what we checked, what we found Five checks · every one with a pass criterion Backup policy and restore test 37/37 backing up SSL certificates and domains 11 expire in 90 days Accounts and leavers 14 with shared logins Capacity, logs, disk headroom 2 above 80% Emergency plan and contacts all refreshed Three recurring problems · systems affected Shared or stale accounts 14 Certificates nobody watches 11 Backups never restored 9 The three can coexist in one system, hence a total above 37. None of them is a software defect; all are ownership gaps after go-live.
Green = the area is broadly healthy; amber = specific items need action

1 · Five checks, each with a pass criterion

CheckPass criterionIf it fails
Backup and restoreDaily full plus incremental; at least one real restore in the past three monthsRestore to a scratch database on the spot and reconcile row counts and totals
Certificates and domainsMore than 90 days remaining; expiry alerts sent to a current employeeRenew on the client's behalf or configure automatic alerts
Accounts and permissionsNo shared logins; leavers disabled; at most three administratorsProduce a permission matrix, adjust after client confirmation
Capacity and logsDisk below 80%; logs kept 90 days or more; errors reviewed dailyAdd storage, archive, configure rotation and alerting
Emergency planWritten steps for power loss, network loss, device failure; phone numbers answeredRewrite a one-page plan and walk through it on site

2 · Three recurring problems, all of them silent

  1. Certificates nobody watches. Eleven systems expire within 90 days. An expired certificate does not stop the server; it makes every browser show a red warning and stops the business — usually on a Monday morning. The root cause is ownership: registered in whose name, renewal notices to whose mailbox, and after handover nobody owns it.
  2. Backups run every night and have never been restored. Nine systems. Files exist and sizes look right, but when we ran restores, two backups were incomplete — the script had errored and only part of the tables had been written. An unverified backup is not a backup.
  3. Shared accounts and leavers still enabled. Fourteen systems, typically a shared admin login with the password taped to the monitor. Not a technical problem but a management one — and when an audit asks, the system owner answers for it.
All three share one trait: they never fail immediately, and when they fail, they fail big. Operations is not about fighting fires; it is about making sure they do not start.

3 · What we changed in the handover package

  • The checklist ships with the deployment documents, so client IT can run the same five checks every quarter.
  • Certificate expiry reminders 30 days ahead — we register the date and notify, instead of waiting for a red browser bar.
  • Restore drills written into operations contracts — at least one real restore a year, with a written record, which also serves as audit evidence.
  • A role-to-person permission sheet at go-live, updated when staff change, so accounts do not travel with people.

4 · A five-minute self-check

You do not need a vendor for this. Four questions tell you whether there is latent risk:

  1. When did you last actually restore a backup? If nobody can answer, that is the risk.
  2. How many days until your certificate and domain expire, and whose mailbox gets the notice?
  3. How many shared accounts exist, and were leavers' accounts disabled?
  4. If the server does not start tomorrow morning, who is the first call — and does that number still get answered?

Planning a similar production line or enterprise system?

Start with our pricing logic, then let's talk requirements — cost structure, technology comparison and acceptance criteria, all laid out in the open.