A fortnight on somebody else's server, and the cost of fixing the thing you were told about instead of the thing that is wrong.
We were called in to a server we had never touched before.
It is a hosting company's machine, carrying a couple of hundred small business websites. Not one of them is ours. Their customers were complaining and they wanted an outside pair of eyes. Everything below happened in about a fortnight, on somebody else's kit, with years of somebody else's history already in it.
Three alarms came in.
All three were real. Not one of them described what was actually wrong. That is not a criticism of whoever raised them. An alarm reports a symptom from where the customer is standing, which is the far end of a long chain.
Individually the sites looked fine. Together they crawled. One retail site was taking 41 seconds to answer.
The usual reading is that a heavy site is eating the box. The measurements said otherwise. Twenty four processor cores, ninety per cent idle, memory barely touched. Nothing was working hard.
The fault was a ceiling, not a load. The web server was capped at 150 worker processes and sitting flat on that cap, with requests banking up behind it. A queue at an open counter with the shutter half down. We lifted the cap and the 41 second site came back in just over 2.
Load looked fine and the box was unusable, both at once. Only one of those was on a dashboard.
Nobody reported the next one. We found it while we were in there: a community club's site serving an expired certificate. It had lapsed six weeks earlier, and visitors had been getting browser warnings that whole time.
The interesting part is why it never renewed itself. Certificates renew automatically on that platform. To prove the domain is yours, the system writes a small validation file into the site's folder. That account was over its disk quota, so it could not write the file. No file, no proof, no certificate.
What filled the quota was one error log. A single file, 16 gigabytes, the same error over and over from an abandoned plugin.
So the chain ran: plugin errors on every page view, log grows for months, log fills the account, full account blocks the renewal, certificate dies quietly. Five links, and only the last one was visible. We kept a sample, emptied the file, and the certificate reissued within minutes.
Then we went looking for the same pattern across the estate. There were 37 expired certificates on that server and roughly 215 gigabytes of runaway logs. We cleared about 227 gigabytes, and several sites got valid certificates back with no work of their own, including a second club broken for 50 days.
Freeing disk only rescues the certificates quota was blocking, though. The first pass took 37 down to 36. Less satisfying than a clean sweep, and it is what happened.
Then a forum on the same box started pulling the whole server down, and it looked exactly like a denial of service. Nearly 49,000 different IP addresses in one sample, from a long list of countries.
We went to the logs before the firewall, and the shape was wrong for an attack.
Six navigation pages were being hit within seven per cent of each other. People do not browse evenly, and attackers concentrate on the login page. Nothing touched the content: across more than 268,000 requests, not one read a discussion, a topic or an attachment. And 83 per cent carried a session ID in the address, which is what forum software hands a visitor that will not keep cookies.
That is not an attack. That is a crawler that cannot hold a cookie, given a brand new address for every link it follows, walking forever through a maze the forum builds in front of it.
It also explained why the obvious fix was impossible. At 1.2 requests per address there is no range to block, and blocking by country would have cut off real members. A password on the directory and a correction to how long sessions were held open did it. Load halved and the site came back.
In the middle of that we found what looked like a live compromise on another site. Repeated POSTs to a WordPress system file, rotating random parameters, forty identical 5.5 megabyte responses going out. That is the signature of data leaving a site.
We moved to block the source addresses, and the block failed, because all 24 of them were already in the allow list, in a labelled entry added months earlier. It was the site's own off site backup service, doing what it is paid to do. Had the block succeeded we would have quietly broken their backups, and nobody finds that out until the day they need one.
A safety rail caught that, not our judgement. It stays in the story for exactly that reason.
The last alarm was the sharpest. Three of a hospitality group's property sites returning an error to the public, reported as the server dropping them. Everything was checked from outside first, read only, before anyone logged in to anything.
The sites were up. All three answered normally when we asked the origin server directly. What visitors were hitting was the content delivery network sitting in front, refusing to pass anything through because it could not validate the origin's certificate.
Two of those certificates had quietly lost the property's own domain name. Renewal proves ownership by fetching a file from the domain, the domain now points at the network in front, the network cannot serve the file, so renewal drops the name it cannot prove and keeps the ones it can.
That loop feeds itself. No valid name on the certificate, so the network will not fetch, so the site is unreachable, so the next renewal has even less chance of proving anything. It never reads as a server fault, because the server is healthy and serving those sites perfectly to anyone who asks it directly.
We handed back three ways to fix it with the recurring cost of each one named. Ending the outage is easy. Ending it so it does not come back at the next expiry is the job, because that is exactly how one of those three died in the first place. Their own IT people took it from there, and all three sites are back up.
The method is not clever, only disciplined. Look before touching, from outside and read only, until the fault is understood. Measure instead of assuming. Fix the cause and not the alarm: the certificate was the alarm, the abandoned plugin was the cause.
Test the check as well as the system. One of ours quietly returned nothing that day, because it was reading the wrong filesystem, and reported a comfortable zero for the biggest problem on the box. We found it, said so, and fixed the check.
And write down what you could not verify. Every report we hand over carries that list, and clients tell us it is the part they trust most, because it is the part nobody writes.
Every one of these had been running that way for months, some of them for years, before anyone called us. The six week dead certificate, the 36 behind it, the 16 gigabyte log file underneath them: nobody was ignoring those. Nobody could see them.
So we built the report that would have. It runs against that server every Friday morning and covers certificate expiry, disk quota, runaway logs, renewal failures, capacity headroom and backup outcomes, and it is built so that a failed run reads as a failed run and never as a clean week.
That is the offer. Come in with fresh eyes, find what is actually wrong, fix the cause, and make the next one visible before it is urgent.
Client and site names are withheld by agreement. Every figure above is taken from the work itself.