Over the past week things have been getting really slow and buggy around here. I reinstalled WordPress a bunch of times to get rid of the slow and timeout comment bug, but last night someone said it’s back.
A system admin has been working on it for the past few days. We found some corrupt database tables. That was causing down time when we do database backup and optimizations. The backup processes and Jungle Disk process would hit that spot too and take the server down.
We ran a disk check. SMART data shows the disk is ok, but pre-failure. A disk scan shows no corrupt blocks. That suggestes maybe the disk controller was bad. Just a few minutes ago our host Hetzner took the site offline and did a system check – no problems were found.
So… Now we’re left with a server going up and down about once every five minutes. Absolutely no concrete cause or solution in sight. Compound that with trying to troubleshoot over totally unbearable Chinese internet that now blocks foreign mail servers, as well as DP half the time, and we’re left with few good options.
My best guess is we will go offline for a while sometime today and transfer to a fully managed host on the US West Coast. We’re currently in a datacenter in Germany because of the strong privacy laws that protect system admins and users, but that’s of no use if I can’t admin the box from China.
Disk SMART reporting is never able to conclusively say that a disk is OK. a pre-failure status is a *definite* indication of a problem, usually with the disk itself, rather than the motherboard controller. Seeing the raw SMART data would allow a better diagnosis, but a pre-failure status is likely to be telling you that the disk *could* be part of (or all of) your current problems. Just my $0.02 based on the info so far. HTH
> We ran a disk check. SMART data shows the disk is ok, but pre-failure. A disk scan shows no corrupt blocks. That suggestes maybe the disk controller was bad.
Not correct. When the firmware starts noticing soft errors it moves the data to new blocks in the reserve table. If a lot of blocks get marked as bad all of a sudden then your disk has already experienced failure. A high-level disk scan will not report any bad blocks until things are already too late.
Whenever you notice SMART re-allocating blocks replace the disk immediately.
Ian, your site crashed when I post a comments. suspect the bug is still in the host. just move to another physical sever to check if this fix the problem. Sometimes, hardware problem could corrupt your databases :-(