Server status: Off the cloud

A few weeks ago we had some erratic down time, followed by a migration to a dedicated server. Things are looking great now. Pingdom says our average response time has gone from 1500ms to 430ms.

A ton of traffic from a single IP address first knocked us offline during the morning, US time. Our ISP blocked it for us, but the next day the blog was offline again. At first our ISP thought it was an attack on the blog admin page, but later suggested a bad plugin.

We now suspect it was an influx of traffic from a link in the Instructables newsletter, probably in conjunction with a poorly working WordPress plugin. It was impossible to troubleshoot the problem on the live site because there’s so little debug data available. We did the proper thing and threw more hardware at the problem.

It would be great to take the Adafruit route and get a fully managed server. ServInt’s cheapest full server is $200 a month, that’s a little out of our range. A quality shared host could handle 90% of our needs, but there’s no good debug data and they tend to catch fire during a high-traffic slashdotting.  A Virtual Private Server works great for a lot of people, but we’re even more nervous about a shared host plus a virtualization layer. We originally moved to Laughing Squid’s Rackspace cloud sublet so the site could magically expand to more servers as needed, but the cloud turned out to be a shared hosting ghetto too.

Begrudgingly, we chose a dedicated self-managed server with Hetzner. Even though it’s cheaper than the cloud, it was the option of last resort. Server setup and management detracts from hardware fun. We’re on our own for all future problems, but we have access to all the debug info. For our current size and budget, this was the best option.  We setup the dedicated box with a ton of help from Arhi, and we retained a system admin to secure it and maintain it long-term.

Our new home is an Athlon X2 5600 with 4GB RAM and 400GB mirrored RAID array. Debian Linux is running a standard Apache/MySQL/PHP server. Automated off-site backups run every 24 hours.

Cloud Flare caches websites at datacenters all over the world, they’re handling more than half our traffic now. Pages load faster from a server close to you than if everyone tries to grab them from our box in Germany.

See the site changelog for a complete list of updates from the past two years.

Thanks to Laughing Squid for being a great host, providing great support, and helping us move. Thanks to Arhi for his  extensive server configuration help. Thanks to everyone for enduring the transition.

Leave a comment

Your email address will not be published. Required fields are marked *

Notify me of followup comments via e-mail. You can also subscribe without commenting.