A lesson learned the very hard way

Two nights ago, I took a quick look at a website I run with a few friends. It’s a sort of book recommendation site, where you describe some problem you’re facing in your life, and we recommend a book to help you through it. It’s fun to try to find just the right book for someone else, and it really makes you consider what you keep on your shelves.

But alas, it wasn’t responding well—the images were all fouled up, and when I tried to open up a particular article, the content was replaced by the text "GEISHA format" over and over again. So now I’m worried. Back to the homepage, and the entire thing—markup and everything—has been replaced by this text.

First things first: has anyone else ever heard of this attack? I can’t find a thing about it on Google, other than five or six other sites that were hit by it when Googlebot indexed them, and one of them at least a year ago.

So anyway, I tried to SSH in, with no response. Pop onto my service provider to access the console (much as I wish I had the machine colocated, or even physically present in my home, I just can’t afford the hardware and the bandwidth fees), and that isn’t looking good, either.

All right, restart the server.

Now HTTP has gone completely nonresponsive. And when I access the console, it’s booted into initramfs instead of a normal Linux login. This thing is hosed. So I click the “Rescue Mode” button on my control panel, but it just falls into an error state. I can’t even rescue the thing. At this point, I’m assuming I’ve been shellshocked.

Very well. Open a ticket with support, describing my symptoms, asking if there’s any hope of getting my data back. I’m assuming, at this point, the filesystem’s been shredded. But late the next morning, I hear back. They’re able to access Rescue Mode, but the filesystem can’t fsck properly. Not feeling especially hopeful, I switch on Rescue Mode and log in.

And everything’s there. My Maildirs, my Subversion repositories, and all the sites I was hosting. Holy shit!

I promptly copied all that important stuff down to my personal computer, over the course of a few hours, and allowed Rescue Mode to end, and the machine to restart into its broken state. All right, I think, this is my cosmic punishment for not upgrading the machine from Ubuntu Hardy LTS, and not keeping the security packages up to date. Reinstall a new OS, with the latest version of Ubuntu they offer, and keep the bastard thing up to date.

Except that it doesn’t quite work that well. On trying to rebuild the new OS image… it goes into a error state again.

Well and truly hosed.

I spun up a new machine, in a new DC, and I’m in the process of reinstalling all the software packages and restoring the databases. Subversion’s staying out; this is definitely the straw that broke the camel’s back in terms of moving my personal projects to Git. Mail comes last, because setting up mail is such a pain in the ass.

And monitoring this time! And backups! Oh my God.

Let this be a lesson: if you aren’t monitoring it, you aren’t running a service. Keep backups (and, if you have the infrastructure, periodically try to refresh from them). And keep your servers up-to-date. Especially security updates!

And, might I add, many many thanks to the Rackspace customer support team. They really saved my bacon here.

In which a scale is found tipping

There’s a certain aphorism that I’ve been thinking about lately: “there are twenty-four useful hours in every day” I don’t recall where I heard it first—as I recall, the version I heard growing up was “you’ve got the same amount of time in the day as the rest of us”—but I’ve realised two things about that first saying:

1. There are decidedly not twenty-four useful hours in every day. Depending on your particularly sleep needs, there are eighteen and, say, fifteen waking hours in every day. Then when you factor in time spent in transit between work/school and home, and mealtimes, along with basic hygiene needs, your time-per-day number drops considerably. I”d estimate around twelve. Your mileage may vary, depending on, well, the mileage between home and what you do to make a living.
2. I need more. I think this is why I’m reminded if my father’s different, truer version of the saying.

There are people in this industry, in this city, who love the freelancing life; people who love seeking out the Next Big Contract, and who get off on working late into the night to make a higher paycheque than the next guy. I’m not one of those people, as I’ve been discovering this year.

Don’t get me wrong. I like networking, and I like making things happens, and I like being able to say, “you know that thing that you use? I made that.” It’s a great feeling. But for the past few months, I’ve really felt like I’ve had at least twenty-four hours of work to do, every single day. So I perpetually feel like I’m behind. And that’s a pretty crappy feeling.

I read a book a few years back called The Hacker Ethic. It discusses the Protestant work ethic, which I think is a huge influence, in this city, of why people will willingly put in ten- or twelve-hour working days, five or six days a week. I don’t get that. I’ve been doing that for months, and it’s awful. All you’re thinking about is what you have to do. Your main focus is making more money. But why? So you can buy more things?

Seriously, everybody should read that book. I got into computers professionally because I love using them, and bending them to my will. But I also love my wife, and it’s important to find that work/life balance that keeps you sane and healthy.

This is becoming a bit of a rant, so I think I’d best cut it off. Too much work to do, anyway.