A lesson learned the very hard way

Two nights ago, I took a quick look at a website I run with a few friends. It’s a sort of book recommendation site, where you describe some problem you’re facing in your life, and we recommend a book to help you through it. It’s fun to try to find just the right book for someone else, and it really makes you consider what you keep on your shelves.

But alas, it wasn’t responding well—the images were all fouled up, and when I tried to open up a particular article, the content was replaced by the text "GEISHA format" over and over again. So now I’m worried. Back to the homepage, and the entire thing—markup and everything—has been replaced by this text.

First things first: has anyone else ever heard of this attack? I can’t find a thing about it on Google, other than five or six other sites that were hit by it when Googlebot indexed them, and one of them at least a year ago.

So anyway, I tried to SSH in, with no response. Pop onto my service provider to access the console (much as I wish I had the machine colocated, or even physically present in my home, I just can’t afford the hardware and the bandwidth fees), and that isn’t looking good, either.

All right, restart the server.

Now HTTP has gone completely nonresponsive. And when I access the console, it’s booted into initramfs instead of a normal Linux login. This thing is hosed. So I click the “Rescue Mode” button on my control panel, but it just falls into an error state. I can’t even rescue the thing. At this point, I’m assuming I’ve been shellshocked.

Very well. Open a ticket with support, describing my symptoms, asking if there’s any hope of getting my data back. I’m assuming, at this point, the filesystem’s been shredded. But late the next morning, I hear back. They’re able to access Rescue Mode, but the filesystem can’t fsck properly. Not feeling especially hopeful, I switch on Rescue Mode and log in.

And everything’s there. My Maildirs, my Subversion repositories, and all the sites I was hosting. Holy shit!

I promptly copied all that important stuff down to my personal computer, over the course of a few hours, and allowed Rescue Mode to end, and the machine to restart into its broken state. All right, I think, this is my cosmic punishment for not upgrading the machine from Ubuntu Hardy LTS, and not keeping the security packages up to date. Reinstall a new OS, with the latest version of Ubuntu they offer, and keep the bastard thing up to date.

Except that it doesn’t quite work that well. On trying to rebuild the new OS image… it goes into a error state again.

Well and truly hosed.

I spun up a new machine, in a new DC, and I’m in the process of reinstalling all the software packages and restoring the databases. Subversion’s staying out; this is definitely the straw that broke the camel’s back in terms of moving my personal projects to Git. Mail comes last, because setting up mail is such a pain in the ass.

And monitoring this time! And backups! Oh my God.

Let this be a lesson: if you aren’t monitoring it, you aren’t running a service. Keep backups (and, if you have the infrastructure, periodically try to refresh from them). And keep your servers up-to-date. Especially security updates!

And, might I add, many many thanks to the Rackspace customer support team. They really saved my bacon here.

Abandoning commits in Subversion

Before we begin, I don’t normally keep my code in Subversion… any more. When I started at Kijiji, Subversion was our local VCS of choice until there was a organisational decision to switch to Git, and we local developers agreed that it made sense. In the roughly two years since then, I’ve become a total convert. However, I’ve continued to use Subversion for personal projects, simply because I’m the only person working on them, and in case I move from machine to machine, I want to have reasonably quick access to the code.

I have since lived to regret this decision. Despite the fact that Subversion externals work vastly better than Git submodules, there are still some things I can easily do in Git that simply aren't available in Subversion.

One of my favourite things about Git is that, as long as you haven’t shared your repository, you can really mess with history. Step back three commits in the history, branch, write a new commit, then even cherry-pick everything else back on… then reset your HEAD hard, fast-forward, and when you push, it’s like those commits never happened!

You don’t really get that option in Subversion. Since it’s a centralised repository, everything you commit goes off immediately, so good luck getting rid of it for good! So if you realise that your last two commits were completely wrong, what do you do?

Fortunately, you can still branch from your history in Subversion:

$ svn copy http://svn.example.com/project/trunk/@56 http://svn.example.com/project/branches/whoops

Your tree now looks something like this:

--55--56--57--58--59 (trunk)
       \
        \-------------60 (whoops)

Now, everything that was on trunk up to, and including, revision 56, is in the whoops branch. Update your project’s root directory (or check out the branch into a separate directory), and get back to work! Once you’re done, the challenge becomes how to squash everything on trunk since 56. This is disturbingly simple, and only incurs two commits. It sounds dangerous as all hell, but trust me on this.

You’re going to delete trunk.

Only for a second! Since you want to destroy the last few changes on trunk anyway, and replace them with what you’ve done on whoops, this is entirely safe. After deleting trunk and committing, you just move branches/whoops to trunk, commit, and you’re off to the races! The whole process looks like this:

$ cd ~/project/
$ svn copy http://svn.example.com/project/trunk/@56 http://svn.example.com/project/branches/whoops
$ svn update
$ cd branches/whoops/
... work ...
$ cd ~/project/
$ svn rm trunk
$ svn commit
$ svn mv branches/whoops/ trunk/
$ svn commit

And, at the end of the day, you get a tree that looks like this:

--55--56--57--58--59    /-- Branched here
       \               /
        \-------------60--61--62 (trunk)
                          /    \
   Deleted trunk here -- /      \-- Renamed whoops to trunk

If you need to get any of 57, 58, or 59 back, they’ll be visible from the project root. Not remotely gone forever!

And so we see that, even in Subversion, you can achieve a measure of Git-like control of your history. The only difficulty is coordinating with your collaborators, along with the obvious evidence in the history of what you’ve done!