A lesson learned the very hard way

Two nights ago, I took a quick look at a website I run with a few friends. It’s a sort of book recommendation site, where you describe some problem you’re facing in your life, and we recommend a book to help you through it. It’s fun to try to find just the right book for someone else, and it really makes you consider what you keep on your shelves.

But alas, it wasn’t responding well—the images were all fouled up, and when I tried to open up a particular article, the content was replaced by the text "GEISHA format" over and over again. So now I’m worried. Back to the homepage, and the entire thing—markup and everything—has been replaced by this text.

First things first: has anyone else ever heard of this attack? I can’t find a thing about it on Google, other than five or six other sites that were hit by it when Googlebot indexed them, and one of them at least a year ago.

So anyway, I tried to SSH in, with no response. Pop onto my service provider to access the console (much as I wish I had the machine colocated, or even physically present in my home, I just can’t afford the hardware and the bandwidth fees), and that isn’t looking good, either.

All right, restart the server.

Now HTTP has gone completely nonresponsive. And when I access the console, it’s booted into initramfs instead of a normal Linux login. This thing is hosed. So I click the “Rescue Mode” button on my control panel, but it just falls into an error state. I can’t even rescue the thing. At this point, I’m assuming I’ve been shellshocked.

Very well. Open a ticket with support, describing my symptoms, asking if there’s any hope of getting my data back. I’m assuming, at this point, the filesystem’s been shredded. But late the next morning, I hear back. They’re able to access Rescue Mode, but the filesystem can’t fsck properly. Not feeling especially hopeful, I switch on Rescue Mode and log in.

And everything’s there. My Maildirs, my Subversion repositories, and all the sites I was hosting. Holy shit!

I promptly copied all that important stuff down to my personal computer, over the course of a few hours, and allowed Rescue Mode to end, and the machine to restart into its broken state. All right, I think, this is my cosmic punishment for not upgrading the machine from Ubuntu Hardy LTS, and not keeping the security packages up to date. Reinstall a new OS, with the latest version of Ubuntu they offer, and keep the bastard thing up to date.

Except that it doesn’t quite work that well. On trying to rebuild the new OS image… it goes into a error state again.

Well and truly hosed.

I spun up a new machine, in a new DC, and I’m in the process of reinstalling all the software packages and restoring the databases. Subversion’s staying out; this is definitely the straw that broke the camel’s back in terms of moving my personal projects to Git. Mail comes last, because setting up mail is such a pain in the ass.

And monitoring this time! And backups! Oh my God.

Let this be a lesson: if you aren’t monitoring it, you aren’t running a service. Keep backups (and, if you have the infrastructure, periodically try to refresh from them). And keep your servers up-to-date. Especially security updates!

And, might I add, many many thanks to the Rackspace customer support team. They really saved my bacon here.

Abandoning commits in Subversion

Before we begin, I don’t normally keep my code in Subversion… any more. When I started at Kijiji, Subversion was our local VCS of choice until there was a organisational decision to switch to Git, and we local developers agreed that it made sense. In the roughly two years since then, I’ve become a total convert. However, I’ve continued to use Subversion for personal projects, simply because I’m the only person working on them, and in case I move from machine to machine, I want to have reasonably quick access to the code.

I have since lived to regret this decision. Despite the fact that Subversion externals work vastly better than Git submodules, there are still some things I can easily do in Git that simply aren't available in Subversion.

One of my favourite things about Git is that, as long as you haven’t shared your repository, you can really mess with history. Step back three commits in the history, branch, write a new commit, then even cherry-pick everything else back on… then reset your HEAD hard, fast-forward, and when you push, it’s like those commits never happened!

You don’t really get that option in Subversion. Since it’s a centralised repository, everything you commit goes off immediately, so good luck getting rid of it for good! So if you realise that your last two commits were completely wrong, what do you do?

Fortunately, you can still branch from your history in Subversion:

$ svn copy http://svn.example.com/project/trunk/@56 http://svn.example.com/project/branches/whoops

Your tree now looks something like this:

--55--56--57--58--59 (trunk)
       \
        \-------------60 (whoops)

Now, everything that was on trunk up to, and including, revision 56, is in the whoops branch. Update your project’s root directory (or check out the branch into a separate directory), and get back to work! Once you’re done, the challenge becomes how to squash everything on trunk since 56. This is disturbingly simple, and only incurs two commits. It sounds dangerous as all hell, but trust me on this.

You’re going to delete trunk.

Only for a second! Since you want to destroy the last few changes on trunk anyway, and replace them with what you’ve done on whoops, this is entirely safe. After deleting trunk and committing, you just move branches/whoops to trunk, commit, and you’re off to the races! The whole process looks like this:

$ cd ~/project/
$ svn copy http://svn.example.com/project/trunk/@56 http://svn.example.com/project/branches/whoops
$ svn update
$ cd branches/whoops/
... work ...
$ cd ~/project/
$ svn rm trunk
$ svn commit
$ svn mv branches/whoops/ trunk/
$ svn commit

And, at the end of the day, you get a tree that looks like this:

--55--56--57--58--59    /-- Branched here
       \               /
        \-------------60--61--62 (trunk)
                          /    \
   Deleted trunk here -- /      \-- Renamed whoops to trunk

If you need to get any of 57, 58, or 59 back, they’ll be visible from the project root. Not remotely gone forever!

And so we see that, even in Subversion, you can achieve a measure of Git-like control of your history. The only difficulty is coordinating with your collaborators, along with the obvious evidence in the history of what you’ve done!

This is not a meritocracy

UPDATE

An hour ago, SendGrid publicly announced Adria Richards’ termination. They say,

While we generally are sensitive and confidential with respect to employee matters, the situation has taken on a public nature. We have taken action that we believe is in the overall best interests of SendGrid, its employees, and our customers.

In other words, they heard the thousands, if not millions, of people calling for Richards’ termination, and delivered. In an effort to do… what? Save their customer base? This is a hell of a message to send—if you embarrass a man for making tasteless jokes at a technical conference, and he gets fired and complains about it, we’ll throw you to the wolves.

The joker’s behaviour at the conference earned him disciplinary action (whether or not he should have been fired or given sensitivity training is academic)—he was acting as a representative of his company, at an event they had sponsored. Necessarily, he should have been on his best behaviour. Richards was probably representing SendGrid as well, insofar as SendGrid probably paid her to be there, and it was probably all over her nametags—and she may have been wearing company gear too. However, what was Richards’ offense? Saying “that’s not cool” loudly, really.

Rather than go to bat for Richards, and say, “we believe that the software industry is best served by a culture of universal respect, and we don’t condone anyone making inappropriate sexual commentary in the workplace or at a technical conference,” SendGrid has sent the message that they don’t have their employees’ backs. That they either don’t believe that the industry is rife with misogyny, or perhaps that they don’t think it’s a bad thing, or maybe just that it can’t be fixed.

I don’t hold with any of this. I believe that the misogyny that pervades this industry must be confronted head-on. New hire sensitivity training that says little more than, “don’t make dirty jokes around girls” is staggeringly insufficient, and if Human Resources requires this training, then everything that company does in public must reflect the beliefs that that training espouses.

SendGrid has told the world that they believe offensive jokes are okay in the workplace, and that if you call it out, you will be silenced.

Is my calendar right? Is it 2013, or 1963?

---

This week is a bad week for how I feel about my gender.

We got an early start on Sunday with shockingly insufficient sentences for a pair of teenage rapists, followed up by horrifying apologia from, well, all the major news outlets, CNN included. I’m not going to comment on it here, but I will suggest that you read I Am Not Your Wife, Sister or Daughter, a fantastic article (that my wife Anne wrote) that’s getting a fantastic amount of coverage. She’s absolutely spot-on when she points out that we, as a society, really need to stop trying to humanise rape victims to rape apologists by suggesting, what if it was your wife? Your sister? Your daughter?. It’s not just objectifying, but it also reinforces your audience’s misogynist worldview.

I could really get into it, because it makes me mad… but the way that the professional software community is treating Adria Richards—and, by extension, every woman in the industry, has got me so upset I can hardly see straight.

You probably know where I’m going with this, but let’s review the facts, shall we?

Richards publicly shamed two attendees for cracking sexual jokes about, among other things, “forking his repo” after a suggestion that forking is the highest form of flattery. I understand the pair of them were going on for quite some time, and the PyCon organiser dealt with the situation privately, and the guys were chastised for their behaviour, and that seemed to be the end of it.

Until when they got back to work, when at least one of the pair of jokers was fired. He then posted a strange apology that suggests that he believes Richards was trying to make that happen. Richards sent her own public apology to him and urged his employer to reconsider their decision.

Regardless of this, the male developer community has worked itself into a mouth-foaming rage. People are specifically calling for her dismissal, and there was at least one suggestion that the guy who was fired should sue her. There’s a whole host of men insisting that “dick jokes aren’t harassment”, as though sexual harassment can only occur through individually-directed comments. I’ve lost count of how many people are suggesting that Richards’ fragile female sensibilities caused her to overreact to a “private joke” (one, I’ll point out, was told in a crowded conference hall, and thus is anything but private, unless it was whispered directly into the other person’s ear).

Virtually every comment I read on the thread following the non-apology is coming to his support, and attacks Richards.

Virtually every commenter seems to believe that a man’s desire to make offensive jokes in a public space, while representing his employer, somehow trumps every other person’s basic right to be in a room without being made to feel uncomfortable because of their race, gender, religion, sexual preference, or even no reason at all. That bad jokes are somehow sacrosanct, and that people who are offended by them should simply “grow up and get over it.”

Look, this isn’t the way adults, and professionals, are supposed to talk to each other. This isn’t the way the developer community constantly tries to describe itself to outsiders. We insist, adamantly, that everyone is considered equal, and that the developer community is a meritocracy above all else.

This is, unfortunately, not the reality. Women have never been afforded the respect they deserve within this industry. RADM Grace Hopper invented the compiler, and assembly language, in order to make programming that little bit easier than having to remember and decipher opcodes, and her male peers couldn’t possibly have taken her less seriously… but because of her, I don’t have to have any idea what the x86 instruction set looks like in order to do my work.

And yet marketers at computing events like CES and E3 continue to hire booth babes—in other words, human furniture to make their booth look good. I’ve read of women who have produced games, who later staffed the conference booth for the game, and tech reporters asked her to get the producer, or technical director, as though the idea of a woman being responsible for creating something as complex as a video game was a foreign concept.

I’ve worked in a variety of companies, some larger and some smaller. But the reality is that I’ve worked with far more men in technical roles than I have with women, and that, invariably, when there haven’t been women in the technical group, it’s turned into a boys’ club.

This is unacceptable. The current software developer community is openly hostile to women asking to be treated like human beings, and this shit has to stop. You wouldn’t make racist jokes at a conference (or would you?), so what makes it magically okay to make jokes that objectify sex, and women?

Right, nothing does, because it’s not okay.

It’s not okay to compare an object to a person’s body. It’s not okay to compare a development process to sex. It’s not remotely okay to tell someone who says, “I’m offended”, that nothing offensive happened, and that they’re overreacting.

And it isn’t fucking okay to make death threats against a person who called out inappropriate jokes. Yes, this happened. Yes, the post has been deleted. Yes, I hope YCombinator does the right thing and assists the police in any investigation that might occur, and yes, I hope that investigation happens.

Finally, it’s not even a little bit okay to attack someone for acting on having been offended. That someone got fired for making inappropriate comments while representing his company at a conference shouldn’t be remotely surprising.

Technologists really need to start showing each other a lot more respect, because right now, it really feels like we don’t show each other any.

In which work ethics are considered

Over the past couple of weeks, I’ve had two very clear indicators—to myself—of just how much I’m enjoying my current job. I say that because there’s a subtle, but important, difference between saying “I love my job because I love doing x” and observing your own behaviours and thought patterns, and noticing that the way that you do your job, and approach your job, demonstrates how much you enjoy it.

The first way is something I realised about two weeks ago, when I met with the general manager (my boss’s boss, and an all-around good guy) for a quick catch-up chat. I’ve been working at this job for eleven weeks, and it feels like I’ve been there a lot longer, and more importantly, the environment, and the nature of the work is just naturally enjoyable, and it’s something I’m keenly interested in, so I like going to work on general principle. But when I was talking to him, it dawned on me that I like my work so much that, for the first time in years (at least two, if not four) have I been able to get so wrapped up in my work that I lose track of the time. Almost every other job I’ve had, with possibly one exception, since I moved to Toronto, I’ve tended to kind of check out around 4:30 or 4:45; I’d start trying to find something to do that would be productive work, but wouldn’t take too long to do, because I wanted to get out the door. Where I am now, as often as not, it’ll be almost 5:30 when I look at the clock and realise that I should probably go home.

The second, even clearer indication came this past Friday. I’ve been working on coming up with a way to better integrate two of the projects I’ve been working on, and particularly a way to do it across a subdomain divide (the products need to communicate on both the server and client sides; I managed to hack it on the pure client side by spawning some IFrames, but getting a particular server-side action in Project A to trigger an action in Project B has been a little less clear-cut. So, I decided that a simple XML interface into Project B was needed, with a wrapper for Project A (and Project C, which another developer is working on was necessary). So, I spent Thursday and Friday afternoons working on this API.

Two very cool things came from this.

1. Apart from a few minor syntax errors (missing a parenthesis, putting a colon where a semicolon should’ve been, &c.), the stripped down API worked perfectly right off the bat. A few hundred lines of code, and it Just Worked. I haven’t been that successful in a while.
2. I ran the first test at about 4:40. I had other things that needed to be done that evening which necessitated my leaving as close to 5:00 as possible, but I had a thought I haven’t had in a long time: I wish I didn’t have to leave right away. I wanted to take my work home with me.

This hasn’t happened in ages, and I didn’t realise how much I missed that feeling until this past Friday. I’ve been telling people how much I like my job based on its perks: catered lunches on Fridays, stocked kitchen, and an amazing sense of community with my co-workers. But being able to say, “there are days that I don’t want to stop working”… that might be the surest sign that you’ve got a great job.

It’s kind of funny, because I normally try to fight against that really, typically Protestant work ethic of, when you boil it down, “living to work”. I can leave my work at the office; most days, it’s a case of losing track of the time because I’m so wrapped up in what I’m doing, so when I realise what time it is, I clean up what I was doing, get it to a state where I can leave, and I go home. And I think that’s what this is an extension of—I got so wrapped up in what I was doing that, had I not had other things to do, I almost certainly would have stuck around, ignoring the clock.

I think that might be the difference. Most days, I don’t care what time it is; it’s irrelevant to me how many hours I spend at the office, as long as I get done what I want to get done. When I compare that against the Toronto workaholics who not only work to live, but take it as a point of some perverse kind of pride that they work sixty- or eighty-hour weeks, I can see much better what the difference is. I’m doing what I love, and I take pride in the result of my work, whereas some people take pride in the amount of work that they do.

That thing you use? I made that.

Monday night (technically Tuesday morning, but who’s counting? Other than Blogger, that is) I mentioned that being able to say to somebody, “you know that thing that you use? I made that” is a great feeling. I just ran into a former colleague from my previous contract, who let me know just on what scale my stuff is operating.

One of the projects I worked on—made, really; the requirements were small enough—was a carbon and cost savings calculator for Sears Canada’s website, so that people looking to replace one of their appliances could see about how much money they’d save by switching. Fairly simple to do; the worst of it was extracting the formulas from the Big Ugly Interactive Spreadsheet that Sears provided. I worked hard to provide the best little jQuery applet I could, complete with pretty transitions and everything. Fully translated into French, too, and I made sure it’d work acceptably well in IE6. It was kind of a focus of their recent/current Green promotion, but how many people, really, were going to wind up using it?

As it turns out, a lot. In the linked article, you can see that last Friday, they opened a six-kiosk booth in the Vancouver Robson store that runs that “little jQuery applet” in a touchscreen interface! I’m… speechless. To the best of my knowledge, nothing I’ve ever made has seen such a wide userbase. They’re adding more booths to more stores, too. I kind of hope that one will show up in Toronto so I can play with it and show people.