Your Encapsulation Is Bad, And You Should Feel Bad

Pass-by-reference is a fantastically powerful tool in object-oriented languages. In Java’s case, it ensures that no argument on a stack frame is longer than a processor word, by only passing along copies of primitives, and the heap locations of objects. It reduces your memory footprint fantastically, because it’s always there, unlike in C, where you had to specifically indicate that this argument is actually a pointer. Java does the same with return values as well—anything that you return from a method that isn’t a primitive is passed by its location on the heap.

And thus are a whole host of encapsulation and coupling issues born—particularly when you work with Collections.

Let’s say, for the sake of a specious example, I run a rental car agency. My agency is represented by an object Location:

public class Location { private Address address; private Place city; private Place province; private ArrayList<Vehicle> vehicles; ... public ArrayList<Vehicle> getVehicles() { return vehicles; } }

According to the Rules and Standards of JavaBeans, I’ve done encapsulation right… until I decide to process a series of updates to my stock in this boneheaded way:

public class Updater { ... public void doUpdates() { Location myLocation = LocationDao.getLocation(1); ArrayList<Vehicle> vehiclesToUpdate = myLocation.getVehicles(); scheduleWashes(vehiclesToUpdate); } public void scheduleWashes(List<Vehicle> vehiclesToUpdate) { for(Vehicle vehicle:vehiclesToUpdate) { if(vehicle.getLastWashed().add(DAY, 7).after(Calendar.getInstance()) { vehiclesToUpdate.remove(vehicle); } } updateScheduler.schedule(vehiclesToUpdate); } }

If, at any point after that for loop, I want to work with the list of vehicles, I’ll only have access to those vehicles that haven’t been washed in a week—I removed them from the same List that my Location object refers to.

Like I say, it’s a pretty specious example, but it shows what kind of unintended consequences can crop up when you pass mutable objects around by reference. Fortunately this doesn’t happen with Strings and Numbers (because they create new objects on the heap just about every time you assign a new value), but as soon as you start doing the same thing with more complex objects, you risk loss of data integrity. My Updater needs to know how my Location stores, and returns, the list of Vehicles in order to prevent problems, when it probably shouldn’t. The encapsulation here is bad, because it permits side-effects.

So what’s the fix? Replace the body of Location.getVehicles() with this: return new ArrayList(vehicles); and keep on going as-is. While each individual Vehicle that the lists are backed on will probably point to the same place on the heap (and this may, itself, have attendant problems, depending on what you’re doing, at least you know, for a fact, that whatever changes you make to the list you got back will be self-contained.

This gets even worse when you start throwing around DTOs for different serialisation methods. Because various annotations used by persistence architectures may not necessarily be compatible, often times you need to create three different DTOs for each of your database, XML, and JSON representations. These DTOs should only exist long enough to prepare your problem domain object for serialisation, or to deserialise something into your problem domain.

Say you created an amazing Web service that’s backed on SOAP (I know, I know, JSON geeks. It’s just an example). When I call your Java API’s method to getThing(), I shouldn’t be aware of the SOAP Body. I shouldn’t have to call, say, thing.getAttr(“thing”) to get something that’s an XML attribute, and then thing.getOtherThing().getValue() to get the String value of something that’s stored as an XML element. As a consumer of your API, I shouldn’t be aware of this; it means two things:

1. You can’t easily move your service away from SOAP, without either...
   1. Forcing all your customers to update their code to use a new API, or
   2. Internally converting your new serialisation into something that can be expressed as a SOAP call,
2. and you’ve told the world that your service is backed on SOAP.

Whether or not you’re proud of the fact that you’re using SOAP is irrelevant; it’s an implementation detail that I, as a consumer of your API, don’t care about. For all I know or care, you could be using a proprietary binary format, or even passing messages around by carrier pigeon. From my application’s perspective, this is all irrelevant. The encapsulation here is bad, because it exposes implementation details.

So, what’s the take-away from all this? Two things:

1. Don’t return references to your member collections and arrays. It’s bad for coupling. Return copies, instead.
2. When designing your API, give the consumers of it a paradigm that makes sense from the problem domain, instead of just blindly representing your storage format.

LinkedIn, you're fired.

Let’s be clear about something, LinkedIn. You’ve done good work. This has nothing to do with your performance as a professional social network. But your attitude about your job—the lackadaisical attitude toward data security, the fact that your communication with users who have question can take weeks, and your constant suggestions that your customers are mistaken about plain and simple facts—just can’t go on anymore. Your services won’t be required anymore. Someone will pack up your things.

Yup, I’m firing LinkedIn. I almost called this, “LinkedIn, I quit”, until I remembered that they are providing me with a service, so the firing metaphor is more apt. Particularly considering they’re really all about getting people jobs.

And it’s not as though LinkedIn has been totally useless to me. I’ve found a job through LinkedIn, coincidentally interviewing with a former classmate. My long-time tagline, “Minor PHP deity/aspiring system administrator/technorenaissance man” was referred to by my hiring manager at my current job as part of why he hired me. Whether it was because it demonstrated my sense of humour or my confidence, or even just a joke, I’ll probably never know. But the point is, LinkedIn has its uses.

The problem is in their customer service—more accurately, their almost-total lack of customer service. First of all, when you have a problem, you are functionally prevented from submitting a question to their help desk without first making a cursory look in their knowledge base. Seems sort of fair, until you remember that, sometimes, you know going into it that your issue isn’t covered by a question: there are problems that come up that quite simply need a human being to resolve. So there’s a barrier to getting the help you need, and when you consider it a little further, implies that they don’t have enough staff working their help desk. This point, in fact, gets demonstrated later on.

The second barrier to getting help, once you’ve actually made it to their “Contact Us” page, is that, once you’ve typed out your question, a modal dialog pops up, where it performs a keyword search in its knowledge base for you, presumably based on your summary of your issue. JavaScript sets the browser caret on the button labelled “I Found My Answer”, which redirects away from the page. When you click the Back button in your browser, your question is gone. This is yet another attempt to obstruct users from getting the help they need, and another vivid illustration of just how understaffed their help desk is. To describe it as “poor user experience” is a bit insufficient.

So, just in order to get to the point where you have successfully asked a question, LinkedIn has done everything they can to prevent you from doing so. During the most recent occasion when I had to submit a ticket, I actually muttered at the website, “I hate you so much, I wish I could hate you to death.” I know, I know, it’s not really my line, but Goddamn if it isn’t evocative of the particular type of impotent rage you feel when you’re already mad at something that you can’t really take out your anger on, and it thwarts you.

Now, all that being said… what are the problems I’ve had so far with LinkedIn? Leaving out their comically bad response to the infamous Gawker Media password crack—resetting everyone’s password—there have been three:

1. Recommending connections to contacts from my email
2. Continuing to send Groups emails to an email address I (thought I had) removed from my account
3. Automatically connecting me to someone hours after they requested a connection

These are, in my opinion, huge problems in terms of data security. What made them all even worse was the fact that, in every case, the help desk agent flatly contradicted that the problem existed. Because I don’t particularly feel like trying to sum them up, I’m just going to copy and paste the ticket contents, and let LinkedIn speak for themselves. I apologise in advance for my language, and I haven’t changed the agent’s declared names. I see no benefit in protected the guilty.

---

Support History » Why is LinkedIn accessing my email account?!

Your Question 04/26/2010 23:42
I just received my weekly LinkedIn network update, letting me know that a person I regularly email has just joined LinkedIn.. which seems somewhat strange to begin with, since this person has little reason, that I know of, to use LinkedIn.

What really set off a flag for me, though, is the fact that LinkedIn *knew that I know this person*. I have *never* instructed LinkedIn to import my contacts from my webmail services, and never intend to. So why is it that LinkedIn is suggesting that I add people from my Google Mail contact list to my LinkedIn network?

Before this event, I have been particularly satisfied with LinkedIn’s service; I’ve been able to keep in touch with colleagues, make new connections with recruiters, and in one instance I was able to secure a job because of it. It’s quite valuable to me, but if the service can’t do something simple like respect my privacy, I may have to leave the service and tell everyone I know that uses it precisely why I left.

The contact in question’s email address is [REDACTED]

Thank you. I hope to hear from someone promptly regarding this matter.

LinkedIn Response 05/01/2010 11:21
Dear Matthew,

Thank you for contacting LinkedIn Customer Support.

Please be assured that LinkedIn would never access your address book or import contacts without your permission. Our policies require all members to enter a password any time an address book is imported or other actions are taken on the account. My records show that you have imported some contacts into your LinkedIn account at some point in time. One of those contacts is the one in question. You can see the contacts you have imported if you log into your account and click on “imported Contacts” under “Contacts”. Please know that you can delete imported contacts at any time by selecting contacts and clicking on the "Delete selected contacts" button.

If you have further questions, please feel free to reply to this message.

Roberta
LinkedIn Customer Support

---

Support History » Remove secondary email

Your Question 01/23/2012 23:47
I attempted to remove [REDACTED] as an email associated with my LinkedIn account a couple of months, preferring [REDACTED] as my email address, due to a brief security compromise of my Google Mail account.

I’ve noticed over the last little while that I’m still receiving email from LinkedIn at the old address. Nowhere in my settings does it indicate that you’re storing the other address, until I came in here to contact you and I found it as a secondary email address.

Remove this address from my account immediately. If this cannot be done, then it must be exposed as my alternate email address within my account settings, so that I might change it to another alternate email address.

LinkedIn Response 01/24/2012 01:06
Hi Matthew,

Thanks for your email letting me know that you still receiving emails to your old email address and I understand how concerned you must be about this matter.

I’m able to locate the account and see that there’s only one associated email address [REDACTED] which have 166 connections. I’ve tried to locate the account with the email address [REDACTED] mentioned, but I couldn't find any account on our records.

[I continue to receive LinkedIn Groups emails at the old address to this day. –ed.]

However, if you want me add your old email address to our “Do Not Contact List” in order to stop receiving emails in future. Please reply to my message so that I’ll be glad to proceed with your request further.

Matthew, I await for your response.

Regards,

Susheel
LinkedIn Customer Service.

---

Support History » Unauthorized connection

Your Question 11/07/2012 20:44
At 7 November 2012, 12:34 AM EST, I received an email notification from LinkedIn, informing me that “[REDACTED]” would like to connect with me. I received another email at 1:51 AM EST, suggesting that I see what [REDACTED] has been up to, because we had now become connected.

I did not authorize this action. In point of fact, I did not know of either email until I woke up this morning and checked my email. I can only assume that either some device in my house performed the authorization on my behalf, or that this authorization was committed fraudulently. Accordingly, I insist that you divulge all the logs you have respecting this authorization, and with the original request made by [REDACTED] that the two of us connect, so that I can verify that the request did not come from any device in my possession.

Please note that this is not the first problem I have had with LinkedIn. As you can surely see from my support case history, I also have reason to believe that some process of yours gained access to my Google Mail account without my permission. Despite the support representative's insistence that I must have kicked it off myself, the fact is, I disagree with such “helpful” services on general principle and would never have done so. I have also had no end of difficulty purging my other email address from systems. I continue to receive LinkedIn Group notifications at my past email address, despite a support rep’s insistence that the old email address has been purged from your systems.

Your Response 11/13/2012 14:38
And now I’ve just discovered I had apparently followed PwC Consulting?! What the fuck, guys. This shit has to stop.

LinkedIn Response 11/20/2012 14:29
Hi Matthew,

There are a couple of scenarios that could explain the possibility of unauthorized access to a LinkedIn account:

1. If you’ve recently logged into your account from a public computer and didn’t completely sign out of your account, the next person to access the site on that computer may have unintentionally logged into your account.
2. If you share a computer with another person either at your workplace or home and didn’t completely sign out of your account, the next person to access the site on that computer may have unintentionally logged into your account.
3. Your account has been compromised by someone with malicious intent.

In order to secure your account, we have taken the following actions:

1. We signed you out of your account from every computer it has ever been accessed on. This will now prompt a new login for your account.
2. We sent a password reset link to the primary email address listed on your account.

We also like to recommend these best practices for your online privacy:

1. Always completely sign out of your LinkedIn account each time you leave a computer.
2. Don’t use the same password on multiple websites. If fraudsters identify your password, they can use it to access your other accounts.
3. Select passwords that can't easily be guessed. Create one that includes 10 or more characters. Hint: Think of a meaningful phrase or quote and turn it into a complex password using the first letter of each word in the sentence and add more complexity by adding capital letters, punctuation and symbols.
4. Never give your password to others or write it down.

If you continue to see anything suspicious, please report it to us immediately.

Regards,

Andrea
LinkedIn Trust & Safety

---

Two canned responses! It’s like they aren’t even trying. You can see that I didn’t bother responding to any of the tickets; there was demonstrably no point, since they weren’t putting forth any effort to resolve the issue in the first place. But then, maybe that’s what they’re trained to do.

Like I said above, I’m firing LinkedIn. Exactly what form that will take on LinkedIn’s servers, I have yet to decide. I want to leave a link to this article there, so that anyone who looks me up can read about my awful experience, but I also want to retain the maximum possible visibility—which would mean maintaining all my connections and job history. Decisions, decisions.

LinkedIn, consider this your two weeks’ notice.

Why Do Not Track doesn't really matter to me

It seems that every couple of weeks, a new article crosses Boing Boing or Slashdot about Do Not Track. Not too long ago, it came out that Microsoft was going to launch Internet Explorer 10 with DNT switched on by default, and all the advertisers were up in arms. Now, Yahoo! has announced that, as a response (read: a "fuck you") to Microsoft, they're going to ignore DNT if the visitor is using IE10, because they can't rely on it truly being a reflection of the user's preference. Fine. It's certainly their prerogative whether or not they intend to adhere to it; it's not as though DNT is a required thing. But in looking inito this very public pissing contest about Do Not Track, I discovered that Microsoft isn't actually doing anything that goes against the standard.

So, without further ado, a few things any Web service provider needs to know about Do Not Track:

1. Do Not Track is not mandatory for providers. DNT is not a requirement for servers. Though a standard is being drawn up for it, respecting that header is purely voluntary. Even indicating that you're respecting the header is voluntary; the W3C draft only defines the response header as something that a server MAY send. Besides, with the standard only in Working Draft status, implementing it at this time may mean that you may have to go back and re-implement Do Not Track... never mind the fact that Yahoo! is being pretty public about intending to ignore it. I really doubt that Web browsers will do much more than say "part of this page didn't return the DNT header", at worst, if a provider decides not to adhere to it, and even that seems unlikely.
2. Most providers don't have to care. DNT only pertains to tracking done by third-party providers to a Web visit. The canonical example of this is an advertiser, such as DoubleClick. DART ads are damn near everywhere on the Web, and I have to admit it's downright spooky when I look at the Roots Canada online catalogue one day, and for the next three days, every ad I see is suddenly Roots, where they'd never appeared before. Clearly, DoubleClick is watching you. But like I say, unless you're providing content that will be included in another organisation's Web pages, then Do Not Track does not apply to your service, and you can go on ignoring it. An exception is that if your service is forwarding tracking data to a third party on the server side, then you'd actually need to worry about what the DNT header contains, if you're bothering to adhere to it at all. However, most providers prefer to offload as much of that work to the user agent, for the sake of apparent site speed, so, like I say, most providers don't have to care.
3. Enabled-by-default isn't actually prohibited by the W3C Working Draft. When Microsoft announced that IE10 would switch on DNT by default, this was a valid option, according to the (in-progress) standard. Only in the most recent revision, dated 2 October, was the default specifically stated to be "assume no preference has been expressed." Until then, the standard only stated that intermediary services (such as proxies) may not change what preference is or is not indicated. Currently, the standard states, “A user agent MUST have a default tracking preference of unset (not enabled) unless a specific tracking preference is implied by the decision to use that agent.” Microsoft has publicly stated that their new browser will enabled DNT by default. Certainly there will be clear statements in all the marketing materials to this effect. It’s clearly a safe assumption that use of IE10 implies a specific tracking preference on the part of the user.
4. The "Acceptable Uses" definition makes a lot of DNT irrelevant for even third-party content providers. My biggest concern about Do Not Track was around maintaining security audit information. Good news! That's one of many acceptable uses of tracking data, that allow a provider to largely ignore the Do Not Track header. The only stipulation made, when claiming "acceptable use" is that you don't pass on that stored data (which you shouldn't anyway), and that you don't use it to personalise ads. That, right there, is the entire crux of Do Not Track: not personalising ads. Track all the information you want, just don't share it and don't expose that you're doing it.
5. Most, if not all, browsers provide a mechanism to pop up a dialog when a site wants to store a cookie. For the most part, browsers already have the technology to largely prevent effective multi-site tracking by advertising providers. While Do Not Track is a little more comprehensive, simply refusing to allow, say, 112.2o7.net to put a cookie in your browser goes a long way to preventing the, from following you around the web. Granted, it would force users to investigate the options on their browsers (and I am feeling a little cynical right now about the motivations of users), but I don't think it's really asking too much that a person learn about the tools they're using.

So, as someone writing a Web application that I'd love to have other people use, how much do I really need to care about Do Not Track? As it turns out, not much. Not adhering to it, in all likelihood, won't affect interoperability, but if it seems like it is, I'll just need to add one file, in one location, indicating that my service is a first-party service, and that's the end of that.

All in all, I don't really think Do Not Track has much in the way of teeth. The advertisers that it's mostly aimed at are such behemoths in terms of coverage that even without being able to personalise some fraction of the users' ads, that they'll still be making money hand over fist. And the advertisers will continue to be able to track you... they just won't be able to make it obvious.

On monitors and error detection

Earlier today, a colleague and I were discussing monitoring tools for Web services. He recently joined our team as a systems administrator, and I was filling him in on a homebrew monitoring service I put together a couple of years ago, to cover a gap in our existing monitor’s configuration, done in the spirit of Big Brother. He had praise for its elegance, and we joked a bit about reusing it outside the company, the fact that it would need to be completely rebuilt in that case (since, though it wasn’t composed of original ideas, just a merger of Big Brother and Cacti, it remains the intellectual property of $EMPLOYER$), and whether or not I would even need such a service for Prophecy.

After thinking about it briefly, I realized that not only will Project Seshat deserve some kind of monitoring once I install it on my server—I guess I’ll just add that to the pile of TODOs—but I remembered that I have a WordPress instance running for the Cu Nim Gliding Club, in Okotoks, Alberta. Surely a production install of WordPress deserves monitoring, in order to make sure that Cu Nim's visitors can access the site.

So, while waiting at a restaurant for my wife and our dinner guests to arrive, I took to The Internet to look for any existing solutions for monitoring WordPress with, say, Nagios. I may not be familiar with many monitors, but I know enough about Nagios to know that it works well with heartbeats—URIs that indicate the health of a particular aspect of a service.

The first hit I found that wasn’t a plugin for one of the two was a blog entry describing how to manually set up a few monitors for a local WordPress instance. It explained how to configu Nagios to run a few basic service checks: that the host in question can serve HTTP, that it can access the MySQL server, and that WordPress is configured, a single check on the homepage.

To me, this seems woefully incomplete. A single check to see that anything is returned by WordPress, even if you are separately checking on Apache and MySQL, strikes me as being little more than an “allswell” test. Certainly, success of this test can be reasonably inferred to indicate good health of the system, but failure of this test could mean any number of things, which would need to be investigated to determine what has gone wrong, and the priority of the fix.

When I use a monitoring system, I want it to be able to tell me exactly what went wrong, to the best of its ability. I want it to be able to tell me when things are behaving out of the ordinary. I want it to tell me that, even though the page loaded, it took longer than some threshold that I've set (which would probably warrant a different level of concern and urgency than the page not loading at all, which would be the case with a single request having a short timeout). In short, I want more than just the night watch to call out, “twelve o’clock and all’s well!”.

The options that I could take to accomplish this goal are myriad. First of all, yes, I want something in place to monitor the WordPress instance. But for original products, like Project Seshat, I would definitely like something not just more robust, but also more automatic. Project Alchemy is intended to create an audit trail for all edits without having to specifically issue calls to auditing methods in the controllers. I’d love to take a page from JavaMelody and create an aspect-oriented monitoring solution that can report request timing, method timing, errors per request, and perhaps even send out notifications the first time an error of a particular severity occurs, instead of the way Big Brother does it, where it polls regularly to gather data.

Don’t get me wrong, it’s probably a huge undertaking. I don’t expect to launch Project Seshat with such a system in place (as much as I’d love to). But it’s certainly food for thought for what to work on next. And when Seshat does launch, I will want to have a few basic checks to make sure that it hasn’t completely fallen over. After all, so far, I’ve been adhering to the principle of “make it work, then make it pretty.” May as well keep it up.

In which the general fear of TDD is discovered

Since I last wrote about test-driven development—since we spent that time at work learning how to do it, I’ve been trying to make use of it in my off-time development. I’ve mentioned before that I’ve been writing an ORM from scratch, to satisfy an itch that I have. In its current incarnation, I haven’t really had many opportunities to write anything using it, other than an aborted attempt to create a tracker for the No-Cry Sleep Solution.

Earlier this year, the note-taking web app that I’ve been using for years made a major overhaul of their user interface…and left mobile web out in the cold. Seriously. If you aren’t accessing the site from something that can fully act like a desktopfull-scale browser, then you’d better be on either an Android or iOS device, because otherwise, you’ve been left out in the cold.

At the time, I was well and truly in the cold. My mobile phone was, and still is, a Palm Centro. My only tablet-like device was my Kobo Touch (my wife owns a Nook Color, but I wasn't about to both commandeer it during the day and install a note taking app), though we’ve since also purchased an iPad with LTE. At work, at the time, I used my Kobo to present myself with my notes during scrums. Since then, I’ve been writing to a static HTML file on those days that I don't bring the iPad to the office, but there’s still a nontrivial issue of synchronisation. While I could probably use Dropbox and a reasonably simple PHP application to read and write to a single note file, that still just doesn’t do it for me.

So, I opted to begin writing my own, using Alchemy and Zend Framework on the back end. The initial progress wasn't so bad, and it isn’t as though I didn’t have alternatives that have worked reasonably well in the meantime. I decided to basically cater to my own use cases, since I could. Mobile Web would be reasonably fully featured, if a degraded experience. My Kobo Touch would get a good interface where I could edit notes, or write new ones, easily. It would all be there.

The problem is that it hasn’t always been smooth sailing. Ignoring the fact that I don’t often have the opportunity to work on it at home, having a toddler, it seems like with every model I implement, I find another thing about Alchemy that needs to be added or fixed. I’ve been trying to adhere to test-driven development to do that, but by God, I made it difficult to do that in some places. Doing the whole “TDD as if you meant it” thing can be particularly tricky when you’re working with an existing codebase that isn't particularly (or even remotely) tested, and particularly when you're writing web application controllers. Controllers are notoriously hard to unit test, if for no other reason that because their very purpose is side effects, which runs somewhat contrary to many of the premises of test-driven development. I’m finding that it’s far more straightforward to perform acceptance testing on your controllers, and actually go through the motions of the task you’re seeking to test.

Where I’ve been running into difficulty with Project Seshat,¹ though, is in code that I not only wrote a long time ago (somewhere on the order of three years), but also works perfectly well in isolation. The Model class, and its database-driven subclass, provide a parent class to all model-like activity in my application. It acts as entity, DAO, and service layer, mainly because that’s what made the most sense to me at the time I started writing it (this was well before I started working with enterprise Java. I still disagree with the notion of the DTO, but have yet to fully articulate why, to my own satisfaction). And that’s fine; it can still work reasonably well within that context. The problem is that, at some point when working with each of the last two Models I’ve added, the logic that stores the information in the database has both succeeded and failed in that regard at the same time.

Huh?

One of the core features of the ORM in Project Alchemy is that every change that’s written to the database with an expectation of long-term persistence (so, basically, everything that isn’t session data) also gets logged elsewhere in the database, so that complete change history is available. This way, if you ever need to figure out who did something stupid, it’s already there. As a developer, you don’t have to create and call that auditing layer, because it was always there, and done for you.

This audit trail, in its current form, is written to the database first—I decided to implement write-ahead logging for some reason that made perfect sense at the time. Not that it doesn’t make sense now, but there are a lot of features that still have to be implemented…like reading from this log and providing a straightforward function for reverting to any previous version. But at least the data will be there, if only, for now, for low-level analysis.

At any rate, because I can see these audits being written, I know that the ORM is at least trying to record the changes to the entities that I've specified; they’re available at the time that I call the write() method in the storage area for uncommitted data. The pain is that when it tries to create a new instance of the Model in the database, the model-specific fields aren’t being written to the entity table, only to the log. The yet-more painful part is that this doesn’t happen in testing, when I try to reproduce it in a controlled environment. This probably just means that these bug-hunting tests are insufficient; that they don’t fully reproduce the environment in which the failure is occurring.

So yeah. TDD, while it’s great for writing new code, is very difficult to integrate into existing code. I’ve had to do what felt like some strange things to shoehorn a test fixture in place around all this code I’ve already written. I recognise that the audit trail makes the testing aspect a little bit more difficult, since it's technically a side-effect. However, I don’t really want to refactor too much, of any, of the Model API, simply because my IDE isn’t nearly clever enough to be able to do it automagically, and because I still really, really want the audit trail to be something that doesn’t have to be specifically called.

I am, however, beginning to understand why so many developers who have never really tried TDD dismiss it, claiming that you end up writing your code twice. At first, you think that the test and the code are completely distinct entities, and that the structure of your tests will necessarily reflect your code. Yeah, this would mean that you’re doing everything twice. But that’s not TDD done properly. But then when you get into it, you realise that it isn’t the new code you have to write twice, but all the existing code that has to be massively refactored (and in some cases, virtually rewritten, so dissimilar is the result from the what you started with), and that’s always a daunting thought. You may even find yourself feeling compelled to throw out things you’ve spent a great deal of time and effort on, purely in order to get it testable.

I get that. That’s where I am right now. But there are two things to remember. First of all, your code is not you. If you want to work effectively in any kind of collaborative environment, whether at work or on an open-source project, you need to be able to write code and leave your ego at the door. Hell, the same thing goes for personal projects. The second is that if you refuse to make something better (whether better means more efficient, more maintainable, or whatever), simply because you invested x hours in it is foolish. You probably made something good, but you can always make it better if you’re willing to put in the effort.

And speaking of persistence issue, I’m sure I’ll fix it eventually. I already did once, though I didn’t properly record how I did it. Gee, if only I had some kind of mechanism for taking notes!

---

¹ Seshat was the Egyptian goddess of wisdom, knowledge, and writing. Seems appropriate to use a name that means "she who scrivens" for the tool you're going to use for your own scrivening.

In which a discipline seems not so impossible to define

At work, I’ve been trying for a while to do to things:

1. Discover what $EMPLOYER considers to be “Software Engineering” at the next pay grade above mine, and
2. Satisfy those requirements.

Every time I think about it, though, I’m revisited by my internal conflict over the title Software Engineer. I am not a Professional Engineer. I am not a trained engineer; I took exactly one course in “software engineering” in university, and it was a crash course in Scrum. So really, calling myself a Software Engineer, and trying to advocate for being called as such seems like attempting something between deceit and fraud.

This isn’t to say that engineering principles and practices can’t be applied to the discipline of software design and development. It absolutely can, and in most commercial cases, should. Part of the difficulty is that software development, and thus, the attempt to derive a working definition of Software Engineering, is still quite young. Not much younger than aerospace engineering, which is quite young, and quite well defined. Professional Engineering organisations are still at odds about whether or not they’re willing to license Software Engineers, because it’s still staggeringly hard to pin down exact what constitutes Software Engineering. So, the title of Software Engineer goes on uncontrolled and unchecked, as long as software engineers don’t call themselves Professional Engineers (unless they already are, from another engineering discipline).

I suspect that a large part of the problem is that software engineering is typically taught in university as a single course; maybe two. Every discipline of engineering recognised by P.Eng. organisations is taught as a four-year degree. Clearly, there’s a disconnect here.

Then, I got to thinking, could software engineering be taught as a four-year degree? What courses would be taught in this degree? What skills and disciplines have I learned, formally or informally, that have made me a better software developer? Certainly this curriculum would require a good foundation in computer science; many first- and second-year courses could be shared with CS. But then time would need to be spent on all the practices that help software to be written according to engineering principles and practices.

The current, most obvious, example is test-driven development. My group has been challenged by our manager to take what we learned in that TDD course to heart, and really try to apply it to our work. And I’ve been trying to do just that. I spent a day and a half refactoring code I’d written earlier that week in order to make it testable. In just one day, I’d done enough to establish 58% line coverage in unit tests. Trying to get the rest seemed like “not worth it” because the specific task should only be roughly a one-time-use process, and anyway, is already integration-tested and spot-check verified to be working correctly. But the refactoring-for-better-testing exercise was a useful reminder of what TDD provides.

Since then, there have been a couple of bugs reported by our QA team that I’ve had to hunt down. In the spirit of TDD as if you meant it, I stopped myself from trying to diagnose the issue and model the solution in my head, and just wrote a simple test that would fail in the presence of the bug. Then I went hunting for the root cause.

Before our morning scrum, it was staring me in the face, and I had a decision to make, of how to implement the fix. With a decision in place, more tests, in different places, because of how I wanted to implement that fix. Then back into the code to make the tests pass.

It ended up being a bit more complex than I’d originally expected (and I’d judged it as being reasonably complex), and took a day to really, properly, nail everything down to satisfy the requirements that weren’t being fulfilled. But the solution is one with thorough unit tests, that handle the full range of possible inputs (as well as some CANTHAPPENs), so I can state with confidence that the solution is robust, and thorough, and that it won’t fail in adverse situations.

These are engineering principles. I think it’s reasonable to posit that test-driven development could be taught as a part of a university degree in Software Engineering. But could you really spend an entire course teaching Test Driven Development (or whatever name it would have on a university timetable)? You could.

The course that we had was two weeks. Three days “in class”, and our team split into two groups that worked, with the instructor, for three days each of, essentially, labs. These were still very teachable, because they were full of real-life examples—the very code that we work with every day. At any rate, each developer got six working days, about seven hours each, spent learning Test Driven Development. Six working days of seven hours is, if you do the math, forty-two hours. The typical university course is three hours per week, for thirteen weeks, for a total of thirty-nine hours.

Suddenly, the possibilities open up for what could comprise a four-year degree in Software Engineering. Consider that any intensive, multi-day, all-day course you take as a professional can map to university credit courses if its length can be measured in whole weeks, one course per week. Suddenly, the notion of establishing a licensing program for Software Engineers that is on par with every other discipline of Professional Engineering looks feasible.

So if you were planning a curriculum, what would you put on it?

Better refactoring... through testing

When you sit down to design a new class for your application, how many things do you normally intend it to do? When you’re finished implementing it… how many things does it actually do?

Are these different numbers?

I thought they might be. It’s far too easy—and I’m as guilty of this sin as the next programmer—to fold subtasks into a single class, because they’re all part of the same subsystem, right?

Wrong. Each class should focus on doing one thing, well. Subtasks need to be foisted off into a separate class that focusses on doing that subtask. There are two reasons for this:

1. You reduce coupling of the class. If you have a class A, that does subtasks a1, a2, and a3, and the subtasks don’t really affect one another, then class A doesn’t need to know everything about a1, a2, and a3.
2. You improve the testability of the class. Presumably the subtasks are all hidden behind private methods, because they shouldn’t be part of the parent class’s API. That much, at least, is correct. But if the subtasks are tightly coupled with the parent class, then you can’t readily test the subtasks without executing the entire parent task… and that may not be something you can express as a unit test, particularly if the parent task is dependent on an external Web service.
3. In addition to improving testability, you also make your tests meaningful. A failing test on a complex method can mean a lot of different things, depending on the nature of the failure—how was the result different from the expectation? If you have simple tests, that focus on short methods (you’re keeping your methods short, right), then when your tests fail (and you may get cascading failures), you’ll be able to identify fairly readily why they failed.

For the sake of a really facile, fairly stupid example, here’s a class that’s doing two entirely separate subtasks:

package com.prophecynewmedia.example.subtask;public class GodObject {  public String performsAComplexTask(String parameter1, String parameter2,                                     String parameter3, String parameter4) {    return combineTransformations(parameter1, parameter2, parameter3, parameter4);  }   private String combineTransformations(String parameter1, String parameter2,                                        String parameter3, String parameter4) {    return transformParametersAnotherWay(parameter1, parameter2)         + transformParametersOneWay(parameter3, parameter4);  }   private String transformParametersOneWay(String parameter1, String parameter2) {    return parameter1 + parameter2;  }   private String transformParametersAnotherWay(String parameter1, String parameter2) {    if (parameter1.contains(parameter2)) {      return "((" + parameter2 + ")" + parameter1 + ")";    }    return "(())";  }}

What do transformParametersOneWay() and transformParametersAnotherWay() have to do with each other, practically? They get called on different parameters, do different things, and are only combined in combineTransformations(). Here's the test class for GodObject:

package com.prophecynewmedia.example.subtask;import org.junit.Test;import static org.junit.Assert.assertEquals;public class GodObjectTest {  private GodObject godObject = new GodObject();  @Test public void confirmsLastTwoParametersCombineCorrectlyWithDifferentFirstParameters() {    assertEquals("(())cd", godObject.performsAComplexTask("a", "b", "c", "d"));  }   @Test public void confirmsFirstTwoParametersDoNotCombineCorrectly() {    assertEquals("(())", godObject.performsAComplexTask("a", "ab", "", ""));  }   @Test public void confirmsFirstTwoParametersCombineCorrectly() {    assertEquals("((a)ab)", godObject.performsAComplexTask("ab", "a", "", ""));  }}

Now, because of the simpleness of what we’re doing, this test doesn’t look so bad. Given. However, the first test also has to take into account what happens to the first two parameters in its expectations. This just isn’t clean. Surely there’s a way we can clean this up!

GodObject is doing a task that requires two completely distinct subtasks, and it&rsuqo;s doing those tasks too. In order to test one of those tasks, we have to take into account the results of the other. Let’s clean this up, one step a time…

Step 1: Make the first subclass.

We’ll modify GodObject.combineTransformations():

  private String combineTransformations(String parameter1, String parameter2, String parameter3, String parameter4) {    return transformParametersAnotherWay(parameter1, parameter2)         + SimpleTransformer.transform(parameter3, parameter4);  }   static public class SimpleTransformer {    static public String transform(String parameter1, String parameter2) {      return parameter1 + parameter2;    }  }

We’ll add a new test for this class:

package com.prophecynewmedia.example.subtask;import org.junit.Test;import static org.junit.Assert.assertEquals;public class SimpleTransformerTest { @Test public void combinesCorrectly() {  assertEquals("ab", GodObject.SimpleTransformer.transform("a", "b")); }}

So far, so good. We'll leave the first ugly test in the GodObjectTest, because it's the only test that confirms that combineTransformations() actually works. We'll rename it later. For now, let’s move that other transformation out into another new class.

Step 2: Make the second subclass

  ...  private String combineTransformations(String parameter1, String parameter2, String parameter3, String parameter4) {    return ComplexTransformer.transform(parameter1, parameter2)         + SimpleTransformer.transform(parameter3, parameter4);  }  ...   static public class ComplexTransformer {    public static String transform(String parameter1, String parameter2) {      if (parameter1.contains(parameter2)) {        return "((" + parameter2 + ")" + parameter1 + ")";      }      return "(())";    }  }

And the tests...

package com.prophecynewmedia.example.subtask;import org.junit.Test;import static org.junit.Assert.assertEquals;public class ComplexTransformerTest {  @Test public void confirmsParametersDoNotCombineCorrectly() {    assertEquals("(())", GodObject.ComplexTransformer.transform("a", "ab"));  }   @Test public void confirmsParametersCombineCorrectly() {    assertEquals("((a)ab)", GodObject.ComplexTransformer.transform("ab", "a"));  }}

This has made the last two tests in GodObjectTest redundant, because now we know that the transformations are being performed correctly (and will learn of failure if they change because the relevant tests will fail. I won’t bother showing the truncated test class here. It’s step 3, at any rate. Step 4 is moving the static inner classes into their own classes within the package.

There. We’ve now turned a facile, ugly, hard-to-test example into a facile, slightly less-ugly, easy-to-test, easy-to-refactor, easy-to-read-the-tests example. Huzzah! Being able to do this—to see where your classes are becoming too complex, and be able to refactor them into multiple simpler classes—is an important part of test-driven development, when you’re working with older code. If you inherit a project that has low test coverage, you may find yourself spending days doing this kind of stuff.

I’m tempted to set up Git and Subversion precommit hooks to prevent me from checking in code that isn’t tested. Granted, it would require parsing a coverage report and a diff, but it would probably pay off in the long run, because I’d know that everything I’m writing is tested, and would force me to leave the code better than I found it. It’s a story for another time, but we’re probably all familiar with the fear of refactoring another developer’s code in case you break it, and the inclination to just cobble on the changes you need to make. I know I am.

On testing and learning

For the past couple of weeks at $EMPLOYER$, we've had a trainer in who specialises in teaching test-driven development to programming teams who have shown any interest (or have been told that they really ought to develop an interest) in trying this particular method of writing software. And during this time, I've had three things pop into my head:

First, IntelliJ, when used by someone who is familiar with the Refactor menu, is like magic. Yes, I know that there was a lot of work that went into making sure that the IDE knows about (or can readily track down) all the references to a particular method, the methods on an interface, the number of usages of a particular method, and all that. A lot of work went into it. But, to quote Sir Arthur C Clarke, Any sufficiently advanced technology is indistinguishable from magic. Thus, I insist that IntelliJ is magic.

Secondly, test driven development, done right, is a completely different paradigm to programming than any of us in the office are used to, familiar with, or, dare I say it, comfortable with. We learned to write software by breaking the requirements down into discrete, modelable chunks, establishing a rough idea of how these chunks interact with each other, and having at it. Once fairly complete pieces of code are done, tests are usually written, and primarily as an afterthought--"Did I do that right?" TDD turns this completely on its head. You read through the requirements, and pick one that you can test. Then you write a unit test that will verify that the chosen requirement is satisfied. This unit test should not pass. Now, you write the bare minimum amount of code necessary to satisfy that requirement. Absolute bare minimum; make it as simple as possible. You don't even make new classes, just put new methods into your test class. Repeat until the methods in your test class start to develop some structure. This is the point where you start refactoring the code into standalone classes. Don't worry, it's supposed to be hard to wrap your head around.

If you've never developed according to TDD before, it will be unlike anything you've ever done. But the strange thing is, if you strip out the source code that supports a fully developed test suite, anyone else will be able reimplement your code, almost exactly as you did, in a fraction of the time. TDD does not, contrary to popular belief, double your development time, because you still only write the code once. It does, however, give you full test coverage, which means that when you go bug hunting, you can be sure that you won't introduce new bugs!

Finally, I was reminded of something that's very important to keep in mind as a professional programmer: you don't know everything. Not only are you not perfect, but there's always something to learn. There might be a better way to accomplish the task you're trying to achieve, and it's vitally important to be open to this possibility, or it will be far harder to grow as a professional than it deserves to be.

I heard a line in a software engineering class I took while I was taking my degree: "in a room full of programmers, if any two agree on something, then they have a majority opinion." As with a lot of male-dominated technical fields, this is tragically true. Men can be very stubborn, particularly when, in essence, we're being paid to Get It Right, Damn It! The job of a programmer is to write bug-free code, so there's a bit of a tendency to develop an inappropriate confidence in your ability. Taken to the extreme, we become arrogant, to an extent that some programmers, after coming to an opinion about a topic, will be absolutely immovable in their opinion until (and in some cases, despite) they have been proven wrong. You can't just tell these guys, "Hey, I find that with this other technique, there's thus-and-such a benefit," you actually have to demonstrate an improvement over their technique... and God help you if your difference of opinion is in coding style. I once had an argument that dragged out, off and on, over days, about the potential side-effect reduction benefit of preferring prefix to postfix increment and decrement operators, when compared against how many decades of programmers are familiar with postfix courtesy of K&R.

Here's an important thing to remember: Even Knuth got it wrong on occasion, and offered a bug bounty. Kernighan and Ritchie admitted that the code samples in their book weren't perfect. Hell, if you think the Turing machine sprung from Turing's head, fully formed, I guarantee you're fooling yourself.

There's always something more to learn. If someone is going around claiming that their method of doing a thing is a good way of doing things, listen. They might just be right, and if you can teach them (and not browbeat them, preferably) if they're wrong, then everyone wins. And if learning a new programming technique, which is completely alien to everything you've done before, is the best way to be reminded of this fact, then so be it. I'm just looking forward to figuring out ways of having my programming tools force me to actually follow this technique, because I think maximising test coverage is a bare minimum for going from merely writing software to actually engineering it.

The sleep-deprived ramblings of a developer in transition...

Late-night software deployments are no fun. No fun whatsoever, particularly when you feel compelled (out of a little niggling paranoia that you’ve done something just a hair’s-breadth away from perfect) to stay up to watch the automated jobs pick up and start running the new code.

I’m going to be very happy when this project can be put to bed, so that I can get back to doing things that I’m a little more emotionally invested in. I have a great opportunity to do some systems architecture coming up; we’re rewriting my core projects into Java and JSF, which is allowing for some much-needed central refactoring, so that things can be decoupled and streamlined. I might not have the technical knowledge that the other developers on the project have (I’m terrifically out-of-date on Java), but I do have solid domain knowledge, so I’ll be able to establish the game plan—write up most of the user stories and make sure that that’s all coherent, make sure that all the systems that need to be redesign get done correctly, &c, &c.

Of course, the worst thing of staying up late to work on code is when you forget that there’s a time zone difference between you and the server it’s running on… and that when it runs at “2:00 am”, it’s actually referring to Mountain Standard Time. Bah!

Time for bed.

In which documentation leads to rearchitecture

All year—well, mainly just the second half of 2011—I’ve been intending to write some XML Schema documents to formalise the input and output of an XML API I wrote a year and change ago. Is it really Important to do this? Yes and no. Mostly no (which explains why it hasn’t been done yet), since there are only three applications that use the API so far and I wrote all of them (and the PHP and JS libraries that handle it). From the functional perspective, it’s not critical that these schemata get written. However, I’m also trying to finish off some formal documentation of the tool that the API serves, which includes these schemata. Why would it need to include the schemata? Hopefully, we’re going to get some more developers working on this project, so having a formal document that describes the input and output would be good; it would give these developers something to refer to, that could be used for validation, and improve the testability of the whole system.

I’ve been trying finally finish these schemata, and I’ve found two things about how I implemented it:

• The output has a small logic flaw, I think—I use a <message/> element both inside and outside the element that indicates what objects have been affected, depending (primarily) on the context of the action being performed.
• In order to be able to validate against these schemata, I’d have to make some obnoxiously redundant changes to the libraries that generate the input. The XML is somewhat polymorphic—depending on the value of the command attribute on an <action/> element, the legal child elements change. I’d love to be able to handle that without having to use the xsi:type attribute to indicate that the delete action is an instance of type ActionDelete, but it’s becoming apparent that XML parsers just don’t work that way.

So, the decision to make the XML polymorphic based on an <action/> element may have been shortsighted. Probably was, in fact. The question is, though, do I rewrite the API entirely to allow this business logic to be expressed in the schema, or do I write a more generalised pair of schemata now, then clean up the API in a v2.0 so that it can be more rigourously and specifically validated against the business? The output definitely needs to be revisited (especially since it's currently sending both XML and JSON, depending on the action), but what to do about the input? Probably simpler to do the minimal amount of redesign now, then when the applications get refactored later this year, look into the more comprehensive enhancements.

We shall see, we shall see…