Steve Yegge's Rant on Languages

I stumbled on the following blogpost originally made in 2004 (updated in 2006) by Steve Yegge:
http://steve.yegge.googlepages.com/tour-de-babel

I vaguely recall the name and from the page that linked to his post, he apparently worked at Amazon for a time before moving on to Google. That said, I highly suggest taking a look at the original blog post as it's a pretty fascinating (and opinionated) read. I'm amazed by how much I agree with, though I didn't like his C++ bashing. But then, his comments about only liking C++ when he was in college are true of me as well. I suspect if I actually had to do something real in C++ (after having used Java) I'd want to blow my brains out.

In any event, a few highlights from my perspective:

C

You just have to know C. Why? Because for all practical purposes, every computer in the world you'll ever use is a von Neumann machine, and C is a lightweight, expressive syntax for the von Neumann machine's capabilities.
...
You also have to know C because it's the language that Unix is written in, and happens also to be the language that Windows and virtually all other operating systems are written in, because they're OSes for von Neumann machines, so what else would you use? Anything significantly different from C is going to be too far removed from the actual capabilities of the hardware to perform well enough, at least for an OS — at least in the last century, which is when they were all written.

C++

C++ is the dumbest language on earth, in the very real sense of being the least sentient.
...
Stuff takes forever to do around here. An Amazon engineer once described our code base as "a huge mountain of poop, the biggest mountain you've ever seen, and your job is to crawl into the very center of it, every time you need to fix something."
...
It's all C++'s fault. Don't argue. It is. We're using the dumbest language in the world. That's kind of meta-dumb, don't you think? The original brilliant guys and gals here only allowed two languages in Amazon's hallowed source repository: C and Lisp

Lisp/Emacs

All of the greatest engineers in the world use Emacs. The world-changer types. Not the great gal in the cube next to you. Not Fred, the amazing guy down the hall. I'm talking about the greatest software developers of our profession, the ones who changed the face of the industry. The James Goslings, the Donald Knuths, the Paul Grahams2, the Jamie Zawinskis, the Eric Bensons. Real engineers use Emacs. You have to be way smart to use it well, and it makes you incredibly powerful if you can master it. Go look over Paul Nordstrom's shoulder while he works sometime, if you don't believe me. It's a real eye-opener for someone who's used Visual Blub .NET-like IDEs their whole career.
...
Now C++, Java and Perl are all we write in. The elders have moved on to greener pastures too.
...
Religion isn't the opiate of the masses anymore, Karl. IDEs are.

Java

Java is simultaneously the best and the worst thing that has happened to computing in the past 10 years.
...
But Java's missing some nice features from C++, such as pass-by-reference(-to-stack-object), typedefs, macros, and operator overloading. Stuff that comes in handy now and again.
...
Oh, and multiple inheritance, which now I've come to appreciate in my old age. If you think my Opinionated Elf was a good counterpoint to polymorphism dogma, I've got several brilliant examples of why you need multiple inheritance, or at least Ruby-style mixins or automatic delegation. Ask me about the Glowing Sword or Cloak of Thieving sometime. Interfaces suck.

Gosling even said, a few years ago, that if he had to do it all over again,he wouldn't have used interfaces.

But that's just exactly what the problem with Java is. When James said that, people were shocked. I could feel the shock waves, could feel the marketing and legal folks at Sun maneuvering to hush him up, brush it off, say it wasn't so.

The problem with Java is that people are blinded by the marketing hype. That's the problem with C++, with Perl, with any language that's popular, and it's a serious one, because languages can't become popular without hype. So if the language designer suggests innocently that the language might not have been designed perfectly, it's time to shoot the language designer full of horse tranquilizers and shut down the conference.
...
Bad developers, who constitute the majority of all developers worldwide, can write bad code in any language you throw at them.
...
When in doubt, hire Java programmers who are polyglots, who detest large spongy frameworks like J2EE and EJB, and who use Emacs. All good rules of thumb.

Perl

There are "better" languages than Perl — hell, there are lots of them, if you define "better" as "not being insane". Lisp, Smalltalk, Python, gosh, I could probably name 20 or 30 languages that are "better" than Perl, inasmuch as they don't look like that Sperm Whale that exploded in the streets of Taiwan over the summer. Whale guts everywhere, covering cars, motorcycles, pedestrians. That's Perl. It's charming, really.
...
But Perl has many, many things going for it that, until recently, no other language had, and they compensated for its exo-intestinal qualities. You can make all sorts of useful things out of exploded whale, including perfume. It's quite useful. And so is Perl.
...
Now you can't read a book or tutorial or PowerPoint on Perl without spending at least a third of your time learning about "references", which are Larry's pathetic, broken, Goldbergian fix for his list-flattening insanity. But Perl's marketing is so incredibly good that it makes you feel as if references are the best thing that ever happened to you. You can take a reference to anything! It's fun! Smells good, too!
...
Like I said, though — until recently, nothing could get the job done like Perl could.

Ruby

Anyway, Ruby stole everything good from Perl; in fact, Matz, Ruby's author (Yukihiro Matsumoto, if I recall correctly, but he goes by "Matz"), feels he may have stolen a little too much from Perl, and got some whale guts on his shoes. But only a little.
...
And he somehow made it all work together so well that you don't even notice that it has all that stuff. I learned Ruby faster than any other language, out of maybe 30 or 40 total; it took me about 3 days before I was more comfortable using Ruby than I was in Perl, after eight years of Perl hacking. It's so consistent that you start being able to guess how things will work, and you're right most of the time. It's beautiful. And fun. And practical.
...
The leap from Perl to Ruby is as significant as the leap from C++ to Java, but without any of the downsides, because Ruby's essentially a proper superset of Perl's functionality, whereas Java took some things away that people missed, and didn't offer real replacements for them.

Python

Python would have taken over the world, but it has two fatal flaws: the whitespace thing, and the permafrost thing.
...
The whitespace thing is simply that Python uses indentation to determine block nesting. It forces you to indent everything a certain way, and they do this so that everyone's code will look the same. A surprising number of programmers hate this, because it feels to them like their freedom is being taken away; it feels as if Python is trampling their constitutional right to use shotgun formatting and obfuscated one-liners.
...
What's the frost thing, you ask? Well, I used to have a lot of exceptionally mean stuff written here, but since Python's actually quite pleasant to work with (if you can overlook its warts), I no longer think it's such a great idea to bash on Pythonistas. The "frost thing" is just that they used to have a tendency to be a bit, well, frosty. Why?

Because they were so tired of hearing about the whitespace thing!

StackOverflow DevDays Seattle

Back when I was at my former company, it was always a sore point with me that "continued education" seemed to be a foreign word. Even as I climbed the ranks of the company I felt like I was losing all the hard-won skills and knowledge that I'd accumulated over my years in misery at university. Sure, I knew our codebase, but I didn't feel like I knew very much else. With my skills becoming increasingly narrowed, and focused around *only* what my company did, I wondered if I'd ever be able to find a job *outside* the company.

So, when I became a dev manager at my current company, one of the things I hoped to do waa be the advocate for continued education for the guys under me. Now, for the first couple years all I had were contractors...and it's fairly hard to justify spending money on continued education for people who aren't even FTEs (full-time employees).

So having taken over a new group of devs, comprised entirely of FTEs, I was thrilled to learn Joel Spolsky and Jeff Atwood were organizing the Stack Overflow DevDays set of conferences. Given the extremely low cost for a ticket (A good conference is often $500-700, whereas DevDays was a mere $100 a head) I lobbied to get my entire dev team to go. And, to his credit, my Director agreed.

So, today, my whole development group and I got to spend the day listening to the speakers at the Stack Overflow DevDays conference in Seattle. On the whole I was extremely impressed and if they do it again next year, I highly recommend attending (or if there are still tickets in one of the remaining cities on the tour and you're nearby, snatch one up).

The keynote speach from Joel was about the dichotomy of power and simplicity when it comes to creating software. He made his way through a bunch of examples demonstrating the way in which we're constantly assailed by choices, many of them meaningless. These, he said, are simply an example of the designer being unable to make a decision. One example he gave was the Options bar in IE, which has an option named 'Enable Carat browsing'.

What does 'Enable Carat browsing' mean he asked? And even if you know, is it really a choice that's important? Shouldn't the designer have figured out the right default choice. Instead, it's one more in a series of bewildering decisions, that slow you down and rather than letting you do what you want with your software, force you to be its slave.

So, he asked, does this mean the answer simplicity? Companies like 37 Signals have gone this direction with their BaseCamp software (http://basecamphq.com/). Their "manifesto" is about keeping the feature set small, do one thing, but do it incredibly well. Keep the feature set minimum even if your customers are clamoring for new features. But, the thing is, he went on, that hte features you deem "absolutely essential' and the "minimum" requirements may not be exactly what the customer needs.

So, he said somewhat disappointedly, the answer is not simplicity. What then, is the answer?

In his mind, it's to only offer the choices that matter. He used the quote:

A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”
—Antoine De Saint-Exupery

You need to listen to the customer and give them the choices that are *important* but not assail them with things that they don't care about. Enable Carat browsing is one example, another the constant reminders in Windows Vista about whether you want to proceed, even after you've just clicked another very similar dialog box before it. Why would it ask you twice?

It's a reasonable answer I suppose, but also somewhat of a non-answer in my view. I personally think the 37 signals people are closer to the truth, but that's the topic for a whole other blog.

The next speaker was Scott Hanselman, who has the podcast Hanselminutes and he spoke about ASP.NET MVC and some new features. In some ways, I thought his presentation was the most entertaining. My face hurt from how much I was laughing and smiling during his presentation. To give an example, while mentioning he worked for Microsoft and that it 'really isn't so evil anymore', a huge picture of the Death Star was projected behind him. He even showed a language some at Microsoft were working on implementing in .NET that I thought hilarious, called LOLCODE. See it here:
http://lolcode.com/examples/hai-world

The content of Hanselman's talk was actually a bit light, his focus seeming to be more on entertainment than education, but having sat through years of boring Com Sci lecturs in school, I wasn't about to complain. He gave a brief peak into how the new MVC features of .NET worked and it appeared to me that Microsoft is going along the currently popular route of convention over configuration, which is something I was happy to see.

The next speaker, Rory Blyth, then was introduced and gave a talk about developing for the iPhone. My favorite part was when he listed the prerequisites and one of them was:
* A Mac
* "But are you sure?"
* "Yes"
* "But I have a friend who said..."
* "They heard wrong"
He talked about how the iPhone development platform had grown out of the IDE Steve Jobs had put together while at NeXT around 1986 and that, "it was cutting edge about 20 years ago...and it's pretty much the same now". As he went through the examples, it was interesting to see a bit of what Objective C looks like and some of the goofiness of the InterfaceBuilder. He then showed an example of how doing hte same thing in MonoDevelop (which allows one to use C# rather than Objective C) simplified development a great deal. MonoDevelop however is still under development and his example actually crashed while he was trying to run it. One thing Rory stressed again and again if doing iPhone development was that to read Apple's style "guidelines". He made huge air quotes around "guidelines" saying that essentially if you don't follow them, your app will not be published. He continued, "If you ever find yourself wondering how the UI should look, go look at an existing app on your phone (since presumably it made it past Apple's style-dictators) and do that".

Next, Spolsky came back up and gave a rundown of FogBuz7 and its new features. Being a "competitor" to ZPlanner (ha), I was quite interested to see some of the features. Back to his keynote in which he somewhat dismissed the notion of "simplicity" I feared some of the flashier features around his "Evidence based scheduling" might just be confusing and unnecessary, but I'd really have to look at it more closely. Of course, some may have been put off by what amounted to a 30 minute sales pitch, but for whatever else one can say about Joel he is a shrewd businessman. It's very rare for a conference of this caliber to be given for this price and I suspect a large part of the motivation is that FogCreek makes tools for developers. This is the best way to reach them. So, I view it as a symbiotic relationship and didn't mind listening to his spiel. And, like I said, it was interesting to see how his company had approached some of the same problems I have been trying to solve (far more simplisticially) with ZPlanner. I'll probably write an entire blog about Joel and some of my thoughts around his approach and FogBugz, but that will have to wait until next time.

Next up Cody Lindley talked a bit about jQuery. I've played around with it a bit myself while working on ZPlanner but was amazed to learn that something like 35% of all websites now use jQuery. He mentioned that many people only think of using jQuery to manipulated things already in the DOM, but it can also add things to the DOM. He went through some basic examples simply reinforcing what I already knew, that jQuery seems to be the preeminent choice for doing UI work these days. He also gave some useful websites to use when experimenting:
http://codylindley.com/jqueryselectors/
http://jsbin.com/
http://jqueryenlightenment.com/

After Cody, Daniel Rocha came up to speak about QT. I'd not been previously familiar with it, but apparently it's a tool that allows one to cross-compile code (including UI work) to native code across platforms. Currently, it already or will support cross compilation to the following OSes:
* Embedded Linux
* Mac OS X
* Windows
* Linux/X11
* Windows CE/Mobile
* Symbian
* Maemo

Nokia purchased the technology from a company called TrollTech, which had been selling it at a hefty-license fee ($3500 or so) and open sourced the SDK and tools in a hope to have more developers doing work for their phones. Daniel showed a simple example of creating a simple Windows app, then recompiled the same thing to Ubuntu and, hey presto, it worked.

And unlike, Java which has the overhead of running with a JVM, QT compiles to native code so it can do some pretty impressive things. In one example, he showed an OpenGL game (OpenGL is built in) written in QT and compiled to a Nokia device. Very impressive.

After Daniel finished, Ted Leung spoke a bit about Python. His speech was the dryest of the bunch by far and he'd unfortunately made the choice to make his slides from VIM screenshots, not realizing his highlighting scheme (using purple for keywords) was essentially unreadable to the audience. Even after they dimmed the lights my eyes were hurting from straining to see what he was showing, and because his presentation had a bit less pizzazz (it was essentially just showing hte contructs of the language ala a university course), I kind of tuned out a bit. The biggest failure on his part, I think, was that he failed to show me why I'd want to *learn* Python...other than academic curiousity.

Next Dan Sanderson, spoke about the Google App Engine and how it allows one to write apps that can scale as needed. It can run both the JVM as well as Python. It makes guarantees that for the 1st user or the 100000000th concurrent user, the experience will be the same, but imposes some restrictions on developers as a result, one being that it doesn't use a traditional RDBMS. There is a video on the AppEngine code page and I'd recommend watching it as I found the whole topic fascinating.

Last up, was Steve Seitz a local professor here at the University of Washington and while it was the least "usable", it was also the most fascinating. He spoke about Photo Tourism and their work to reconstruct three-dimensional models in space from photos from the internet. You have to watch the video for yourself because it simply blew me away:
http://phototour.cs.washington.edu/
(Watch the "Long Video")

Apparently, this technology is in commercial use on
http://photosynth.net/
Seemed to be old news to some, but I'd never heard of it.

Currently, Steve is working on recreating Rome from pictures downloaded from flickr, you can find his work here:
http://grail.cs.washington.edu/rome

That's it. It was a fairly exhausting day, but a ton of fun. Other than during the Python presentation I was never bored and though what I learned was fairly superficial, it was all fascinating and now I have a bunch of new, shiny things to play around in with my (meager) spare time.

ZPlanner: Making Things as Dumb as Possible

In my entry last week, I wrote about my experiences using TDD for the little personal project I've been working on called ZPlanner. At that point I was just beginning to refactor. I think that's one of the biggest things forgotten when people try to do some variant of TDD, or any programming really. The thought of starting to code without having spent hours and hours designing your solution to the last method signature is a really scary prospect to some.

If you're anythiung like me, much of your formal education tried to ensure you had all sorts of documentation, that you'd thought out everything throughly and it tends to be difficult to break out of this frame of mind. The thing is, TDD *does* work. The thing is TDD assumes that after you've written your test and you've gotten your code to pass it (by hook or crook), you go back and refactor. The problem is many people skip the refactoring step and it's absolutely essential.

That's where you fix all the crappy stuff you did to pass your tests, where you try to tighten up your design. Otherwise, you end up with just want TDD detractors say you will, crappily designed and implemented code.

In any event, TDD isn't really the subject of this blog. It's more what my experiences with refactoring ZPlanner and something I noticed about my own coding (at least for ZPlanner) in the process.

If I were to sum up the result of my this refactoring, in a single sentence it would be this:
"Make the code as stupid as possible".

For a long time, I've always claimed I liked to keep things simple. That complexity is the enemy of maintainability and that it's important to "do the simplest thing possible" per TDD. When I started looking at my ZPlanner code, though, there was all sorts of complexity. My primary class, EstimatedItem, was an abstract base class which represented both stories and tasks (neither had a more concrete representation), I'd subclassed it to create my Iteration object, used a recursive Hibernate relationship (which took some time to figure out), and because the object was recursive, I had a bunch of recursive functions and complex logic to figure out how to sum estimates and where a given EstimateItem might be in a node-like structure. Okay, great! But the thing is all of that likely made it pretty damn difficult for anyone who was first coming to my code.

I'd also created an abstract base class in an attempt to represent all form actions. Of course, that wasn't good enough for me, so I tossed in Java Generics, with the notion that every item that was modifiable via the interface had a parent and child type, each passed in as template arguments. The fact that in some cases this abstraction really had to be hammered together to fit--for example, in the case of a Project there was no parent entity so I passed in Object as the Parent type of Project (which really makes absolutely no sense).

Of course, I was quite happy with myself for all of this at the time! Oh, look at me, I have a recursive Hibernate relationship! Oh, look at my cool use of Generics! Oh, look at all this recursive logic I know!

I think that's often the case with us programmers. we want to prove to others that we know our shit. All that complexity was almost like a badge of honor to show that I, too, was worth my meddle. Of course, I didn't justify it this way. No, it all makes sense, I told myself. It's the simplest thing possible after all, because I'm using the least code. But then using the least code, is often not the same thing as doing the simplest thing possible. Just look at any gob of hacked up Perl where the goal seems to be to put as much logic on one line as possible and obfuscate it to the greatest degree possible. "If it was hard to write, it should be hard to debug!", goes the old saying.

When I stepped back, though, I realized that all my cleverness really hadn't gotten me anything that meaningful. Maybe I had a few less classes, but the ones I did have could only be understood via my huge swaths of comments. Nothing was clear.

So, I started to try and make my code as un-clever as possible. Rather than using a recursive object relationship with an abstract base class, i copied and pasted my EstimatedItem into separate classes for each of Project, Iteration, Story, and Task. I eliminated the recursive nature. After all, did I really need infinitely nestable tasks? When the hell would anyone use that, clever as it may be? Sure, I had a bit more code, but it was obvious. Suddenly, I *had* a Story class. And guess one, it has a private member variable called "Iteration", which go figure, was the parent iteration to which the story belong. That's not very clever is it?

It also meant that rather than having a single table for all my objects (because in Hibernate, that's the preferred strategy when using inheritance) with join tables for the data of subclasses, I had an 'iterations' table with just the iterations, a 'project' class with just hte projects, and so on. I also got rid of my BaseAction class that was poorly representing the abstract notion of a http request. Sure my action classes ended up having a couple extra private members, and a few extra getter and setters, but it also meant you could just look at the class and pretty much know what it was doing.

Thankfully, all of this was relatively easy, because I had a lot of test code. It made the refactoring relatively easy and gave me some assurance that things actually worked when I made changes. I probably "rewrote" about 50-75% of the code and it only took 20-25 hours. I still have around 85-90% test coverage and everything pretty much works.

All of this means, that when I look at the code now, there's nothing really to pat myself on the back about. There's nothing particularly clever. There is a bit more code now, but it's not that much more (maybe 25%), but what's there is all quite mundane and obvious.

And that's the whole point.

Interviewing Project Managers

Recently, I've been asked by my Director to help interview a number of Program Manager candidates. In the last two days, I've interviewed four different people. There was another flurry of candidates a few months ago in which I also participated, but we didn't end up hiring any of them.

Now I have a tremendous amount of experience in hiring developers. I've been responsible for staffing large teams in short order on at least three different occasions now. My rough guess is if you combined the people I've phone screened or interviewed in person, the number is somewhere around the three hundred mark. I've probably greenlit upwards of forty of fifty developers. This is just in the past three years mind you. Needless to say I have formed some very definite thoughts about the interview process for developers in this time.

What's interesting, though, is that as concrete and refined as my thoughts are when it comes to what constitutes a good developer interview (I'll leave such thoughts to a subsequent blog entry), I don't really have any similiar set of defined criteria for other positions. It ends up being much more 'intuition' based, which frankly bothers me.

When interviewing a developer, I have expectations that the candidate understands basic data structures, that when presented with one or more problems he can work through them logically and methodically, even if he can't quite arrive at the right answer. If, for example, he doesn't know what a linked list is or a hash table, and the situations in which one of the two data structures would be used over the other, he's going to have a really difficult time getting a thumbs up from me. If he can't solve some simple (and I mean SIMPLE) exercises in his preferred language, then he really is not getting through.

These past few days, though, have really highlighted that I don't have any similar questions for other positions. What I've done for these past few interviews with program managers is to look over the resume, look for things with which I'm familiar and spot check the candidates
knowledge there. Yesterday, for example, I had a Program Manager with an extremely impressive resume. It listed his workat numerous, huge companies, with all sorts of awards.

I also noticed that his education was in programming and that first six years of his career were spent as a Java developer in Java. So, I asked him to explain inheritance in a few sentences. I don't even recall exactly what his answer was as it was mostly gibberish. He certainly he didn't mention the idea of using the criteria of 'is a' to determine whether it a particular inheritance scheme was appropriate, or that inheritance, along with encapsulation and polymorphism, is one of the fundamental concepts of object-oriented programming. Nor did he say anything about using inheritance to help reduce code duplication by centralizing shared functionality in a base class from which other classes could be derived. Any of these would have been acceptable. I wasn't looking for a text book answer, just something that gave me an indication he knew what inheritance was at some basic level.

He couldn't do it.

He also listed Scrum on his resume, and so I gently asked him to give a basic explanation of that as well. Where one might expect him to mention the various roles within a Scrum team, such as Product Owner, Scrum Master, things like Sprint planning sessions, the daily Scrum, or retrospectives, he gave me a vague answer about how it 'helped focus on past and future requirements' and some other drivel.

In fact, I was happy that he had these things on his resume. He choose to list them, thereby making them fair game, and could provide a basic explanation of neither. I felt quite justified in justified in excluding him from future consideration.

More often, however, there is nothing so concrete on a PGM's resume that I can spot check. If the person doesn't have a development background, if he doesn't have any experience with Scrum (or another program management methodology with which I'm familiar) I end up having to rely almost solely on hypotheticals and questions about his past roles, which I still don't like much at all. The problem is such questions are (relatively) easy to answer.

Not to say that many candidates don't still do a miserable job with them, but if you're asking what a PGM would do when the client is making unreasonable demands, it's all well and good for them to say "I'd stick to my guns and refer them to the SOW, but see if we could accomodate their request reasonably with minimal risk"

When they have someone at the client company yelling at them saying how they missed some requirement or that the company promised this feature, though, it's much less likely they'll act that way though. Like many things, it is easy in theory, but in practice tend to be horribly difficult.

I may ask the person what was the most challenging project he worked on and why? What are the characteristics of a well-run project (beyond hitting milestones), and what the characteristics of a poorly run one? I may pose questions based on current difficulties faced by the team and ask him what he would do?

But the problem is, none of this really gives me any great sense the person is qualified. At best, it disqualifies him if he can't give a plausible, well-reasoned answer, or if he veers totally off topic. But a "good" answer guarantees almost nothing. I'm still struggling with the question of whether there are "good" ways to evaluate a potential PGM beyond a reference from someone who's previously worked with the person (which is the best thing) or hypotheticals like this, which really aren't all that useful.

The things is I think a *good* PGM can contribute immense value to a project. A poor one thought simply create work and stress, derails process, and is a huge liability to a team. And unfortunately most PGMs in my experience are poor ones. Bureaucrats more concerned with action items and holding status meetings that with thinking critically about the projects to which they are assigned and how to intelligently manage scope and the customer.

That what makes the good ones even more remarkable and valuable. I just wish I had a better way to find the good ones.

ZPlanner, TDD, and Over abstraction

As I've written once or twice before, I've been working on a lightweight Scrum/Agile piece of project management software. It's called ZPlanner and I've probably put all of 200-300 hours into it, though it's hard for me to say precisely. I started writing it when I thought I was going to be laid off from my job as development manager and wanted to brush up a bit on my Java programming skills. I somewhat arbitrarily chose to write a piece of project management software as it was a domain I knew and I've used XPlanner fairly frequently over the years. It's a decent tool, but has some serious shortcomings. But again, the motivation was mainly to try a few new technologies and do things "the right way" since I had no timelines or customers chomping at the bit.

Though I've been a big proponent of unit tests for some time and even TDD my formal computer science education instilled in me the absolute need to "architect" everything I do prior to writing a line of code. Ever since I came into contact with TDD, I've questioned this approach and I've done my best to move away from big upfront design (BUFD), but if I'm honest with myself I've not really done TDD proper. With ZPlanner, however, there was nothing on the line really, so for the first time I decided to really make a legitimate attempt at using TDD from the beginning.

I didn't think about a line of code or or spend days upon days trying to come up with some perfect object hierarchy, instead for once, I just started coding.

1. Write a failing unit test for new functionality
2. Do the simplest possible thing to make the code pass the test
3. Refactor
4. Repeat

What's interesting was that in an amazingly short period of time I actually had something that worked at some basic level. My first milestone was to have a simple web page in which one could enter a task with a name, description, and an estimate and save it to the database. As trivial as that may seem, using my old approach I would have spent days and days (or longer) agonizing over some design, thinking of every possible use case, going back and forth on which technology to use, and so on. It probably would have been several weeks before I even was at the stage to write a line of code.

Instead, within 40 hours I had set up Maven2 for dependency management and to do builds, Hibernate Annotations to manage database persistence, Cobertura to do unit test coverage reports, and Mercurial to manage my revisioning. And I even had a working web application to boot!

And it did something meaningful, which was allow me to save tasks with estimates.

It was basically a slightly glorified Excel sheet (minus all the crazy macros and editing and so on).
As I kept moving forward I tried to repeat the TDD mantra

1. Write a failing unit test for new functionality
2. Do the simplest possible thing to make the code pass the test
3. Refactor
4. Repeat

Of course, "doing the simplest possible thing" is a subjective assessment. Sometimes what seems the "simplest thing" is not at all the simplest thing. And though I'd overcome my education with regard to doing some BFUD, and my need to be a "good computer scientist" and abstract things to reduce duplication reared it's head.

Unit test by unit test I built up the application, refactoring, removing duplication, doing "the simplest thing possible" until a week or two ago. I actually had a system that could store a hierarchical tree of projects, iterations, stories, and infinitely recursed tasks, it could roll up estimates using some fairly complex logic, it had validations for the web app, a clean, easy build process, the ability to create users, an Ajax drag-and-drop interface to move around stories/tasks, and about 80-90% code coverage. All with about 20 files and only a few thousand lines of code.

Talking to one of my colleagues (thanks Jim!), though, he pointed something obvious out. Part of my interpretation of doing the "simplest thing possible" had been trying to reuse code as much as possible. This in turn ended up resulting in object model in which Iterations, Stories, and Tasks had all been abstracted into a notion of an EstimatedItem.

This meant most of my functionality was able to be captured in only a few lines of code, but it also meant it was pretty non-intuitive. The comments were strewn with explanations about how an EstimatedItem in one case represented a Story and in another case represented a Task.

I'd also oddly (and wrongly) decided to subclass the Iteration class from EstimatedItem as well, reasoning that the fundamental characteristic of an estimated item was that it had a name, description, and an estimate. Well, I figured at the time, an iteration has all of these things. It only differs in that it has a start date and an end date. It seemed to make sense at the time and it also seemed the "simplest thing possible"

The problem of course is that an Iteration IS NOT a story.
A task ISN'T even a story.

One can abstract things sufficiently such that all things are the same. I blame Java and its ilk for this to some degree with its notion of everything being an Object. Okay, great! Everything's an object. What does that MEAN? But Java isn't really to blame either, I've seen plenty of other examples of over abstraction. The problem is we're taught in school and by our peers that reuse is bad.

But then if our abstraction makes the code much harder to read. Rather than looking at the code and seeing Stories and Tasks and things that have concrete meaning, we're dealing with abstractions that it really takes some thought to understand. Maybe it's a little less code, but it's much harder to understand.

I don't really blame TDD for my mistake. I think it could just as easily (perhaps even more likely) happened with a traditional approach. In any event, I'm no rewriting a large portion of the code, copy-and-pasting even and where I had three classes I know have 9-12. I guess this could be seen as a bad thing, but suddenly what was abstruse and explained in comments makes perfect sense just looking at the code

Whereas one used to add a task like this

EstimatedItem story = new EstimatedItem(...)
story.addSubItem(new EstimatedItem(...))

is replaced with this

Story story = new Story(...)
story.addTask(new Task(...))

The latter is infinitely easier to understand in my view. And hopefully once I've exploded my classes I might be able to see some commonalities that *are* meaningful, rather than my flawed notion of abstraction. Now, since I have unit tests for all this stuff, it's pretty easy to change them and make sure I'm still passing the same tests. So, TDD is what makes this *better* design possible. And now we're back on TDD and we've come full circle.

Until next time.

Thoughts on Offshoring, Part IV

This is the fourth and final installment I'm going to write about offshoring. I've already written about the two implementations of offshoring I've been part of, what I thought worked, and what didn't.

Let me recap the different approachs and my thoughts about what worked and what didn't.

The first approach:

• Use Scrum with teams spanning geographical boundaries. Each onshore team was augmented with a few offshore resources.
• The offshore location was chosen so as to minimize the time difference. Using resources basedin Costa Rica reduced this difference to 3 hours, even with us being on the West coast
• Offshore resources were screened so as to try and ensure competency and language fluency.
• Direction for these offshore resources came from onshore leads and all management of the project was done onshore.
  

The problems:

• Offshore team members did not feel like "full-fledged team members", frequently waiting for onshore leads to tell them what to do, despite our attempts to involve them in Sprint Planning and retrospectives.
• Because the offshore developers did not take ownership, it fell to the onshore devs to "assign" them work. Often the onshore devs did not feel confident in giving the offshore resources critical work items, and so they ended up being assigned low-priority work that did not contribute significantly to the business value of our project
• Even with our daily Scrum being held via conference call, due to communication difficulties, cultural differences, or something else frequently the offshore resources wouldn't give much context to what they were doing and so these Scrums were of relatively low value
  

When we finally made this approach worked well:

• We rolled-off all the offshore developers who had been seen as contributing marginal value, keeping only the best resources
• We reduced the team sizes, so the offshore resources were not "extra capacity" but were absolutely essential to the team's ability to deliver on its commitments.We started using online collaborative tools (namely the Wiki) to ensure the offshore team members knew exactly what their priorities were and so that onshore leads/managers could effectively communicate the highest priority work items to them.
• We removed the members of the onshore management team who had failed to give clear direction to the offshore team members and consolidated the management of the project in the hands of a few senior team members

The second approach:

• More traditional Waterfall development schedule
• Onsite architects and leads largely flesh out the initial design and implementation for new features, which are then handed off to largely self-contained offshore teams
• The offshore company provides "embedded" team members who are onsite. This includes technical program managers as well as development staff. There are also managers who are co-located with the offshore developers
• Onshore developers serve largely as giving direction to the offshore developers and reviewing the work they have done

The problems we currently face:

• The offshore developers were not directly screened. Instead, we choose the leads who were allowed to choose their developers. In practice, who knows if they screened them. I suspect they were simply "given" resources by their offshoring company. This meant that the quality of the offshore resources was somewhat inconsistent at times.
• Projects estimated to require a single dev for a few days, are assigned to multiple offshore developers since there is so much capacity offshore. This ends up negating the cost saving of using offshore in hte first place.
• Related to the previous point, there are no tools to easily know what the offshore members are working on.
• Given the ebb and flow of project work and the massive size of hte offshore team (almost 30 developers), we often end up with a bunch of developers with nothing to do. We are rate limited in our ability to farm work out to them, because it tends to need analysis by onsite leads which is rate limited.

What is working and current improvements we are attempting to make:

• Some of the offshore developers are extremely competent.
• The onsite technical PMs are also extremely competent and better than many of our FTE managers as they have technical backgrounds. It should be noted there are only a few and they are over 10-15K per month per head, so the cost savings are debatable.
• Having onsite technical PMs and resource managers also helps to overcome the pain associated to a significant time difference between locations. Of course, it only "transfers" the pain to the onsite representatives.
  

Concluding thoughts:

I have done something of an about-face with regard to offshore. I'm now convinced that there is some great talent out there. I still, however, have yet to be shown that there is anything better than a motivated staff of full-time, onsite developers, both in terms of the quality of what they produce, but also the cost at which they can produce it. The communication overhead and the low visibility onsite members have into what offshore team members are doing are severe impediments.

These impediments can be overcome by using onsite representative to "bridge the gap", but the cost of these people is significant and negates much of the value proposition of offshoring.

If *I* was the CEO, the way I would use offshoring would be:

• Recruit offshore developers using even the same criteria as used for an FTE. In fact, they have to be *better* than an equivalent onshore developer as they have signficiant geographical and language barriers to overcome.
• Build your offshore talent pool slowly, add one or two here and there, augmenting existing staff. Just as you could not expect to build a great programming staff by hiring 30 people all at once, don't expect that your going to end up with a "great" team by replacing all of your staff with offshore devs.
• Ensure you have the tools/communication vectors in place to communicate work to your offshore devs. it can be as simple as a Wiki. But you absolutely need to know *what* the offshore resources are doing on any given day.
• Prefer resources located in a timezone as close to your own as possible. Costa Rica has some great talent.
• Ensure your onsite program management team is top notch. Having mediocre PGMs can be worked through wtih onsite people picking up the slack. If you're using offshore, it ends up being a disaster.
• Continue to recruit onshore developers and do the majority of your work via these resources. Use offshore only as an augmentation to this staff rather than a replacement.

I'm not sure that totally captures my thoughts, but it will have to do for now.