Thursday, September 10, 2009

Thoughts on Offshoring, Pt I


My first experience with offshoring came about 4 or 5 years ago. My company at the time had hired developers in Romania to rewrite our in-house log grinding software. I hadn't really been involved in any of it, but I was forwarded an email from our Director of Technology. The gist of the email was that the code made him weep. It had enough gems to keep http://www.thedailywtf.com/ going for a year.

One bit of code to calculate the current date that ran something like 80 or 90 lines of Perl. Apparently, this:


($DAY, $MONTH, $YEAR) = (localtime)[3,4,5];

was just too easy for our Romanian friends. The rest of the code was similarly awful. This set the tone for my thoughts about offshore, which could be summarized in a single word:
No

Of course, with however much vehemence I've argued against the offshoring trend, I'm usually not in a position to make the final decision. And the bean counters at the top always love it. So, I've had many subsequent encounters with it over the past several years. Some of it has confirmed what I believed then, but I've also had a bit of a reversal on some of my opinions, namely I now feel there is a (limited) place for it. I kinda feel about it how I feel about the 'goto' statement. Yeah, really fundamentally, I think it's a poor choice, but sometimes it's the best option. You really need to know *why* you're doing it, though.

The largest chunk of my experience has been over the past 8-9 months. I've worked extensively with two different offshoring companies, located in two different countries, and with very different implementation strategies. Both of them have had significant problems, but these experiences are also what have led to my somewhat more nuanced view of the subject.

There are some great developers offshore. Let's get that out of the way first. The best of the best overseas are just as good as the best of the best in the U.S. Now, I didn't originally feel that way. I used to think that the mindset of those trained overseas just lacked the creativity and incisiveness of those trained here in the US or Europe. But I really no longer think that's the case. The problem of course is this. It's immensely difficult to recruit and hire *great* programmers even when you're interviewing and working wtih them face to face. Compound that with the fact they are located in another time zone and it's near impossible.

Of course, you're also going through a third party to hire these guys. Their companies will always tell you how great every one of them is, but they are looking to make a profit and it's unlikely their criteria for a good developer is anything similar to yours. Hell, I don't trust most of the people within my own company to make intelligent hiring decisions when it comes to developers, so it's pretty much a lost cause when trying to bring on offshore resources. But from experience the *only* way you have any hope to make offshoring work is to have the exact same (if not higher) criteria for every one of your offshore resources that you'd have for your onsite devs. Really, given the communication barriers, the time zone differences, and so on they need to be *better* than the guys you'd have onshore. You *must* insist on interviewing every one of them. Yeah, it takes a lot of time, but to do otherwise will guarantee that you're dragging around a bunch of dead weight. That a huge problem onshore *or* offshore. Of course, the problem with this is that interviewing and having stringent criteria for your devs, flies in the face of the primary motivation for using offshore.

Let's not be coy. The *only* reason any company ever chooses to use offshore is to save money. That's understandable to an extent. The problem is the people responsible for the decision to use offshore are almost invariably those from a business background with very little technical knowledge. To them it's as simple as replace X programmers in the US costing Y dollars with X programmers in India costing Y/2 dollars. The idea that one *great* programmer will be ten times as productive as a mediocre programmer is completely incomprehensible to them. This is of course a problem with any large company where the senior leadership does not have strong technical backgrounds. They make decisions that simply don't make sense (I'll leave further discussion of this to another blog).

Of course, that's why it's so difficult to convince the people at the top that offshoring is not the clear win it seems to be. They just don't understand how much difference one *great* programmer can make. Making them understand that 3-4 great programmers who are working as a cohesive team can equal or exceed the output of 40 mediocre programmers in India is not something they can really wrap their heads around. I've tried to explain it many times, but (most) business people simply can't see past that bottom line on an Excel sheet.

I guess that'll wrap things up for now. Next blog I'll try to get into more details, talk about the two main strategies I've seen employed for offshoring and the relative advantages/disadvantages of each, as well what I feel are the necessary criteria to make offshoring work, and where it may actually be worthwhile.

4 comments:

Adam said...

You're pretty dead on in everything you said here (for a change ;)). The company I work for has developers in several countries outside the U.S., including Great Britain, Germany, Japan, India, China, and several other smaller locations that we don't hear about on a regular basis. I'm talking about developer teams, because we have data centers and satellite offices in more places than I can count.

So, how does such a setup work? Pretty much as you described and as you would expect. Some of it works, some of it doesn't. For instance, our developers in the U.K. and Germany are very strong, but the same can't be said for those in India and China. That sounds racist, but it's not meant to be. It's just pure mathematics and culture. If you have 100,000 people interviewing for one job, the chances of properly screening each one and choosing the best candidate are highly diminished.

Most of our technical support (read: non-dev) is located in India. The reason for that, more likely than not, is once again a game of numbers. Lots of mediocre people at a cheap price.

It's almost like a distributed computing setup: if you throw enough low performance machines at a problem, it's bound to get solved faster than with a few uber powerful machines. That may work with technology (read: SETI@Home), but it doesn't really scale when applied to humans.

In the end, the bottom line always wins.

Shaun said...

"...3-4 great programmers who are working as a cohesive team can equal or exceed the output of 40 mediocre programmers in India"

I just had to repeat this because I agree with it wholeheartedly.

Code Monkey said...

@Silent Observer

Definitely interesting to know how it works at your company (since I know who they are. lol). If I was using an argument of mathematics, though, I think I'd conclude the opposite. It would seem if the population of a given country is orders of magnitude larger, there would also be more *brilliant* programmers, so you'd have a better chance of getting them. But I don't think it's simply mathematics. I think a strong component is cultural, how the schooling system in a given country works, what's valued by the overall society, and so on.

But I would completely agree that much of the impetus for offshoring is replacing expensive good people with a lot of medicore ones, which is, to be frank, retarded and will *never* be effective. That's not to say it can't "work" just that if people are claiming cost savings they are deluding themselves.

And, yes, sadly the bottom line usually does win.

Code Monkey said...

@Shaun

Even though I wrote that, I'd caveat by saying 3-4 great programmers are worth 30-40 mediocre programmers, irrespective of whether they're onsite or offshore.

I feel like I just keep saying the same thing, but it really is the crux of the matter: individuals matter.