Thought on Offshoring, Pt II

My first experience with offshoring, about which I wrote a bit last time, was fairly typical I suspect: have the expensive architect-level folks come up with a high-level design, then throw it over the fence to cheap code monkeys to implement. This is the dream of the bean counters and like many of the most flawed ideas in software engineering, the concept attempts to draw (unsuccessfully) from other disciplines, in this case manufacturing.

In fact, when my current company started making a concerted push towards offshoring a number of us raised signficant concerns, to which one director said simply "Well, Apple had made it work with the iPod".

"Designed in California, Assembled in China" says the back of the iPod.

Well, if Apple does it, it must be a good idea!

Nevermind the fact that manufacturing is completely different from software engineering. I don't even think the most ardent proponent of waterfall design would argue otherwise. And this is the fundamental flaw with much of the thinking that comes from business-oriented executives: they don't understand how moving "thought work" to 3rd world countries difers from opening plants for the manufacture of textiles (The fact that the director mentioned above was someone who came from an engineering background made the statement even more baffling). The only way I can understand their decision making is if the truly believe software is nothing more than a widget to be manufactured.

About 9 months ago I found myself in charge of development for a fairly large scale migration project. It had already been underway for about six months when I joined the project. The staff was largely comprised of onsite contractors, but recently the decision had been made to start moving to offshore to save money.

A few of us in leadership roles on the technical quickly circled around to figure out what the hell this meant to us all. We had quite a few concerns, most of which centered around the difficulty we expected in communicating with non-native English speakers located in another timezone.

We knew our company had several choices for offshore companies. One of the three companies had locations in the Ukraine and India, the other only in India, and the last in Costa Rica. Given that Costa Rica was only three hours off our timezone, we decided this offered a clear advantage in communication, even if they cost a bit more. We successfully convinced those above us that the marginal additional cost was worth using this company for the project.

In addition to their location, we were also worried about the quality of the resources. Of course, we needed them to be competent programmers, but we also needed people who could communicate clearly via email AND via phone. We told our Director that we wanted to screen every potential resource. There was reticence on the HR end of things to do this but the team volunteered to round robin the screens so we could speak to every individual.

Obviously, flying these people out to do in person interviews was out of the question. Even more than that the sheer number of people we had to screen through was fairly signficant. So, we settled upon a brief 30 minute screen conducted via WebEx. Obviously, we'd never hire full-time employees this way, but in lieu of something better a basic screen where we asked some basic questions, could assess communication (and specifcally English as a second language) skills, and toss them a simple programming exercise was a huge step up from simply taking on whomever the offshore company threw us.

In fact, I found this accelerated screening process to be fairly effective as in most cases those who couldn't speak English at all or who were completely incompetent as programmers were weeded out in the first 15 minutes. And watching another programmer via WebEx while he programmed away in his IDE of choice on his *own* desktop was the truest test of a dev.

Ultimately, I'd say the resources we saw from Costa Rica were just as good as those in the US. There were plenty of incompetent people who the offshoring company swore up and down had years of experience, but then that's no different than my experiences interviewing good ol' Americans in onsite programmers. The simple fact is most professional "programmers" can't program. Nationality has nothing to do with it. Granted, some of the offshore resources had difficulty in communicating and in most cases this actaully resulted in us screening them out. The main thing I think it's important to remember *isn't* that the resources are from offshore. They are not intrisically less talented than Americans. No, the real problem is the genesis of the offshoring idea, the real motivation stems from a devaluation of the resources themselves. The qualitative aspect is forgotten, so the idea of *screening* these people, which almost no one would sanction when considering hiring full-time employees, seems somehow acceptable.

It isn't.

Now staffed, we had to decide if the offshore resources would be a distinct team or augment the onshore staff. Since even our onshore developers weren't completely clear on what we were building, it seemed like it'd be pretty difficult to"design" a solution then hand it off to a self-contained team to implement. There was going to have to be a lot of back and forth between onsite and offshore resources. That combined with the fact that our offshore resources did not have much experience with our code base led to the decision to augment our existing teams with offshore resources, rather than spinning up a siloed team.

As we made out way through the next 6 months of work, sprint by sprint, there were a few definite problems that arose. One of the biggest issues was that it was very difficult to have visibility into what the offshore team members were doing at any one time. Part of this was certainly attributable to the fact that We 3-4 Scrum teams with over 25 total devs. I'm not convinced the problem would not have also occurred with onsite devs. The difference was that if the offshore developers had been in the office, it would have been relatively easy to wander over to their desks, ask them they were working on, and sit them down next to someone else to assist. Doing the equivalent via email and phone calls ended up being quite difficult and if many cases, simply didn't happen.

One of the biggest issues with using the offshore team members effectively was that our implementatoin of Scrum was itself very poor. I triend to make sure we had some type of sprint planning with each of the teams and made every effort to include the offshore resources, but typically the offshore resources would not speak up at all during hte planning sesssions and were not really engaged. Once we'd figured out the work, it was frequently the case that it had been determined by the onsite dev leads and they didn't feel confident in giving it over to the offshore resources without having to do most of hte work themselves since it required too much explanation and back and forth. This often left our offshore resources with either nothing to do or stuck fixing trivial, low priority bugs which didn't really matter that much.

Additionally, even when we had very clear tasks to give to the offshore team members, many of them would quickly get stuck, requiring as much assistance from the onsite leads was it would have taken the leads to do the work themselves. Additionally, they rarely spoke up when they had no work.

Many of the problems in my view, weren't so much *because* of the offshore team members, it was simply that we did not have a good enough handle on our process to overcome the geographical distance. Additionally, while we had screened many of the team members, a good number of them were inherited from other teams that had previously used offshoring, and so language and skill issues were still present undermining hte confidence of the onsite devs in their offshore counterparts.

What was most interesting to me, however, was what happened as we started to 'ramp down' the project. As we neared the launch date for the project I was under immense pressure to roll off as many resources, onsite contractors and offshore developers, as possible and as quickly as possible. At the time, it seemed like the worst of all possible worlds as I was being asked to roll off team members when the project was at its more critical period. I thought it spelled doom for us.

In fact, somewhat countintuitively (though it shouldn't have been as it's the same principal described in the Mythical Man Month, by Fred Brooks) we started to see the opposite. As our onsite program managers were rolled off, one by one, and as we rolled off the those on the offshore team which seemed to be contributing the least, we started picking up considerable momentum.

It was not so much the fact that we were using offshore resources, but rather that, even in spite of our best efforts, we were carrying a lot of dead weight that weighed us down. This dead weight was present both onsite and offshore Towards the end when we'd rolled off all but hte very best of the offshore resources, we only had 3-4 offshore devs.

In fact, when we'd whittled our offshore resoruces down to about 3-4 from the original 15 we were left with very capable programmers. In fact, the best of them were the equal of any of our onsite devs. They thought critically, communicated status regularly, and took proactive action when necessary.

And now that the program managers who were managing the teams had been gone and they were put in the hands of myself and another technical person and we gave the offshore devs clear direction. Soon, we started to gain a good deal of momentum. And what had seemed impossible to do with 30 people, seemed now achievable with our team of 10.

And that, to me, is the biggest problem with offshoring. It many ways it originates from a mindset that denigrates the value of the individual. Often you'll hear it in the context of, we'll replace 10 full-time developers onshore with 30 offshore developers. But the *only* way it can be successful is if the same values you'd apply to your onshore developers are applied also to the offshore resources.

Like everything it's all about the people.

I guess this is part 2 of 4. Next time I'll write about the implementation of offshore with my current team, how it is working and NOT working. and in the final blog entry, I'll summarize my thoughts on where offshoring is useful, my own thoughts on how to help make it successful, and conclude.