Scrum and the Real World

Recently, I attended a two day CSM (Certified Scrum Master) training class from a local company which both practices Scrum internally as well as serving as coaches and teachers to other companies wishing to move to Scrum. It was a good class, though I'd attended essentially the same class about two years ago when I first came to my current company.

Let me start by saying I'm a fan of Scrum, I tend to think it's a reasonably sane approach to managing a project. But I don't think it the be-all-end-all fo software development and I think there are other ways to get to the same destination, namely, high quality software, happy people, and minimal overhead.

My biggest problem with Scrum is that I feel sometimes it's proponents sweep "real world" issues under the table a little too quickly. Namely, they often lack a certain pragmatism. In fact, at least one of the two instructors this time around seemed fairly pragmatic. But I wrote him after the training and here's the email I sent.

I hate to sound like a broken record, but where I come into conflict with those who preach Scrum (being a big fan) are that they presuppose a "start state" that just does not exist in many cases. Some of the biggest mistakes I feel that I’ve made with Scrum have been to cede authority to the team, who were simply not equipped to handle such responsibility properly.

I liken it to giving a credit card to a college student away from home for the first time. This student should be responsible with his credit card. He should pay off the balance each month so there is no interest. He shouldn’t make purchases he would not otherwise make if he didn’t have the credit card. These are all absolutely reasonable
expectations to have for the recipient of our credit card.

So, we give him the credit card. He’s "empowered". What does he precede to do? Maybe he does exactly those things. But for some, even though previously they might have been fiscally responsible, the notion of a credit card is intoxicating. A person from such a group suddenly finds himself racking up all sorts of purchases he can never pay off and with no one watching over him slides deeper and deeper into debt from which he can never extricate himself.

So, is the answer then to never give this person a credit card? How do teach the person to use the card responsibly without giving him a card with a 20,000 limit off the bat?

Some of my failures with Scrum were due to placing *too* much into people who didn’t really deserve it. Oh, they seemed to deserve it, but they didn’t. And fundamentally that comes down to people. But "people" are always the hard part. As I think you guys mentioned, put a team of great people on a project and irrespective of methodology they’ll figure it out.

If the prerequisite to make Scrum work is a team of highly motivated, competent people who value the things that Scrum values, I’d submit it’s not adding much value. They would succeed anyway. Sure, Scrum’s a great way to get to that destination, but I’d submit plenty of other approaches that would work just as well.

Just teaching people Scrum and repeating the tenants of the Agile manifesto to them doesn’t make the *truly* value those things. Some percentage will really convert, but many will still value what they always valued. They will never apply Scrum with enough thought or criticality to be successful. And if they do, I would submit they’d probably succeed just as well without Scrum (or with a whole lot of Scrum-buts).

To me, the real promise of Scrum is for teams that *are* dysfunctional. What if everyone isn’t motivated? What if everyone isn’t competent? Because in more companies than not this is the case. Sure, you can say fire the bad people, get only good ones, etc (and I try to do this), but let’s be frank about the reality of many people’s lives. I would suggest Scrum still holds an awful lot of value. And that’s where the gap is. Making Scrum work with the dream team is easy. How do you make it work with reality is what is so tough and what I think sometimes is neglected.

Looking Backwards

Recently, my director published a new "capacity sheet". It's an Excel sheet with a high-level summary of all the projects scheduled for the upcoming year, with breakdowns in hours. The hours themselves are based on some simple formulae. A salesperson assigns the project a t-shirt size (S, M, L, or XL) and this translates to a predefined number of hours.

The intention of the sheet is not to plan capacity of our systems (i.e. transactions per second, etc), but rather see if we have sufficient resources to do the work. Let me start off by saying, I like the idea...in principle. In fact, at my old company it seemed no thought was given to actually staffing for the work coming in, which meant all of us ended up working ridiculous hours. So, the idea is a good one.

The problem is the implementation. It’s fine to assign t-shirt sizes to projects that are 9-10 months out and haven't even been defined (you can't really do anything else), but we use these boiler-plate numbers even for the projects that are nearly started. Further, as you move further out in time, the exercise becomes more and more meaningless because invariably the client changes his wishes and prioritization of work. Now, I made these points last time my group went through the exercise and that's not really the point of this blog.

The point of this blog is that no one is that I've not heard one person mention the need to look at the exercise we went through last year. Having gone through the exercise before and having voiced a number of concerns about the value of the exercise, I decided to go back and look at the sheet from last year. What I found there was very telling and hopefully will persuade others to reevaluate the way in which we're handling things this year.

Almost 70% of the projects on our plan never even happened. And for those that we did do, the boiler plate estimates we used were not even close to the actuals. Now, this didn't specifically cause us problems insofar as we delivered on our commitments. That is primarily because in general the formulae grossly overestimate the amount of resources needed for any given project. Of course, I'm not sure the actuals in our time-tracking system will prove this, because as we know from Parkinson's Law, work expands to fill time.

The problem is going through the exercise this time around using these numbers uncritically, tells us that we're short staffed. Nothing could be further from the truth. I have many more developers than I need and many of them end up billing huge amounts of time to "sustainment" activities or training. Since these guys are offshore, I really couldn't say definitely what the hell they're doing half the time. I'd say we're probably 25-40% overstaffed.

But again, that's somewhat incidental to my point (I do have one!) for the blog and that is when doing something. George Santayana made the famous statement that ‘Those who do not remember the past are doomed to repeat it’. In general, I prefer Kurt Vonnegut’s rebuttal ‘I’ve got news for Mr. Santayana, we’re doomed to repeat the past no matter what. That’s what it is to be human’.

In the case of software engineering (and most sciences), you absolutely *have* to go back and look at what went right and what went wrong to improve. That’s what of the things I think Scrum and Agile absolutely nailed, the idea of iteration/sprint retrospectives. Unfortunately, things like that seem to be the exception rather than the rule. More often than not people don’t do this, so we continue to tread a sea of mediocrity in which nothing ever really improves.

TFS Migration Or: There's Still No Silver Bullet

Recently my company started a migration project to move all of our internals tools over to Microsoft Team Foundation Server (TFS). If you're not familiar with TFS, Microsoft bills it as an Application Lifecycle Management (ALM) suite which handles everything from source code control, to bug tracking, to project planning.

Currently, tools and processes vary across teams. My company has been the result of mergers, of growth and shrinkage, and the toolset reflects this. Many tools are shared across teams, but there are differences. For example some teams use HP's QualityCenter for bug tracking, while others use Bugzilla, while my group uses an in-house bug tracking system written years ago and no longer actively maintained.

The promise of TFS is to standardize our processes and tools across all teams and to make life simpler by having a single toolset that can track any given item of work from inception through completion and after.

TFS has been the pet project of one of my fellow development manager's within the organization. In fact, he was my boss when I first joined the company. He's the one that has pushed hard for TFS, including bringing a consulting agency onsite to evaluate our tools and see how TFS might help us (funded by Microsoft it should be noted), putting together the proposal for the company’s operating committee, and serving as evangelist for the tools.

He’s also made repeated (and somewhat forced) attempts to show how TFS will revolutionize our lives.

Don’t get me wrong, some aspects of TFS seem intriguing. It has some very powerful features. One of the most interesting is its ability to recreate the environment within which a bug occurred, so that, say a tester encounters a bug while testing, he can save the state of the servers on which it occurred, and the developer can at some point later, actually spin that exact environment back up and step through the code as if it was executing at that moment. That’s pretty cool.

As a source control system, TFS seems to be fairly decent. I think it misses the boat in its reliance on a centralized repository (versus the more recent developments in distributed revision control systems), but its notion of discrete changesets, and its management of branches seems to be significantly better than CVS.

But at its core, it’s a Microsoft product. And that has some very serious implications. Joel Spolsky’s keynote address at the recent Stack Overflow conference was about the dynamic between keeping things simple and giving people the power and flexibility they need to do what they want. He came down to the conclusion that the problem isn’t necessarily in giving people too many choices, it’s in giving them choices that are meaningful to them.

To me, the TFS interface is typical Microsoft. Incredibly powerful and flexible, but for you base cases, the stuff that you want to be able to do intuitively and easily, it makes something that should be a click or two into a 10-step process, that is needlessly complex.

One example is creating a new task in TFS. So, you have a story in the system and want to break it down into tasks. You’d think that’d be simple. So, you click ‘Create Link’, then you are taken to another menu, where you select the ‘Link type’, of which ‘Task’ is one choice. Then you make a selection of ‘affected by’, ‘affects’, etc.

I don’t know about you, but to me this should be a one click operation. All the abstraction and flexibility it just annoying for 95% of the time I’m using it.

But I suspect TFS will be an improvement on what we have currently. The thing is, I don’t think it will make that much of a difference. A good toolset (and one can argue over whether TFS is a “good” toolset) is all well and good. But the real problems in software engineering are not the tools, but how they're used. And without a significant investment into making our processes simpler, having people who apply those processes with critical thought and not blindly following some set number of steps, is what makes things more efficient.

And even if TFS is the greatest toolset is the world, go back to Fred Brooks' seminal ‘No Silver Bullet’ article. Such systems only address the accidental complexity of software engineering, not the inherent complexity. As such, they cannot result in order of magnitude increases in productivity.

To me, the money which has been spent on TFS (which is not insignificant) would have been far better spent in bringing in experts to look at our processes, our team organization, at reducing the bureaucratic processes that waste so much money and time, not some shiny new toy.

Business Viability Or: Should I Open Source ZPlanner?

I've now been working on my little project tracking application, ZPlanner, for nearly a year now. It's certainly not been a full time effort, rather I work on it here and there when time allows and when my attention doesn't drift elsewhere (to blogging, the book I've been working on, and other random stuff), but I've made reasonable progress on it. It's actually getting reasonably close to being a *uable*. Mind you it's nothing revolutionary. It's just a simple web application, but nonetheless I'm fairly happy with where things are at.

I started working on ZPlanner because I thought I was going to be laid off from my job as a development manager around last March. Due to a reorg, my job was promised to someone they'd relocated from the East coast offices which left me without a position. The job market at that time was so horrible, I felt like I'd be lucky to find a position as a developer somewhere, so I figured I should brush up a bit on my programming skills, seeing as how I do so little for my job these days. Luckily, however, I was able to snag another position within the company--it was a better one to boot. But I kept working on ZPlanner anyway as I think there's a need for a simple, project tracking application.

I've toyed with the idea of actually trying to sell it--at an affordable price--as the only options for project tracking are either free (and kind of crappy) or fairly expensive. But to be honest, I'm not sure how viable it is as a product. There are a lot of entries in the market, even more if you consider that many software stacks that label themselves as "bug trackers", such as Jira, have also incorporated project planning features.

But the idea of starting up my own company, even if it barely made any money is incredibly alluring. Like many developers, the idea of being in charge of things and running a company how *I* think it should be run is an exciting prospect. And in lieu of any better product, I figured, why not try selling ZPlanner.

But the thing is, over the winter break, I had what I think is a really good idea for a piece of software. I won't go much into the details, but from what I can tell from some brief Googling, it's a niche that has not yet been solved, has the potential to be sold to larger companies and deliver measurable increases in profit and efficiency, and is a problem which would not be trivial to solve. In fact, the reason I thought of the idea was because the lack of it caused me considerable frustration. I was dealing with a company I would have assumed to have software as I describe, but they did not and as a result I ended up being one very pissed off customer. I've also found some theoretical whitepapers on the subject, but very little in the way of concrete implementations. I think the potential is pretty large.

Which brings me back to ZPlanner? The product I'm describing above will take quite some effort to implement and I'm anxious to get started. But I really want to polish off ZPlanner. I've decided I will. But then the question becomes do I try to sell it still? Or should I just open source the thing? If there really is no market for "yet another project tracking tool", maybe it's better to open source it. ZPlanner had it's genesis due to the failings of XPlanner, another open source project tracking tool. Despite it's failings, however, XPlanner has seen wide usage and even though it seems to be fairly moribund, with the last release almost 3 years ago there are still a ton of people using it...because it's free.

Even if it didn't profit me directly, the thought that Zplanner might replace XPlanner as the open-source, free tool for choice for tracking is pretty cool and might indirectly profit me in other areas. It's hard, given that I've sunk somewhere between 300-400 hours into it at this point, to just give it away. Though I certainly make use of a lot of open source stuff and maybe it's time to give back.

I don't know. I guess we'll see what I decide.