It’s very likely that the development groups in your company follow some form of agile process, whether it be Scrum, XP, RUP, DSDM, or one of a myriad other acronym-laden approaches. Why are they doing this? Because iterative development has been proven to be more effective and more likely to produce useful products in a timely manner—particularly compared to the pitiful 14 percent of waterfall-based software projects that are completely successful.
I’m not going to spend any time defining or justifying agile or iterative development. There are dozens of good articles and books on the topic. Suffice it to say that if even the US Department of Defense moved on from waterfall twenty years ago, then the ship can safely be said to have sailed.
The elephant in the room, however, is that most agile processes are based on the idea that your entire organization is also agile, or at least able to work within the constraints of an agile development process. Many agile books talk about converting your organization, usually over a number of years, and they give examples of exemplary organizations that have entirely embraced agile and the fantastic results they’ve achieved.
I’m going to hazard a guess, based on the law of averages, and say that your organization is not one of those. Like most things in this world, reality probably falls somewhere in the middle. Your key “development” groups (development, documentation, QA) probably work in an agile manner, but while management wants to be agile—making and saving money with successful projects—it has to balance a lot of competing interests.
In this article, I’m going to talk about some of the most common problems encountered by agile development groups in waterfall organizations, and some approaches that can help bridge the gap between them. Over time, these methods can help your organization become somewhat more agile, while also helping you work within real-world constraints. These are techniques I’ve either used myself or seen used successfully, although some adaption and allowances may need to be made for your particular circumstances. You may need to be (ahem) agile.
The Road Map Conundrum
If your company develops commercial software, this might sound familiar: the sales team wants a detailed road map of what’s coming and when, possibly far in advance of the start of your project. It’s not a completely unreasonable request. If a product doesn’t have the features customers want, they may look elsewhere, and if there’s no projected release date, customers won’t believe what’s coming is real (whether they believe those dates is another matter entirely).
The same scenario plays out with internal development. If your constituency doesn’t know what you will deliver and when, they might consider outsourcing development or canceling the project altogether.
In most iterative processes, you’ll start with a vision statement and a high-level list of requirements. You’ll probably also have a general idea of how to approach the project and the tasks that will make up the first couple of iterations. Is this enough information to start making commitments? Absolutely not. The problem is that you’re fighting the Cone of Uncertainty, which basically shows that the earlier you are in a project, the less able you are to predict effort.
Still, as a good corporate citizen, you want to be helpful. But there are only two realistic ways (short of outright fiction) that you can provide a detailed road map, rich in details and dates:
- You can conduct a thorough waterfall-based requirements audit, then do analysis and provide estimates. Still, assuming you can even get enough time to go through that part of the process, 88 percent of the time you’ll end up delivering the wrong product—late!
- You can do some rough upfront analysis and put together some general ideas of what you think you can deliver and when. You can explain that the information is rough, not final. (If you’re considering this, then remember the immortal words of Admiral Ackbar: “It’s a trap!” People will remember the features and the date, but they’ll forget that all of that information was preliminary.)
On the other hand, you usually can’t just refuse to provide any details (and even if you can, you shouldn’t). One of the most useful things you can do is refer to the resource triangle: time, scope, and resources. You can set two of the three, but not all three. Most businesspeople will be reasonable, so long as you can:
- Explain why you can’t provide details up front. This can include explaining some of the major advantages of the agile approach, such as earlier feature demos and customer validation. It often helps to provide a brief explanation of the agile approach.
- Provide enough information to satisfy the actual need (vs. the need for full information). For example, your marketing team probably just needs to know about the big features and when to expect them, while the support center may only need to know when to schedule training.
- Commit to regular updates as the project continues.
It’s a common misconception that agile development means no planning. In fact, agile projects require a lot of planning, but it’s done at a different time and in different ways. By the time a project starts, you should have a good idea of the major stories the project is designed to address, and a good sense of the priorities. Of course, these things may (will) change—particularly at the detail level—but the goals and categories generally won’t.
Most agile processes are date-driven, not functionality-driven. Aside from being good from a development perspective, this works well in organizations that have to schedule rollouts, write marketing materials, develop training, etc., possibly far in advance.
For a date-driven release, the initial road map should contain:
- The target release period (e.g., Q1/Q2 2014). This period should take into account postdevelopment activities, which can cause some issues since they’re often beyond the control of the development group.
- The categories of functionality that will be addressed.
- Priorities and/or confidence levels.
The priorities and confidence levels are a critical component of a successful road map. Most people understand priorities, so they’ll understand that the priority 1 stuff will be addressed first. Confidence levels are another great way of letting your audience know that some items are more likely to be delivered than others.
To the right is an example of a road map from the beginning of a project. At this point, the road map doesn’t have much information; it won’t really start to become useful until it’s gone through a few iterations.
If you were just gathering requirements, you wouldn’t really have any information at this point—although you might still be making promises. With the iterative approach, you’re communicating real information about where the project is and what’s likely to be delivered.
You can always include more categories, and you can decide on the minimum percentage threshold that allows an item to be included (it should be fairly high, although there are always some “must-have” items that can bubble to the top).
Note that an external-facing road map might need to hide probabilities. In that case, you should make sure that only very high-probability items are included. For internal communication, though, even to other departments, the extra information is usually appreciated since it allows for better planning. You will eventually reach a point where there’s enough detail to make it worth making a road map of this nature public.
The key to this whole approach is balance. It’s about making trustworthy estimates that give stakeholders a feel for progress and enable them to make rational decisions. Of course, in a fully agile environment, stakeholders aren’t just seeing high-level descriptions of what’s in development; they’re actively involved in refining the details. This leads us to the second potential problem.
Where’s the Customer Representative?
One of the really powerful aspects of agile development is the ability to get feedback from the customer far enough in advance to make a difference. Ideally, this is done by having either the customer or a customer representative, such as a product manager, involved in the day-to-day decisions made by developers.
The definition of “customer” can vary greatly depending on the type of project, but, in general, the customer is the person(s) for whom the product is being built. No one has a better idea of whether a solution is good than the person who will use it, and if you’re in a situation where you can have customer representation available continuously, you have the best chance of delivering the right solution.
In the real world, though, it’s much less common for a customer representative to actually be available continuously. For internal development, you’re usually building something for someone who already has a job, and while that person is invested in your success, he’s probably quite busy with his own responsibilities.
Even with commercial software, where there are product managers dedicated to representing the company, the average product manager also has to work with marketing, sales, and analysts, not to mention working directly with customers, doing research, and making sure she truly represents the customer. That’s not to say the customer representative won’t be available sometimes, but sometimes isn’t always.
One of the major differences between agile and waterfall development is that agile doesn’t rely on a massive tome of requirements. But that doesn’t mean that there shouldn’t be any documentation. For any project, there should be a clear top-level description of what the project is supposed to accomplish, and a list of ten to twenty important goals/requirements/points for that project. This doesn’t have to be a lengthy document—possibly less than a page—and it doesn’t necessarily have to be written by the customer representative. However, it’s something that every team member should be familiar with, and something that, if possible, should have been discussed with and agreed to by the customer representative and the team at the beginning of the project.
This document should be reviewed and updated at the beginning of every iteration. It’s remarkable how often projects (agile or otherwise) can wander in the wrong direction because the team members get involved in secondary issues and forget the actual goals of the project. Of course, in an agile world, the priorities will change as the project progresses, but if you do something as simple as keep your priority list up to date, and use it as a guide when choosing your stories, you’ll be amazed at how much of a difference it will make.
With any luck, even if you don’t have continuous access to the customer representative, he will at least be present at the iteration kickoff. By going through the exercise of discussing the priority list, you’ll be able to determine whether features/activities/tasks have become more or less important. It may sound strange, but it’s extremely common for customer representatives to be so taken with whatever they’ve just seen that they forget about key items. Of course, if a “key item” is no longer important, or has become less important, you can always drop it down on the list or remove it altogether.
The next recommendation comes straight out of just about every book on agile: customer demos. If you can’t have continuous presence, then you at least need to ensure that you’re showing your work to your customer representatives as often as possible. There really should be a demo scheduled for ongoing work at the end of each iteration, and if you can do smaller demos more frequently, that’s even better. Smaller demos work especially well when different team members are working on different areas; that way it’s easier for the customer representative to provide detailed feedback in a particular area. You can even go as far as delivering a minimum viable product (MVP) that various customers can actually play with, even though it only has a subset of target functionality.
One important side issue is that a single customer representative may not actually be a good customer representative. A very common situation, particularly with internal development, is that the customer representative is the manager of the group that will use your tool. While the manager will be able to verify that the features he needs are covered, he may not be able to speak well for other end users. For this reason, you should invite members from as many potential constituencies as you can to attend demos.
This final recommendation applies whether you have solid customer representation or not. Developers tend to be smart, quick learners. In many situations, there’s no reason why developers can’t learn enough about a problem domain to be able to make intelligent business decisions by themselves. In fact, when it can be done, this can have a huge impact on a project. Developers will often think of all sorts of different ways to build/improve tools that customer representatives might not think about or won’t ask for because they might think is too complicated.
Some common approaches for getting a developer up to speed on a problem domain include having the developer tag along with an end user, attend business training, or even just spend some time meeting with and listening to customers. The developer’s judgment isn’t a replacement for customer representation, but it can help fill in the gaps, and it can provide better outcomes.
In an ideal world, the developers will constantly be involved with the customers and the business, although adding this to a developer’s plate may add more to an already full (or overfull) plate.
Fighting Fires
Imagine you’re a contractor. You have been asked to build a new room on a house, but when you arrive the house is literally burning down. Suppose you then help the homeowner put out the fire, rescue the cats, etc. As you stand in the driveway, staring at the scorch marks on your clothes, how would you feel if the homeowner asked for a status update on the new room?
This is (hopefully) a slight exaggeration, but it’s something many developers can probably relate to. Maintenance/firefighting can account for 40–80 percent of the work of an average developer. Some organizations do better, but even so, most teams spend a significant chunk of time dealing with existing customers and issues.
Few things can destroy an agile project faster than dealing with unplanned issues. However, given the numbers above, there’s no excuse for not planning for the unplanned.
There are a few approaches that can be taken, and a mix of them is often desirable, but it really comes down to how far in advance you know about things and how important those things really are.
You probably have a system for tracking issues, responding to end users, etc. For a significant number of the requests that come in, it’s possible to simply identify the things that need to be addressed for your next iterations(s) and create stories for those items. From that perspective, you’re entirely in line with your agile process. Once you have stories, those stories should be prioritized along with everything else.
However, there will some more important issues that come up: bugs in existing software, a request for training, a troubleshooting request, etc. These issues are the ones where having a strong leader can really help. A significant number of these issues are probably not really things that have to be dealt with that day/week/month. If you can start every conversation about an issue by asking if it’s something that has to be done right away or if it can be scheduled for the next iteration, you’ll find that, grumbling aside, the delay is acceptable. Once again, you’re back to a reasonable agile process—so long as the number of new items doesn’t exceed the number of things you can accomplish in the next iteration!
But, except for brand-new developers working on brand-new projects, you’re almost guaranteed to run up against issues that need to be handled immediately, whether you think they’re priorities or not. These issues might be related your products, or it might simply be the fifteenth request to attend a meeting to talk about an issue. If you don’t have dedicated resources for dealing with these types of issues—a maintenance team for fixing bugs, a manager who can be sacrificed to the meeting gods, etc.—then there’s still a reasonably agile approach you can take: simply assign story points to each potentially affected team member. The number of points that are needed will vary based on team and team member, but your velocity measurements will help you refine the number over time.
The other really good thing about this approach is that it’s extremely measurable. You can easily show that you spent 25 percent of your time on maintenance and adjust your future scope accordingly.
The Agile Manifesto
In 2001, The Agile Alliance issued the “Manifesto for Agile Software Development.” One of its major assertions can be paraphrased thusly: While there is value in following a plan, there’s greater value in responding to change. It is often amusing, then, to see “agile” projects slavishly trying to follow a particular agile process. Process and planning are important, but reality has a nasty way of reasserting itself. If you have a process that requires your entire organization to change quickly to be successful, the odds are not in your favor.
Three additional assertions relate to the issues we’ve addressed in this article: communicating without overcommitting, working around a lack of on-hand customer expertise, and firefighting. I have seen variations of these problems in numerous companies, whether they profess to be agile or not. There are many other problems you’ll face when dealing with organizations that have a waterfall mindset. The trick is finding a solution that provides what the organization needs (which may be less than it wants) without completely breaking your agile processes.
It takes a long time for organizations to change, and it’s okay to let a little waterfall into your agile process. This will increase your odds of success, and, over time, it’ll help you convince the rest of your organization that there may be something to that whole agile thing!
Cherwell Software cofounder Arlen Feldman is a renowned software developer in the help desk, service desk, and customer relationship management (CRM) software industry. With nearly thirty years of experience, Arlen spearheaded many revolutionary software advances, most notably Codeless Business Application Technology (CBAT) and Platform as a Service (PaaS). Arlen serves on the Microsoft C# Customer Council, a small group of industry experts who advise Microsoft on the direction of .NET, Visual Studio, and the C# programming language. He also authored two books,
ADO.NET Programming (2002) and WPF in Action with Visual Studio 2008 (2008). Previously, Arlen was the chief architect at FrontRange Solutions.