When we formed LShift, we sat with a group of engineers and asked them what the new company would have to be like for them to want to join it. Their first request was that we get rid of a situation every developer is familiar with—mid-project, the Account Handler or Program Manager turns up to tell you that, in a fit of largesse during lunch with the client they have agreed to add new features / swap out the core platform / complete the project a month earlier than planned: bad management.
Our solution involves putting our Lead Developers in front of the clients as much as we can, immersing them in the business drivers of the project, and then (to the extent the situation allows) letting them control the design, planning and delivery of the project. Our mantra is that we “disintermediate middle management to the extent it makes sense to do so”.
In our ideal scenario, the developers manage their projects. This doesn’t magically solve all problems, but at least this way if the developers are working a late shift because of a rash promise they’ve made, they are the authors of their own misfortune, which is easier to bear. It’s a social contract: we give people control, they accept responsibility. People are free to mismanage themselves—free of middle managers doing it for them. As it happens, quite apart from the moral advantage, this turns out to be a good way to run a project efficiently; there are few late nights any more, and our projects tend to come in on time and on budget.
I was reminded of this in a recent presentation selling the joys of Agile. The speaker began by pointing out the decisive weakness of Waterfall; if the original design is wrong, the final product will be wrong to the same extent because there is no way to fix things during development. That truth being established it’s easy to sell Agile on the basis that it allows repeated redefinition of the project (scope, features), so the deliverables evolve as the team learns more about the client’s needs, and as they run tests, experiment with different solutions, have sudden, brilliant thoughts, etc., etc.
So far, so good. But I was suddenly seized by a horrible thought: what if this new-found agility was used, not teleologically to approach the right outcome over the course of a project, but simply to enshrine the right of middle management to change their minds, to provide a methodological license for arbitrary management? At least under a Waterfall regime they had to apologise when they departed from the plan. With Agile they are allowed, in principle, to make as many changes of direction as they like. But what if Agile was used merely as a license to justify keeping the team in the office night after night in a never-ending saga of rapidly accumulating requirements and dizzying changes of direction? And what if the talk of developer ‘agility’ was just a way of softening up developers for a life of methodologically sanctioned pliability? In short, what if Agile turned out to be worse than Waterfall? That, it suddenly seemed to me, was the likely outcome of using Agile tools in the absence of a governance regime to keep projects on target. Agile without governance is ‘headless Agile’, hence ‘acephalic’.(1)
Project vs. feature roll-out
In what follows I’m assuming that we’re talking about a project rather than merely a production line for rolling out bug fixes and features. In the latter case you can use Agile tools like Scrum to deliver code incrementally, and come to no great harm. In talking of a ‘project’ however, I’m thinking of a larger program of work in which an entire solution has to be designed and delivered, (more or less) complete. Such projects have distinct beginning, middle and end phases, all of which must complete successfully if the project is to succeed. In the real world there’s a continuous spectrum stretching from colossal projects through to incremental bug fixing, but in what follows I’m talking about what must happen at the project end of that spectrum for the work to succeed.
The project vector
To understand the necessity of Agile governance, over and above Agile tooling, think of your project as having a trajectory at any time. Ideally, this trajectory is aimed toward the perfect conclusion. You might think of the project as having a vector path, aimed more or less accurately at the correct end state. This end state is conceived Platonically—you might not know what it is exactly, but it definitely exists as an ideal state you want to achieve.
In this way of thinking, a Waterfall project has a single vector, determined at the project outset, which can’t be deviated from. I think of it like the 1970s game show, The Golden Shot, where contestants would give a blindfolded archer instructions (“Left… right… back a little… right a bit… stop! Fire!”) to get his crossbow bolt to cut through a thread. A successful shot severed the thread to unlock the prize. The point is that, once you told the archer to fire, there was nothing you could do to alter course: the bolt either hit the target or it didn’t. Everything hinged on how accurately you aimed to begin with. Very few people succeeded.
In any project, using any delivery methodology, there are such control points at which the direction of the project vector can be modified. These are the key points (the circles below, marked ‘G’) at which project governance is exercised, allowing us to steer the project. In a Waterfall project, as in The Golden Shot, the project vector is set at the beginning, where you have the project’s main control point, and where you set its course (the project vector) once and for all. In a Waterfall project, in principle at east, the only control point is at the outset, when all the requirements gathering, analysis, planning and specification is done. Despite this, it is traditional (and usually necessary) to exercise a second control point at the project’s end, during acceptance, at which changes are accepted in order to bring the things into line with the client’s goals.
As we know, while using the Waterfall approach people soon realised that there is much to be discovered during the course of a project about this desirable end state (unanticipated requirements and client preferences, the suitability of our technology choices, etc.), such that it is unlikely you will have understood the objective correctly at the project outset. If you did understand the project goals, you still couldn’t fully anticipate how best to achieve them. And even if you correctly interpreted the requirements at the start and aimed with complete accuracy, you still won’t hit the sweet spot because it is a moving target—clients also learn during the course of the project and change their expectations and requirements accordingly. If the project vector isn’t properly aligned, you finish the project with a product the client is unhappy with. That’s when you have to go in and ask for extra time and a new budget to adjust the solution. It’s at this point, during ‘acceptance’, that we exercise our second, terminal control point, where the client tells you all the ways in which you’ve fallen short, and you have to plan further operations to put things right.
Getting carried away with our vector notion, you could define the client’s pain as being proportionate to the area of the triangle created at the end of the project, determined by the amount you are off target, and the time and effort it takes to repair matters. The further off beam your original plan was, the more off target the vector, and the more work you have to do at the project’s end to give the client what they need. This quotient of pain and dissatisfaction is represented by the red area below.
An Agile response
How are matters improved by Agile? The benefit of Agile is that, by splitting the project into sprints, we have more opportunities to tweak the project vector, to exercise governance, altering the project’s course. Each sprint offers new control points (during both planning and acceptance), at which we can amend direction. Splitting the project as a whole into a series of such sprints offers us a corresponding number of opportunities to reorientate. Each iteration offers more chances to retune the project. We have much greater granularity of control, and more opportunities to exercise that control.
We have now come some way from our one-off ‘Golden Shot’ scenario. The new situation is more akin to that of someone working a remote control helicopter drone, able to tweak the drone’s direction to get it on target. The wind may blow it off course mid-flight, but we now have the opportunity to steer into the wind for a while to get back on track.
The crucial point is that Agile itself provides us with the ability and opportunity to correct course, it allows us to steer, but it does nothing as such to help us steer correctly. Using Agile methods, our pilot can redirect the drone, but those methods do nothing to help him keep it on course: Agile tools provide a steering mechanism, not a homing device. Consequently, our Agile-empowered project manager is just as likely to be steering off-course rather than toward it. At least, there is nothing in the use of Agile tools as such that makes one outcome more likely than the other.
Everything hinges on the effective use of the control points in each sprint. Used judiciously, they help us to stay on target. But there is nothing in Agile as such to help wire them into the Agile process correctly. All of which means to say that, in and of itself, Agile is just as likely to be a tool for empowering chaotic management as it is to be a mechanism for delivering successful projects. At least in Waterfall if you did a good job of designing the project, and if you stuck to the spirit of the method, you were to some extent insulated from management noise and ‘dreck effekt’(2). But the outcome in a badly governed Agile project is that each iteration takes us further off course. Or at least it doesn’t (‘asymptotically’) converge on the right outcome as it should. The result can be even worse than a badly planned Waterfall project, and with more late nights worked along the way. The pain coefficient at the project’s end can also be greater than with the Waterfall. Here it is that Agile can actually be worse than Waterfall.
The lesson is that Agile methods as such do nothing to keep a project on course—they make course correction possible, but they don’t make it any more likely. Agile enables pliability; it is up to the project management team to exercise that pliability usefully, to create a truly Agile project. The key is to introduce effective project governance. This refers to the detail of the control mechanisms used to direct the project, how they are used to capture and incorporate the necessary information so as to guide the project direction.
A successful project depends on the correct project setup (feasibility and foundation phases), but here I’ll ignore such niceties and talk only of the active development phase of the project. With our iterative approach to development we have the opportunity to capture client (and other) input continuously and use it in each iteration to effectively frame the project, reorienting it to keep it on track with both client needs and technical discoveries made during implementation.
Having said that Agile itself doesn’t provide mechanisms for exercising governance correctly, everything now hinges on how this is going to be achieved. This is where it is necessary to go beyond the use of Agile tools to lean on a wider Agile framework. At LShift we have used the DSDM framework (now rechristened AgilePM) to provide this function. DSDM (‘Dynamic Systems Development Method’) is too rich a system to do it justice here. Its flexibility and the range of its methods and concepts means that it fits a wide range of approaches, from extreme agile development through to projects with a significant Waterfall component.
Here it is enough to note that, among other things, DSDM provides a context and methodology for ensuring that at each control / governance point input is gathered that synthesises project management, business and technical inputs into a coherent whole. In the diagram below, the control point is expanded to emphasise that it is not merely an opportunity to exercise management input, but to coordinate and combine inputs from management, business and technology domains.
Using these AgilePM / DSDM tools we can begin to exercise Agile’s pliability effectively, steering our project nearer the goal with each iteration of the project. Now, each iteration offers not only the chance for project management to be exercised, but more specifically, to ensure that it is exercised by capturing the appropriate, timely information from all the appropriate stakeholders.
That’s all easier said than done, of course. Everything hangs on correctly applying governance tools and methods, and it can take a great deal of experience and a lot of failed experiments before you get that right. But the better you get at it, the more likely that you’ll really be in control of your project, in the sense of consistently steering it toward the optimal result.
1. Acephalic (ey-suh-fal-ik). Zoology. headless; lacking a distinct head.
2. drek: ”filth, trash,” 1922, from Yiddish drek (German: dreck), from Middle High German drec, from Proto-Germanic *threkka (cf. Old English þreax “rubbish,” Old Frisian threkk), perhaps connected to Greek skatos “dung,” Latin stercus “excrement,” from PIE root *(s)ker- “to cut” (see shear (v.)).