One of the best comments I’ve heard about Agile development is that “adopting Agile is itself an agile process”. It sums up very nicely the fact that each team and each project will have their own needs when it comes to finding a process that works for everyone, including the stakeholders. It also means that no matter how much you like a specific methodology, it’s virtually impossible to implement that process on your team following its strict definition.
As a Scrum Master I’ve always been partial to Scrum, but after spending six months trying to make it work on one of our large client projects, I reevaluated and started from the ground up to design a whole new Agile process. We’ll be introducing our new process in a future blog post, but for now I wanted to talk about some of the challenges we faced implementing Scrum.
There was nothing specific that was wrong with Scrum for our project. We just ran into enough minor hiccups that we started to lose some of the benefits that we should have been able to reap by using it. Before starting, I needed to figure out exactly what wasn’t working (and also what was working well) with Scrum so that those issues could be addressed by our new process.
Here are some highlights of our challenges:
-
Missing or ambiguous roles
Scrum works best when a strong Product Owner and a strong Scrum Master maintain a bit of tension between them. The PO needs to demand more and the SM needs to keep things reasonable. The result is an aggressive schedule with a great deal getting done, but development kept to a sustainable pace.
This can break down if your Scrum Master needs to do a lot of the product planning and prioritization (or isn’t involved enough in the planning process). Likewise you can also run into significant problems if you have trouble finding someone who is both empowered and knowledgable enough to be a strong Product Owner.
Another potential issue is that as painful as it may be, priorities sometimes need to be set by a committee. If the client can’t abide by a single person who selects the features for the upcoming iteration, the process needs to be flexible enough to support that. This also means that if the person who is most responsible for setting priorities is unavailable, then one or more surrogates can easily step in.
-
Self-organization doesn’t work for every project
Even when you have competent and self-motivated developers, not all teams work well when attempting to self-organize. For our team, there were a few factors that made it a challenge.
First, we were regularly adding and removing people. As other projects spun up or down, we would have fairly regular shifts in team composition. While we always had the same number of developers, we frequently had new developers coming on who were new to the system. Since our product is highly complex, they simply could not figure out which of the backlog items from the current iteration were good ones to try to work on.
Second, with a team spread out across the world, real-time collaboration is hard. In our case we have developers across the US and China and were working with client contacts in every US time zone as well as Spain and South Africa, making it impractical to rely on instant availability. The consequence is that your team needs to be more autonomous and focused on working independently. In my experience, self-organization works best when the whole development team is collocated and able to work with one another face-to-face.
Last, the client will sometimes need more detailed insight into who is working on what. In traditional scrum, having a backlog story hanging out and unassigned at the beginning of a iteration can make some clients uneasy.
To get around these issues, we found that the project manager for the development team needs to assign tasking to provide focus and avoid issues with non-real-time collaboration. This needs to be done in such a way that the clients are aware of the assignments, but the developers are still empowered with technical and design decisions without needing approval.
-
Rigidly timed iterations can break velocity
There are a number of situations where having a fixed-time iteration doesn’t work. In the web development world, you will often have short (e.g. 1 week) iterations. Many features easily fit into one iteration, but often a large feature needs to be spread across multiple iterations and simply cannot be broken down into concrete backlog items that can be deemed complete during a single iteration. Also, changes in client schedules, shifting personnel, and unexpected things like conferences, holidays, etc. can make iteration timing problematic.
Scrum requires a stable team that has a consistent velocity. That velocity is used to determine how much to tackle in the next iteration. The problem is that a team’s velocity is based on the past performance of that team (i.e. how much they got done during the last few iterations). When you have a team that either changes frequently because of personnel or unstable iteration times and shifting customer requirements, then the velocity can become unstable.
When these factors affect velocity, your most important planning metric becomes meaningless.
-
Better transparency into release schedules
With web development, you often end up in a cycle where major new features come out each iteration and marketing of those features can get lost in the noise. We need a way to say “the next major version of our site is online and it includes features foo, bar, and baz”. The client needs an opportunity to brag about progress and plan ahead for potential downtime.
-
Incorporate a mechanism for QA and bug fix cycles
There needs to be a clean way for the QA team to be fully integrated into the iteration cycle. The acceptance criteria and the planning of regression tests need to be a core part of the process while not holding up the development team. If you have separate QA and development teams then your developers can end up sitting around waiting for QA at the end of a sprint in Scrum.
At the same time, there needs to be a real dedication to continuous code improvement and bug fixing that isn’t dependent on the Product Owner deciding to prioritize those fixes.
While accommodating all of these changes, we still needed to harvest some of the benefits of Scrum and other Agile processes.
Even though we were looking for a process that would allow our clients more insight into who was working on what, we still wanted our developers to be able to focus on the task at hand and not be concerned with shifting priorities or popup crises. The client needs to remain hands-off and the process needs to be set up to make that comfortable for both the client and the development team.
Obviously one of the biggest advantages with Agile development is that you are able to respond to changing requirements (the idea of dropping “ready, aim, fire” in favor of “ready, fire, aim, aim, aim, aim, …”). The new process needs to support that as well or better than the process it is replacing.
These are some of the most important issues that we needed to incorporate into our new process. In the next post in this series, we’ll actually describe our new process and how it works. After that, we’ll include some tips for implementing it on your project and some of the challenges that might come up.
