Agile software development was pitched to the business as the panacea to save costs and deliver desired functionality in an incremental way, thereby achieving a faster time to market. The pundits claimed that it would solve the problems of the waterfall model - where too much time was spent on documenting requirements and then building a product which ended up being costly and possibly outdated and which in the worst case didn't end up solving the problem. The waterfall process also didn't have the flexibility of co-creating with customers and lacked the quick feedback loop which could go back to the development team into refining the product.
The agile framework flipped the solicitation of the requirements from a feature driven approach to that of a problem solving and discovery approach with the end user in mind, through the creation of real world use cases which were translated into numerous user stories.
In the traditional approach a business requirements may have been crafted as :
Waterfall Requirement : Display the stock ticker at the bottom of the website showing all the stock prices from Nasdaq.
Now compare the same requirement as a user story in the agile framework. The product owner would have provided the user story as:
Agile User Story : As a investor I want to be able to see the live stock prices from Nasdaq, so that I can the place a trade with the broker immediately and not miss the trading opportunities.
The why?
It was deemed to be a efficient way to deliver usable working software which could be easily commercialised through multiple release, where user feedback could be prioritised incrementally. There is certainly some truth to this, but to the vast majority of product managers who have to work with fixed budgets, time and scope the agile approach is a practical nuisance to manage as the demands are time consuming and the process puts the product manager through loops of repetitive prioritisation, backlog grooming, justifications and communication. In some projects I have even witnessed hostility by engineers when they receive complex user stories or when there is vagueness.
Many product managers were pushed in to the agile approach and not knowing what to expect they had to come to grips with how to work through this unfamiliar operating model which had so much uncertainty, personal insecurity and execution promise. As management had already signed up for Agile, the product managers had to quickly come up to speed and start working with technology to participate in the sprints to make it happen. Training and coaching classes were setup, it was like teaching a dog a bunch of new tricks, but with the same goals in mind. As if the product manager didn't have enough work to do, they now were put in a precarious situation of having to potentially double hat as the product owner as well - and devote considerable time to the various ceremonies.
The elephant in the room
Crudely put the problem which IT faced in delivering working software on time and within budget has been passed over to the business who now have to go down an agile approach where the product manager now needs to deal with the prioritization problem by constantly having to prioritize the backlog and drop user stories and make various other concessions so that the sprint can deliver working code! (In waterfall this effort was less regular, typically at the start of the project and at specific project milestones)
This somewhat unpredictable nature of what is being delivered is frustrating for the business to manage as there are numerous stakeholders who do not work on a piece meal approach. For example the product risk management framework assesses a product as a whole, similarly the communications, marketing and sales machines need clarity and certainty. The additional burden of managing the gap with these stakeholders now falls on the product manager, over the longer term the entire approach of managing risk and product marketing may itself need to be tweaked.
To understand this frustration a bit better lets focus on who the product manager is and who a product owner is.
The product manager (PM) comes from the business and is responsible for all aspects of the product. Some like to call themselves as the CEO of the product. The product owner (PO) in most cases comes from the technology team.
The PO works with the PM to define the product vision, prioritise the product backlog and deliver commercially valuable working software which can be deployed to customers. The product owner would also engage the business throughout for alignment and decision making. In an ideal world the product manager should be the product owner as well for better quality of decision making and feature prioritisation.
In reality the roles are done by different individuals and this is because of the intense engagement required of the product owner role which the business product manager cannot do justice to, due to the resource commitment which is required. Product managers cant break away for lengthy sprint planning sessions and join daily stand-ups as the remit of their role doesn't give them this kind of time flexibility. Almost 80% of a product managers time is focused on customer, business and product management & development matters, so the PM can only allocated anywhere between 5%to 15% of their time to the product owner role.
I particularly don't care too much about Agile having been a product owner for many years now, there have been many days where I just wanted to bang my head against the wall. I wasn't proud of some of the early releases as I had to ship ugly looking user experiences even though the functionality worked just fine. Funny how agile is so focused on delivery and at the start of the agile movement there is very little care to deliver a delightful user experience. To me this goes hand in hand, a product owner should not be forced to choose function over UX. However the nature of the beast is such that hard prioritization decisions need to be made regularly, a not so ideal situation.
The delivery of software is the role of technology, how they choose to deliver it just needs to meet the business objectives of rapid deployment, flexibility and cost effective.
I have seen success stories on agile and some not so good outcomes as well which has led me to believe that Agile is not for everyone, or every project or for every organisation. I am generally supportive of agile but I prefer to take a measured approach based on the nuances of each and every project and more importantly the capability and maturity of the individuals involved in the project.
Agile projects can give desirable outcomes if all the technology development is done in-house without any external vendor resourcing. Working through an agile project with a untested vendor can be risky as you do not know what the velocity of delivery is and the quality of their work (bugs, defects), there is also a tendency to include technical debt within the sprint there by reducing the amount of feature development which is delivered. When there are multiple agile teams on the same project there is bound to be confusion.
Jumping onto the bandwagon
Implementing agile has a steep learning curve. Its not an intuitive process for the project team unlike the waterfall approach where everything is clearly documented and then deliberated on before the commencement of the building process.
Agile is typically an ongoing orchestration of many ongoing activities where the conductor - in this case the scrum master has the task of making the entire team tune in to a set of objectives to deliver the accepted user stories. Every musician is required to know their notes give off their best and stay in synch with the rest of the orchestra, this to me is also what every member of the agile project team must do - individually everyone is accountable to take their own lead and then co-ordinate their execution amongst the team to deliver what is expected. There is very little margin for error / slippage, just like the Cello player if you think your out of tune you have to recalibrate very quickly.
Agile requires the participants to step up in maturity and to be responsible for their pace of work. Once the user stories are accepted everyone needs to do their part to deliver the outcome. Issues and obstacles need to be raised in the stand ups, and actions need to be taken promptly. Collaboration needs to be the mantra to resolve problems and given there are fixed timelines the most important aspect the team needs to adopt is mutual respect and trust on the decisions which are made. This will lead to higher quality outputs, efficiency and better product design. The entire team should be open and connected and not shy away from difficult conversations. There will be hard talk and that should be expected, but with the right spirit of respect and trust the outcomes will be better.
The politics at the top of the house
Somehow the IT executives have convinced the business that Agile is the solution for everything. The C-suite as we all know loves buzz words and thinks that they have evolved to a new level of intelligence and so the agile movement begins to take shape and the thinking at the top of the house is that all projects should be agile.
What is not foreseen is the organizational impact of adopting agile as a way of delivering product and projects. Some of the key impact areas are:
The business needs to revisit their governance and product management processes to support the agile software development approach.
Management needs to accept the uncertainty of what may not be delivered in the end and get comfortable with the process of constant prioritisation. It has to be accepted that somethings will not make the cut.
Focused questioning on the need for a user story in line with the problems which are being solved. There is no space for bells and whistles and nice to have functionality.
There is a cost for everything and nothing can be snuck in under the carpet as each requirements has its own set of user stories and acceptance criteria and goes through a formal acceptance framework.
The result of poor product design now is now more on the product manager / owner. The business needs to do its homework before the sprint planning sessions and make sure they know exactly what need to be built.
Agile requires the product owner to have the decision making authority on what goes in and that often decisions are made without consulting the wider business stakeholder group. Thats the agile life of a PO, they face everyone off and have to defend and justify their decisions regularly.
The product owner makes or breaks agile. Great product owners are hard to come by and they need to be recognised for the work they do and be financially rewarded for the value they bring in to the execution. Take a product owner away and you can loose momentum very quickly.
Customers don't care about whether you call your release an MVP or by some other name and nor do they care about the whole agile movement - they just want products to work so that they can go about doing their jobs.
Agile leaves us with mixed feelings and like any execution tool its got its benefits when you can use it in a controlled environment. To harness its benefits there should be an open discussions for each project to determine the suitability of the approach for that specific project.
Many organisations have adopted a a WAGILE approach which is a hybrid of waterfall and agile which is acceptable way to address the shortcoming and risks which agile exposes the business to. Having been an agile product owner for a decade now I have fond memories of having worked with some great people, the journey had its back breaking moments, frustrations and disappointments.
Today i deliberate the merit of agile for a particular project before committing to it and I have also got comfortable with working in the dark and dealing with uncertainty. Do share you best and most fragile moments with us and lets keep the conversations and the learnings going by sharing our experiences with the community.