The Setup
Consider the following scenario:
Waterfall
A not-for-profit company wants to keep track of their contacts in a centralized location. They want to keep a record of donations from each contact, as well as a history of calls from each contact. They can not afford to purchase specialized software but have a consultant who has volunteered his services.Within the scenario is the nugget of an idea (a “contact database”) and a vague notion of the requirements (“record donations and calls”). There are also presumably many unstated requirements (e.g. viewing this data in reports). How can they get from this abstract concept to a working and tested application? Broadly, there are two mainstream methodologies of software development: Waterfall and Agile.
Waterfall
In the Waterfall methodology, the Consultant would sit down with representatives from the company (the “Users”) and build out a complete requirements matrix, perhaps even including User Interface wire diagrams and workflows. He would then proceed to plan out the complete development and testing effort, estimating various work units and arriving at a reasonable delivery date.
The problems with this methodology are numerous and documented. There are two fundamental flaws: 1) it pre-supposes that Users actually know everything they might possibly want up-front, and 2) it pre-supposes that the Consultant has an accurate estimating model. Neither of these conditions is likely to be true: Users will always have new ideas (it is hard to staunch the flow of creative juices once they get flowing), and Consultants will always mis-estimate (either over or under) the effort required to deliver.
Even if a Consultant had an exact estimating model, User Change Requests would quickly destroy any semblance of a plan. Further, if one were to assume a perfect estimate and zero tolerance for Change Requests, once the application were delivered and in use there would undoubtedly be a backlog of desired enhancements and changes to the application, many of which might be very desirable/valuable.
Agile
The problems with this methodology are numerous and documented. There are two fundamental flaws: 1) it pre-supposes that Users actually know everything they might possibly want up-front, and 2) it pre-supposes that the Consultant has an accurate estimating model. Neither of these conditions is likely to be true: Users will always have new ideas (it is hard to staunch the flow of creative juices once they get flowing), and Consultants will always mis-estimate (either over or under) the effort required to deliver.
Even if a Consultant had an exact estimating model, User Change Requests would quickly destroy any semblance of a plan. Further, if one were to assume a perfect estimate and zero tolerance for Change Requests, once the application were delivered and in use there would undoubtedly be a backlog of desired enhancements and changes to the application, many of which might be very desirable/valuable.
Agile
The Agile methodology works differently; it is a Lean methodology applied to software. In Lean, the expenditure of effort on anything other than adding value is wasted effort and should be stopped as quickly as possible.
Rather than pretend that Users and Consultants are perfect, it embraces the fact that they are not. Instead, Agile focuses on the delivery of maximum value as quickly as possible. Furthermore, what a user might perceive as “high value” one day could change the next morning when they’re taking a shower and discovered a completely new way of working.
The problem with Agile is that deadlines lose meaning: when does “done” ever happen? Large enterprises in particular struggle with this, because there are many moving parts: System A has new functionality that requires System B to be updated, but if both these changes aren’t in place and working by Date X, N millions of Dollars/Euros/Pounds/Seashells will be LOST!
Agile has many problems, they are just not as well-known because it is the "flavor of the day" and, quite frankly, isn't as mature as Waterfall. There is certainly ample documentation demonstrating that stand-alone applications are developed more efficiently using Agile than using Waterfall, but the enterprise world is littered with the abandoned corpses of "new development methodologies" gone wrong.
Why Stick With Waterfall?
For lack of a proven-to-be-better delivery model, enterprise shops end up choosing Waterfall over Agile: Project and Program Managers build in contingency to increase the probability of deadlines being met, giving the executive team some comfort in the knowledge that “we have a deadline, milestones, and estimates, so we will be able to keep our millions Seashells.”
However, this is something of a fallacy: contingency is almost always eaten. “Work expands to the time allotted for the work to be completed.” It is why hour-long meetings always take an hour, even if the meeting topics could be completed in 20 minutes.
Rather than pretend that Users and Consultants are perfect, it embraces the fact that they are not. Instead, Agile focuses on the delivery of maximum value as quickly as possible. Furthermore, what a user might perceive as “high value” one day could change the next morning when they’re taking a shower and discovered a completely new way of working.
The problem with Agile is that deadlines lose meaning: when does “done” ever happen? Large enterprises in particular struggle with this, because there are many moving parts: System A has new functionality that requires System B to be updated, but if both these changes aren’t in place and working by Date X, N millions of Dollars/Euros/Pounds/Seashells will be LOST!
Agile has many problems, they are just not as well-known because it is the "flavor of the day" and, quite frankly, isn't as mature as Waterfall. There is certainly ample documentation demonstrating that stand-alone applications are developed more efficiently using Agile than using Waterfall, but the enterprise world is littered with the abandoned corpses of "new development methodologies" gone wrong.
Why Stick With Waterfall?
For lack of a proven-to-be-better delivery model, enterprise shops end up choosing Waterfall over Agile: Project and Program Managers build in contingency to increase the probability of deadlines being met, giving the executive team some comfort in the knowledge that “we have a deadline, milestones, and estimates, so we will be able to keep our millions Seashells.”
However, this is something of a fallacy: contingency is almost always eaten. “Work expands to the time allotted for the work to be completed.” It is why hour-long meetings always take an hour, even if the meeting topics could be completed in 20 minutes.
Contingency is waste: it is a block of time allocated not for the purpose of creating value but rather to buffer deadlines. (In Lean manufacturing, buffers are waste.) Furthermore, because Waterfall spends so much time up-front focusing on capturing a "complete" set of requirements, it "back-loads" the creation of actual value. More times than not, the contingency gets eaten before the value is delivered ... leading to a considered abbreviation of testing (there's a deadline to meet, after all). Finally, "newly discovered" value opportunities in the middle of a Waterfall project are either highly disruptive ("Change Requests") or deferred.
The Premise
If there is one thing that I've learned in 16 years of writing enterprise software, it is this: Waterfall wastes lots of money and rare is the project that delivers exactly the list of requirements it set out to at the start. I've also yet to see a working alternative to Waterfall in an enterprise shop.
So, I intend to undertake a journey of sorts: I am the volunteer consultant in The Setup.
A partial list of my qualifications are this:
- I have been writing software in various forms for more than 25 years, starting with BASIC on a Commodore 64, dabbling with 8088 assembler, Turbo Pascal 1.0, working up to the present day where I am quite accomplished in Oracle PL/SQL.
- That said, I am not a "programmer"in the sense that I a) took a single Computer Science class in college, b) spend a large portion of my working day thinking about architecture, design, requirements, and how to deliver working enterprise software. I do write programs, but really it is of a nature of building database-oriented frameworks that allow our Java developers to do their job more effectively.
- I am not an expert in Agile. My current employer (Quantum Retail) uses Agile (or a a bastardized version thereof...we work with retailers who would much rather we behave in a Waterfall manner). I've been a victim...er...developer of an enterprise XP (eXtreme Programming) regime. BUT, I most definitely am a student/fan of Lean.
- I know Java reasonably well...I started out on v1.0.2 (the first version) and have dabbled with it ever since. However, were you to ask any of my colleagues about my style of programming, they might describe it as "arcane."
- Spring? Hibernate? AJAX? Heard of them, dabbled with them, would do very well if I could build something with these technologies from scratch. I understand the problem these are trying to solve, though I would suggest that they sometimes create too much complexity in their implementations while trying to cater for every possibility in the quest to simplify development.
- .NET? I'd probably be a fan if I worked with it more. My main qualm with the world of Java is that there are too many splinters; Microsoft provides a very solid, integrated, and complete development platform. If you're happy to drink their flavor of kool-aid you don't need to look elsewhere.
- User Interface design and development? Apart from "Hello, GUI World" and a few other very simple learning exercises, I know very little about how to do this.
So, my language of choice? I've been reading about Scala and a related development framework called Lift. After looking into Ruby on Rails and reminded myself of the loose-type nightmare that is Javascript, I've decided that Rails is not the framework for me. My instinct is that type specificity is easier to test.
What do I hope to learn from this? Well, at this point I'd like to learn Scala and Lift. I'd also like to explore Agile development from a Consultant/Developer perspective (at the coalface, as it were), without all of the bother and pressure that comes from having a customer who is actually going to pay you money for a job.
No comments:
Post a Comment