Goals and the limits of self-organization

Thought-provoking post by Jurgen Appelo on the teleology of software projects (post here, check the perceptive comments too).  More properly, he points out that projects do not have a goal in and of themselves.  In his words, they don’t have intrinsic goals (other than self-preservation).

For me, this insight points to the limits of self-organization in initiatives.   IMO, without some degree of design — or extrinsic goals — a self-organized system (or pieces of that system) can get off the rails.  There is a tremendous amount of power in emergent-friendly systems – that’s what social media is all about.   For example, what emerges from Wikipedia is clearly emergent, but it has a explicit goal and with an extrinsic design model:

Wikipedia’s purpose is to act as an encyclopedia, a comprehensive written compendium that contains information on all branches of knowledge. 

Finding the proper balance between design and emergence is a fascinating topic.  In fact, my take is that this topic is the subtext of many of the arguments among methodology adherents — waterfall vs. agile. 

I’m always a bit leery of purist arguments.  In fact, I have a syncretist’s instinct to “square the circle”.  Perhaps what I’m looking for is something like Deng Xiaoping’s modifications to traditional Marxist dogma…   How about “Waterfall with Agile Characteristics”?

Methodology Scaling/Selection Outline

Or maybe I should say “Methodology Scaling/Selection Graphic”… Glen Alleman’s recent post on PM and Agile (here) reminded me to post this picture (click to enlarge) of a slide we use when discussing which methodologies to use for which project.  Note that we do not directly oppose “Agile” vs. “Waterfall” in our continuum.  Sure, SAPScrum is pretty pure agile, but that doesn’t mean that ASAP doesn’t contain agile concepts (e.g., emphasis on iteration short cycles during the Realization phase). 

Also, we’re emphasizing that initiative leaders must be comfortable combining philosophies when implementing solutions, which is why we’re driving program management knowledge out to our general PM community (as implied in the bottom box).

Squaring up the PMBOK Guide and Agile

Glen Alleman shoots down some of the common misconceptions about the relationship between the PMBOK Guide and agile principles (here).  As regular readers know, I’m sympathetic to agile approaches, I’ve found them very powerful in the right project context, and I push to incorporate agile principles further into our own content. 

Of course, if I commented on Glen’s post itself, I’d be writing “ditto” or “I agree” a lot, so I’m just going to add on a bit:

• While many agile advocates over-read the PMBOK Guide, the guide did imply a waterfall approach, especially in earlier editions (Chapter 2).  I can understand the confusion or over-reading.
• The pending 4th edition does have a more sophisticated treatment of project life cycles and explicitly addresses iterative relationships among project phases.  It is hardly sufficient, but it is heading the right direction.
• PMI itself is quite eager to engage agile advocates for future editions (or extensions) of the PMBOK Guide.  Several Global Corporate Council members were approached about how and whom to engage in the agile (or agile-savvy) community.
• My take is that creating an Agile extension to the PMBOK Guide would be the best initial approach.  If nothing else, integrating agile-oriented PMs into the standard-setting process should be easier in an extension project.

Planning and flexibility aren’t mutually exclusive

Excellent post by Craig (here) which gets to the way we need to be thinking about our projects and programs these days.  Our methodological ideologies can get in the way of using the appropriate approach for what we’re building, implementing, or upgrading. 

What is presented as an either/or choice, isn’t.  The answer is “both”.  A plan is not “that which is carved in stone just after the project is chartered” (and shall not be deviated from save for acts of one God as one understands God).  And agile isn’t “doing whatever the heck we think should be done as quickly as possible” (and the product owner be damned). 

The post comes with some great comments as well. 

Finally, nice graph.  I would go with “rigid” vs. “stubborn”.

How much complexity theory can we apply in IT?

I’m fascinated by complexity theory and attempts to apply its insights to the software business.  If you’re interested in those topics you could do worse that to add two bloggers — Jurgen and Bas — to your newsreader (don’t forget about Crossderry). 

Both touch on complexity regularly (Jurgen’s latest here, Bas’s latest here) and they’re clearly big fans of the theory and its implications.  I agree there’s much that’s applicable, especially the concepts of iteration and feedback, which can even be used in “waterfall” approaches (here and here).  My academic background makes me especially sympathetic to the limits of central planning (start here re: Hayek).

That said, I’m not sure we can rely on self-organization for everything.  The most effective models of complex adaptive systems are derived from simple rules that generate complex phenomena.  This approach is mimicked effectively in agile, iterative, and other rapid development techniques (list of SW methodologies here).  Simple feature lists, regular interactions with stakeholders, short cycles, many versions of usable work product, etc. can generate feature-rich and useful applications.

However, the scalability and stability of these applications is often problematic.  IMO, this result is to be expected given the evolution of complexity among living things.  We like to point to complex creatures and structures — e.g., human brains — to support applications of complexity theory. 

But do we remember that most life is still very simple (about half of the biomass is microscopic)?  Also, aren’t complex creatures the ones that have had the spectacular denouements over the eons?  Betting on self-organization isn’t always a winning bet.  As I said, I instinctively like leveraging complexity concepts, but we must remember that they cut both ways.