Order Out of Chaos

In the early 1990s my manager gave me a copy of an article by Margarate Wheatly on "Chaos Theory of Management". It resonated with what I was observing in the very large team of teams I was managing.  I then started reading everything I could on Chaos, Chaos Theory, Complexity, and Complex Adaptive Systems.

I eventually landed on Mitchell Waldrop's book on "Complexity: The Emerging Order at the Edge of Chaos". Bang! Paradigm shift. I woke up in the middle of the night and drew up a diagram that 28+ years later still holds true, at least in my own mind.

"All models are wrong, but some are useful" - George E. P. Box. This basic model has proven useful to me over the course of my career.

Basic Model

Key components of the model include Chaos, The Edge of Chaos, and Order. And in order to generate order out of chaos we need an Attractor as shown in Figure 1.

Figure 1 Basic Model

Chaos - Most everything we observe in the world is nonlinear and chaotic.

Edge Of Chaos - the jagged boundary between Chaos and Order. The Edge of Chaos is the boundary in which innovation, risks, creativity, and decisions occur. Core values and guiding principles are major factors of what patterns will emerge.

Order - Some level of predictability emerges. Processes & patterns help to create order.

Attractor - A goal that enables order to emerge out of Chaos.

An individual or small team of individuals who are trying to achieve a goal (the Attractor) will invent patterns or practices to get there. The patterns that emerge as a result of achieving the goal are frequently the best ones.

Many core tenets of Scrum are based on chaos, complexity and complex adaptive systems theory. During Sprint Planning we let chaos in. During the sprint we close chaos out and stay heads down to hit the sprint goal. At the end of the sprint, during Sprint Review and Sprint Retrospective, we let chaos back in.

Lather, rinse & repeat.

Figure 2 Ecosystem

Ecosystem - includes culture, environment, people, processes, practices, tools, customers, users, feedback mechanisms, performance management, etc.

Sometimes the ecosystem is a byproduct of a team inventing practices to achieve a goal (like in a startup). Ecosystems may also already exist (as in a large company). Ecosystems can also over time be shaped. Along with vision and purpose, core values and guiding principles are major drivers of the ecosystem that will emerge.

When leading a new team or working with a new team of teams, I always like to formulate a compelling vision (attractor), purpose (why are we here), values and goals up front. 

Increasing Complexity

Over time there is a strong desire to shut chaos out. Complexity is increased and the focus changes inward to operations.

Figure 3 Increasing Complexity

The human nature of teams and organizations is to not have things be so chaotic and messy. We invent additional processes, procedures, add people, etc. to shut chaos out so we can more predictably deliver. We focus on optimizing processes and patterns. We begin looking inward on operations instead of outward on our customers or competitors.

In many contexts, increasing complexity may be necessary to reduce risk and achieve new goals. The potential downfall over time is the ecosystem becomes so complex and bureaucratic that nothing gets done. We fall behind smaller, more nimble competitors.

Staying Close To The Edge

Most things in the world are non-linear. Staying as close to the edge of chaos as possible enables a team or organization to learn, improvise, adapt and achieve their goals. They should be unafraid to simplify existing patterns or invent entirely new patterns or ecosystems.

Figure 4 Staying Close To The Edge

A key concept that emerged from discussions with Alistair Cockburn and Jim Highsmith is worth keeping in mind. It goes something like this:

"A little less than enough rigor is better than a little more than enough".

More on this concept in future blog posts.