Why Simplifying Isn’t Always The Best Answer to Complexity
June 4, 2019 (Adapted from my newsletter, The Briefing. Originally published May 22, 2019.)
How do chiefs of staff navigate, and help their execs and leadership teams navigate, decision support in an increasingly complex world? This week’s guest writer is Aaron Rinehart, Founder and CTO at Verica, who pioneered the application of chaos engineering to computer security during his tenure as the Chief Security Architect at the largest private healthcare company in the world, UnitedHealth Group (UHG). He argues that in an increasingly complex world, simplifying isn’t always the best way for leadership teams or organizations at large to deal with complexity. Check out why and his tips for helping leadership teams navigate that world.
“Software is eating the world” – Marc Andreessen
We are in the middle of a dramatic and broad technological and economic shift in which software and the internet have changed how we consume and deliver products and services, how we work, how and how much we interact with other humans, and other aspects of how we live. It used to be that a company was known for producing or delivering X, and they used technology to do so. Now, companies are technology companies that produce and deliver X.
Given this broad trend toward technology-first business models, it’s important to begin to understand a few key principles we have inherited as a result of the choices we have collectively made:
1. Software only increases in complexity over time.
2. Software abstraction hides that complexity.
3. There are different kinds of complexity, including essential and accidental.
4. Complexity, rather than something to be avoided (even if it could be), IS your value or significantly contributes to your value in this new world.
5. We avoid complexity at great risk to our organizations and careers.
Software only increases in complexity over time
It may seem as if software has become easier and more accessible, but software suffers from a number of inherent ironies and paradoxes. As a result of years of rapid technological expansion (e.g. microprocessors, the internet, social networks, cloud computing, etc.) software has also grown through continuously being abstracted as a result.
“Software complexity both reflects and drives the complexity we face when making critical business decisions.”
Software abstraction hides that complexity
Software abstraction occurs by simplifying the steps required for a person to interact with a system. Such simplification “suppresses” or masks the more complex series of steps that must take place behind the scenes. For example, computers still function using binary code despite the fact that no one is really writing modern software in 1s and 0s. This has allowed engineers to add additional levels of functionality and business-enabling capabilities that would otherwise have been too complex to for us as humans to grok. Software complexity both reflects and drives the complexity we face when making critical business decisions. Abstraction is only one of many contributing factors to software complexity.
There are different kinds of complexity, including essential complexity and accidental complexity
Essential complexity is a direct result of phenomena such as Conway’s Law, which states that “organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations.” What this means is that the designs of the computer systems we build to operate our business are a direct reflection of how that business functions as a whole and all the assumptions, processes, and decisions involved in how it functions. The complexity inherent in how a business operates is also correlated to the reasons why it is successful. The technological foundations we have replaced them with inherit these same characteristics. For example, we didn’t stop making special customer widgets. We just started building and delivering them faster and cheaper online. The business processes, procedures, requirements, and functions that take place in making these special customer widgets didn’t change. Rather the way we built and delivered them did. The complexity in producing, managing, and selling these widgets is directly reflected in its technological replacement, typically through software.
Accidental complexity occurs through the essence of what software is and how it is built. Software systems are more complex for their size than perhaps any other human construct, for the reason that no two single parts are alike. Software is also constantly subject to pressures for change, just like buildings, cars, and airplanes. But buildings, cars, and airplanes are infrequently changed after they are delivered. Software is thought to be “infinitely malleable,” in that it can be changed as often as we like and is quite easy to do so. So, accidental complexity in software comes from the fact that software can change as much or as often as we want it to.
“With change comes complexity, but both are core components of delivering business value.”
Complexity, rather than something to be avoided (even if it could be), IS your value or significantly contributes to your value in this new world
Software that never changed would be very unsatisfying after a while. We come to expect more from it with each iteration. Likewise, a business that never changed anything would not continue to provide value. With change comes complexity, but both are core components of delivering business value. Therefore, we must learn to navigate complexity instead of trying to simplifying it away.
We avoid complexity at great risk to our organizations and careers.
Often, we respond to complexity by attempting to “simplify” away the changes. This shows up as, or as part of, “digital transformation” initiatives and the application of Agile to just about everything. The problem is, attempts to simplify typically result in a similar level of complexity somewhere else. They displace complexity but don’t make it go away.
For example, in the name of Agile, many organizations have adopted a mindset of “just start building something and iterate, and it will all work out.” Or, more specifically, that a useful system design emerges out of a series of mostly independent and disconnected “user stories,” use cases, or scenarios. But, in failing to account for the broader complexity, these organizations might never arrive at a useful solution but a disjointed set of disconnected solutions. Or, they might lose precious time or be leap-frogged by competitors by doing iterative work they could have built in if they had done more analysis or planning early-on, which feels antithetical to Agile. Agile does guard against analysis paralysis, but totally flying by the seats of their collective pants doesn’t serve these organizations any better. They have to make intentional choices about risk tolerance and what level of planning will most likely drive the best outcomes, all of which might or might not be supported by an Agile methodology, at least not initially.
Another example is, in the golden age of so-called big data, an organization might base a go-to-market decision on personas or an “average” customer, when their actual customers might not reflect the average at all. Todd Rose wrote a great article about the time the U.S. Air Force discovered the flaw of averages (see the References section below for more reading on that topic).
In the end, I’m not arguing that we should seek complexity or add to it for its own sake. There’s plenty to go around as a natural consequence of the dynamics I mentioned above. But it’s critical to understand where complexity comes from and find ways to appropriately navigate it in order to make more effective decisions.
A better way forward – navigating complexity versus avoiding it
When it comes to strategic decision making and designing the organization structures and their correspondent interactions between processes, technologies, and people that drive our strategic outcomes, we have to learn to navigate complexity instead of trying to simplify it away. So, how do you do that?
First, evaluate supporting decision information in terms of “what has happened in the past,” “what is currently happening,” and “what might possibly happen”. Past and current information is sometimes only an indicator of the information we knew of or thought to collect based upon our understanding at a point in time. In considering what might happen, we mentally align our understanding of how we believe things work versus how they actually work, thereby lessening the possibility of being blindsided.
Next, consider the decision through a multitude of different vantage points. When dealing with complex systems, perspectives matter. For example, consider using different data, sorting and filtering and querying the same data in different ways, and seeking the perspective of someone you don’t regularly interact with. Overlay and analyze your initial decision information with the alternative perspectives you collected to build a more accurate model of the complexity within the decision landscape.
Lastly, based upon these previous steps, execute a series of small, scoped experiments to slowly reduce uncertainty and validate the decision landscape. Recall that when dealing with complexity we learn to navigate it by making mistakes purposefully instead of avoiding them by design. Making small mistakes cheaply and early allows us to learn from them and make more informed decisions as a result. Taking these three steps can help limit the possibility of adverse impacts, increase confidence, and maximize the potential for success.
References
• Marc Andreessen, “Why Software Is Eating the World,” Wall Street Journal (August 20, 2011): https://a16z.com/2011/08/20/why-software-is-eating-the-world/
• Fred P. Brooks, “No Silver Bullet — Essence and Accident in Software Engineering,” (1986) Proceedings of the IFIP Tenth World Computing Conference: 1069–1076.
• Todd Rose, “When U.S. Air Force discovered the flaw of averages,” The Star (January 16, 2016): https://www.thestar.com/news/insight/2016/01/16/when-us-air-force-discovered-the-flaw-of-averages.html
• Gökçe Sargut & Rita Gunther McGrath, “Learning to Live with Complexity,” Harvard Business Review (September 2011) https://hbr.org/2011/09/learning-to-live-with-complexity