Coping with complexity: the secret to successful digital transformation. Part one: software complexity
“The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and starting on the first one.”
― Mark Twain
Today’s world is becoming ever more complex. As we skyrocket into an increasingly digitalised future where traditional systems become obsolete, it is crucial that we acknowledge complexity, learn how to manage it, and reduce it where possible.
This is particularly relevant in business change programmes. While a number of organisations are now undergoing digital transformation, many are struggling to manage it. This is often the result of complexity: the failure to comprehend it and cope with it. Digital transformation inherently holds many layers of complexity, therefore, should be considered as more than the sum of its parts, and navigated in a way that carefully manages its complexity with the correct processes.
Where is the complexity?
Software complexity arises from the concept that our computational systems are powerful enough to take us to the moon. In the words of Michael Crichton, “to learn how to manage complex systems takes humility”. Software itself is inherently complex. The issues we attempt to solve in software can produce a number of contradictory conditions, to an extent that can even go beyond our cognitive capacity as humans.
We can pinpoint four factors, as identified by Frederick P. Brooks, Jr, which produce inherent complexity:
We need to look at most software complexity as an essential and unavoidable part of our technological development into more mentally challenging areas. This means we do not always need to fully understand every element of complex software, which can be impossible to master in any case, and even more difficult to reduce. Instead, these problems can be navigated by those who have managed complexity and who can bring clarity to complex systems.
Once we have accepted the fundamental limitations of our cognitive capacity as humans, we can address inherent software complexity with a ‘divide and conquer’ strategy. This is helpful in breaking down a complex problem into less complex, ‘bite-sized’ parts. A simple decomposition strategy can be used for such issues:
Therefore, we can address inherent software complexity by forcing the problem to be divided into manageable and comprehensible parts.
Though some complexity is inescapable, sometimes it can be minimised in order to reduce costs. Studies have demonstrated that software complexity can heighten maintenance costs by 25 percent, which is fundamental since these operations comprise about 70 percent of the total acquisition costs. This means that when applied to a project costing $4.5 million project, managing complexity in software systems could mean savings of over $750,000 on maintenance activities.
In light of this potential cost reduction, it is the role of the software development team to produce a veneer of simplicity. In our modern technological landscape, it is not unheard of to encounter systems whose size is measured in millions of lines of code, in particular among safety-critical systems like cars, which are progressing towards model-based tools. Where possible, we must aim to write less code and reutilise existing frameworks, to address the challenges of ever-more complex softwares. In addition, a number of development environments now incorporate the functionality to detect software complexity, for example, in assisting developers to avoid coding errors. Researchers are also developing metrics that determine how the issue might develop in the future, to allow businesses to predict budgets for these systems. In using these new metrics, we must learn to adapt development to avoid complexity (and the extra cost that comes with it) where possible.
Beyond this first essential layer of complexity, it is also crucial to recognise complexity that is external: not an inherent nature of the employed software. This typically arises from the gulf in communication between those that build and those that use a system. Often users struggle to articulate experiential needs to developers while developers struggle to gain important user data. When grappling with complex systems, it is important to bridge this gap between those that build software and its users in order to help reduce complexity in the future.
In digital transformations, it is critical to identify nonessential complexity that can be reduced via communication and specialised management. This will be discussed further in the second part of our discussion of coping with complexity, which focuses on the origins of business complexity and how this can be dealt with.
Brooks, Frederick P., No Silver Bullet: Essence and Accidents of Software Engineering
Crichton, Michael, Fear, Complexity, & Environmental Management in the 21st Century, Talk at Washington Center for Complexity and Public Policy, November 6th 2005
Delange, Julien, Managing Software Complexity in Models, article for Software Engineering Institute
Makabee, Hayim, Divide-and-Conquer: Coping with Complexity, article for effectivesoftwaredesign.com