10. Leading Change

Attempts to manage the transformation of large corporations, organisations, communities, or parts of society, often fail. It’s important to understand the reasons for these failures – and how to avoid these failures.

10.1 Why major change initiatives often fail

Harvard professor John Kotter has analysed the causes of failure of many large change initiatives within corporations or organisations. His research indicates how these change initiatives can falter at eight different points along the journey of transformation:

1. Lack of a sufficient sense of urgency: The suggested new path forward is perceived as being more painful and unpleasant than sticking for the time being with the current trajectory, however awkward that may be; any ideas of far-reaching change are, therefore, put off until another time
2. Lack of an effective guiding coalition: The team of people interested in the change have insufficient power or overall influence to drive the initiative forward
3. Lack of a credible appealing vision of the desired new state of affairs: The envisioned positive scenarios are too vague, have too many unanswered questions, and fail to engage participants both emotionally and intellectually
4. Lack of ongoing communication for buy-in: Without being engaged by regular updates, people lose track of the importance of the change; their attention becomes diverted elsewhere
5. Lack of empowerment of the people who could implement the change: People lack sufficient skills or coaching, are subject to conflicting incentives (for example, for bonus payments or career advancement), or are hampered by unnecessary bureaucracy or a stifling organisational hierarchy
6. Lack of celebration of early small wins: No sense of positive momentum is established
7. Lack of follow through: Gains that are made are not consolidated, and are subsequently unmade, or just linger as isolated special cases
8. Lack of embedding the change at the cultural level: Changes in management personnel or focus can lead to previous progress being unravelled, as older habits reassert themselves.

10.2 Positive methods to manage major change initiatives

With so many points of possible failure, the idea of managing a disruptive transition is daunting. However, forewarned is forearmed. “Eight steps of change” are recommended. Each addresses one of the failure reasons mentioned in the previous section.

These solutions split into three categories. The first category is creating a positive climate for change, and addresses failure points 1.-3. above:

• Increase the sense of urgency, by clarifying the risks if disruptions accelerate in an unmanaged way
• Strengthen the guiding coalition, by forging greater mutual trust, and by noticing and filling gaps in it
• Sharpen the positive vision, so that it is arresting, challenging, plausible, and memorable.

The second category, engaging and enabling the whole organisation, addresses failure points 4.-6. above:

• Communicate often, using multiple media formats, and listening as well as speaking, deepening a sense of shared ownership
• Empower others to contribute, by acting quickly to address any personal or structural blockages that are observed to be hindering progress
• Produce and highlight short-term wins, via a series of quick “sprints” as described by the discipline of agile development.

Finally, the third category, sustaining change, addresses failure points 7.-8. above:

• Don’t let up: consolidate gains, via (if need be) wave after wave of follow-up
• Make it stick: manage the creation and adoption of new elements of organisational culture.

Each of these eight pieces of positive advice can, in different circumstances, make all the difference between the success or failure of a major change initiative. Each piece of advice is important, and the skills to carry it out should be a central part of everyone’s education.

10.2.1 Cultivating a sense of urgency instead of complacency or resignation

10.2.2 Building and managing a coalition to guide vital change

10.2.3 Identifying and addressing misaligned incentives

10.3 Moonshots and moonshot worship

(The text that follows is an extract from the section “Moonshot management” from the book Vital Foresight. More material will be added in due course.)

The word “moonshot” has, like the words “game-changer” and “disruption”, become overused. People with a big idea in their mind often seek to bask in glory reflected from the accomplishments of the Apollo team, by describing their project as a moonshot in its own right. But, again like these other words, there’s more to the concept than first appears.

To be successful, a moonshot needs to involve more than just a large vision – a glorious accomplishment at the end of the project. It also needs at least an outline plan for:

1. Step-by-step progress that can be taken toward that vision
2. The sort of engineering solutions that will be developed and used
3. The resources that could be marshalled to support the project
4. The management of any contingencies that may arise.

Without such a plan, the would-be moonshot is likely to remain wishful thinking. With such a plan, the initiative has more chance of capturing and sustaining the imagination of a broad community of potential supporters – people whose actions can make all the difference between the initiative succeeding or failing. The interest of that larger community will in turn be maintained by the enthusiasm of the smaller community of relevant experts who can assess and review the plan as it unfolds.

10.4 Crossing the chasm

(The text that follows is an extract from the section “Crossing chasms” from the book Vital Foresight. More material will be added in due course.)

The adoption of an innovative trend can be driven by the involvement of greater numbers of engineers, designers, and entrepreneurs in the improvement and distribution of products and services incorporating that innovation. Once the community of solution-builders has reached a critical mass, many hands can make light work.

But as well as considering the growth of the community of people who are improving and distributing these products and services, we should also consider the growth of the community who are investigating and using these solutions. That growth can follow its own pattern of slow, slow, slow, followed, in suitable circumstances, by fast, faster, and extremely fast.

In other words, as well as a potential exponential growth on the supply side of innovation, there can be potential exponential growth on the demand side. The two sets of growth are related, but aren’t the same.

In each case, the growth can be stalled, at an early phase, and fail to shift gears into the faster phase. In the case of an inability to move into the faster phase with a larger community of users, the innovation is said to have failed in “crossing the chasm”, in the terminology of technology marketer Geoffrey Moore, who drew attention to the phenomenon in his 1991 book with that name.

Consider the spread of an innovation throughout a society. It might be the innovation of replacing a petrol-powered car with an electric car – that was an example featured in Moore’s book. It might be lots of other types of innovation in product usage, social attitude, or personal determination:

• Replacing a simple mobile phone with a smartphone
• Installing and using a smart home hub, such as Amazon Alexa
• Switching eating habits to incorporate lab-grown synthetic meat
• Supporting the proposition that marriage should be available to same-sex couples
• Making a determined personal resolution that serious action is needed to reduce the threat of accelerating climate change
• Campaigning for a greater share of public resources to be allocated to investigating and reversing the causes of human biological aging.

For any such innovation, the population can be split into a number of different sections, depending on the factors governing whether they will adopt the innovation:

1. The first group can be called “Innovation enthusiasts”. They are attracted to ideas by their novelty, and are willing to adopt the innovation purely from intellectual curiosity, or in order to stand out from the crowd, as some kind of fashion statement or expression of personal identity
2. The second group can be called “Early adopters” or “Visionaries”. They are interested in the innovation on account of practical benefits it could deliver, such as the apps on a smartphone or smart home hub, or the reduction in numbers of animals being slaughtered for their meat. They are willing to adopt the innovation even though it doesn’t work very well, since they are prepared to put in additional effort by themselves to obtain the potential benefits.
3. The third group can be called “Early majority” or “Pragmatists”. They are also interested in the benefits arising from an innovation, but are less interested in making special effort to work around limitations and problems that arise. They want a solution that works out of the box.
4. Fourth comes the “Late majority” or “Conservatives”, whose instincts are to push back against the innovation and oppose it, but who are willing to change their mind once the majority of society endorses it.
5. Finally, the “Laggards” or “Skeptics” avoid the innovation until no alternatives are available – for example, until petrol-powered cars have been made illegal.

According to pioneering research by University of New Mexico Professor Everett Rogers, populations typically split among these sections in proportions 2.5%, 13.5%, 34%, 34%, and 16%. Of course, the relative numbers depend on how radical or surprising the innovation is. But regardless of the absolute numbers, it’s the case that the majority of potential adopters need a different kind of encouragement, before they will change their habits or disposition, than does the much smaller group of people who are ready to embrace novelty more instinctively.

That core insight of Geoffrey Moore’s Crossing the Chasm can be summarised as follows:

• A new product can gain some initial market interest, merely on account of its novelty, or on account of underlying technical merit
• The early adopter proportion of the market will be ready to engage with a product, just by seeing its technical feature set
• This early surge of interest can mislead the creators of the product, who think that they simply need to deliver “more of the same” to expand their market penetration
• However, the mainstream market operates by different criteria, and is positioned on the other side of a “chasm” from the early market
• Adopters on the other side of the chasm seek complete solutions, reliability, and convenience
• Pre-chasm adopters are willing to put up with poor usability, and make do with some compromises and workarounds; not so the post-chasm adopters, who want solutions with minimal friction or inconvenience
• Pre-chasm adopters are ready to walk a solitary path; post-chasm adopters require social validation that their choice makes sense.

That final distinction deserves emphasis: members of the visionary early adopter community are looking for ways to “get ahead of the herd”, as Moore puts it, whereas members of the pragmatic early majority community are more inclined to “stick with the herd”. Their decisions about adoption are strongly influenced by what they perceive their peers are doing. If they see “people like them” adopting the innovation, it will bolster their own inclination to follow suit. But if the only people they can see using or advocating the innovation are in some ways “strange” or “not like us”, it will deter them: members of the early majority are reluctant to be seen as sticking out from their community.

This tendency is a deep-seated aspect of human nature. The instinct to follow the practice of the others in our local tribe served, usually, to strengthen community bonds. But that instinct can also leave the community stranded in a mistaken view of the world or an unhelpful set of practices.

So how do innovations cross this chasm? Moore refines his metaphor from “crossing the chasm” to “crossing the channel”, referring to the D-Day landings of allied forces across the English Channel in northern France in June 1944. Rather than spreading out the invasion force all throughout the territory controlled by enemy forces, the attack focussed massive strength in a small number of landing areas. Only once these landing areas had been successfully controlled did the invasion force start to spread out. In contrast, Moore warns, most innovation effort is wasted. In effect, innovation initiatives tend to scatter effort all over a large domain, without reaching a critical mass in any one area. That kind of effort may be sufficient to inspire members of the early adopter market, but it’s not sufficient to convince members of the early majority to change their ways. That’s why most innovations lose their momentum. That’s why most start-up companies die in the “valley of death” – another name for the chasm between early market excitement and actual commercial impact. And that’s why potential disruptions frequently fail to live up to their potential.

The equivalent of finding and securing a suitable landing region, on the other side of a channel of water, is to find a “killer app” for a platform – a reason why at least a portion of the early majority of potential users will pay more attention to what the platform can provide. This “killer app” must be something that can deliver real value with minimal effort by the user.

10.5 Managing optimism and pessimism

How can transformational change initiatives deal with the drawbacks of pessimistic attitudes, without succumbing to naïve over-optimism?

Pessimism can be dispelled by:

• Highlighting a credible outline roadmap, of various stages ahead in the intended transformational change
• Highlighting small but meaningful steps forward that are accomplished
• Quickly addressing new blockages that arise.

In some cases, people whose pessimism is grounded in deep psychological roots may need to be removed from the initiative, or moved into a different role where their pessimistic outlook causes less damage.

Note that pessimism should be distinguished from constructive criticism. All projects benefit from constructive criticism. Key characteristics of constructive criticism include:

• Not just pointing out problems but also considering possible solutions
• Seeking objective information about issues and risks ahead.

Whilst pessimism can damage the mood in a project, there are also risks from naïve over-optimism:

• Lack of attention to problems at an early stage, before they become harder to solve
• Resources will be wasted on “solutions” which received excited support, rather than being deployed onto other methods which have a greater likelihood of success
• Potential allies of an initiative can be deterred from collaborating with it due to perceiving it to be overly naïve.

10.6 Existing organisations vs. creating new ones

In seeking to progress a significant transformation in society, what are the pros and cons of working within existing organisations versus creating new organisations?

Existing organisations have many assets:

• Personnel with knowledge and skills
• Processes and methods with a track record
• Accumulated data and information
• Tools and systems
• An established reputation
• A network of existing contacts.

Nevertheless, there are occasions in which it can be more effective to create or join a new small organisation. Existing organisations can:

• Fall into ruts of unproductive behaviour
• Prioritise their own initiatives, that use the majority of available assets
• Insist on processes and methods that are less suited to the needs of a new initiative
• Impose a culture, ideology, or political orientation, that damage the new initiative
• Develop a reputation that is counter-productive in some cases
• Become “captured” by interests of management or financial supporters
• Lose their appetite for potentially riskier activities, preferring to concentrate on incremental improvements

In some cases, a new organisation may be created with the purpose of influencing the operation of existing, larger organisations. On seeing the accomplishments and the potential of the new organisation, an existing organisation may:

• Be inspired to copy the methods and activities of the new organisation
• Acquire the new organisation, as part of its overall structure.