Finding the right successor is a vital role of chief executives. But as high-profile debacles show, a botched job can spell trouble for the business

Ensuring they have the right successor in place is one of the key responsibilities of any business leader. Indeed, says Val Gooding, who retired from Bupa last May after 10 years as its chief executive, it is "possibly the single most important thing the board does, because it can spell the difference between spectacular success and dramatic failure".

While the board makes the ultimate decision, it is the chief executive's role "to deliver really great candidates", explains Gooding, adding that in the later years in the job this should become "the number one priority".

Ray King, who joined Bupa as finance director in 2001, succeeded Gooding. It is testament to his success that not only has the group weathered the downturn well, but also that there have been no senior management defections as a result of his accession to the top job.

"Ray and the board have worked hard to ensure that everyone remains motivated, challenged and appreciated, because they want to keep good people," says Gooding, who resisted the temptation to slip into the chairman's role. "It would have been convenient and I'd have loved it, but it would have been wrong for the company."

Most observers agreed that the Bupa succession has been handled in exemplary fashion, and it has much to do with Gooding's own view of her role there. "My job was to manage the company as well as possible; grow it, improve it and know that it had all the ingredients to continue being successful once I'd handed the baton on to someone else. The real measure of a chief executive's success is how well the company does after you leave."

In sharp contrast to Bupa, and to organisations such as Imperial Tobacco, where chief executive Gareth Davis has announced that chief operating officer Alison Cooper will succeed him in May—causing scarcely a ripple in the market—there were several high-profile examples of companies that appeared to mishandle succession in protracted, dramatic and damaging fashion in 2009.

Marks & Spencer, ITV and Channel 4 were all looking for new chairmen and chief executives, with candidates falling by the wayside on an almost weekly basis. Share prices and corporate reputations suffered accordingly. The problems were exacerbated at M&S and ITV because Sir Stuart Rose and Michael Grade respectively combined the chairman and chief executive roles.

There are occasions when blending the jobs may be a justifiable breach of corporate governance best practice, but companies rarely think through how to extricate themselves. The two positions are markedly different, and such a high concentration of power in one person often militates against the kind of balanced decision-making required to achieve a seamless succession.

The appointments in November of Archie Norman as ITV chairman and Marc Bolland as chief executive of M&S were greeted by five per cent rises in each company's share price. Some commentators argue that such debacles are the exception rather than the rule, but research suggests that succession planning in even the best companies is usually far from admirable.

A survey due for release this month from Korn/Ferry Whitehead Mann shows that just eight per cent of chairmen, chief executive and HR director respondents judged succession planning in their own company to be "excellent". Yet respondents to a separate poll by the company said that chief executive succession is one of the three most important board functions.

Research by PricewaterhouseCoopers has found that although the best organisations pick at least two possible successors for each key position, in practice only 40 per cent of roles are filled by those identified for promotion. This suggests, says Michael Rendell, PwC's global head of HR services, "that the processes are not working efficiently even in the best organisations".

If succession planning is so important, why do many companies do it so badly? The main reasons are that they don't prioritise it, see the process as continuous, plan far enough ahead or think sufficiently carefully about the composition of the board, say observers.

"Most companies leave it too late, so they panic and get boxed into a corner," says Marc Jobling, head of investment affairs at the Association of British Insurers (ABI). Instead, says Peter Breen, a partner at executive search firm Heidrick & Struggles, succession should be a regular item for the board. "Chairmen should be alerting headhunters to the fact that someone might be standing down in two or three years' time so that headhunters can keep their eyes and ears open for successors," he says.

Yet this implies that candidates should come from outside the business, when a growing body of evidence suggests that internal successors are more successful. Over 90 per cent of the "great" companies Jim Collins identified in his 2001 book, Good to Great, were run by chief executives who grew up in the company. "You need executives who have ambitions for the company rather than themselves, and those people tend to be insiders rather than outsiders who can be bought," writes Collins.

Companies often pay more for outsider chief executives, Collins points out. This would certainly have been the case had ITV appointed former BSkyB boss Tony Ball as its top man. Ball was said at one point to be demanding £42m in a five-year share-and-pay arrangement to take on the role. While the external search rumbles on, acting chief executive John Creswell, formerly chief operating officer and one of the broadcaster's longest-serving senior directors, has said he will seek "a fresh challenge" once a new chief executive is named. Who can blame him?

While all organisations benefit from an injection of new blood, usually the best way to do it is to bring in talented people mid-career and then develop them, marrying the twin benefits of an outside perspective with internal knowledge, suggests one veteran investor.

Guy Beresford, a partner at Korn/Ferry Whitehead Mann, explains that succession planning is part of talent management. "The sooner you can identify potential successors for any leadership role, the sooner you can start to develop and prepare them with different and challenging roles," he says. Yet even where it is done well, the process is fraught with problems. For example, says Beresford, internal candidates struggle to gain experience of the external exposure that chief executives must contend with—a problem exacerbated by the fewer numbers of executives now sitting on boards.

But Mike Haffenden, director of the Corporate Research Forum, a network for leadership professionals, believes companies can overcome such difficulties by eschewing the classic succession planning approach in favour of broader talent pools.

"Putting names in boxes doesn't really work anymore, not least because it fosters a silo mentality, when companies need to be flexible and adaptable to change," Haffenden explains. "Talent pools, on the other hand, allow you to see right across the organisation and encourage cross-functional moves, which broaden people's skills and experience. They also help to minimise unhealthy internal competition."

But Haffenden believes the chief executive has to own the process, with strong HR back-up. "That support includes the provision of hard data about individuals' accomplishments, rather than the kind of soft, gut-feel anecdotal stuff about people's behaviour or achievements that is so often used as a basis for promotion."

Injecting more science into succession planning is a long-overdue development. As Beresford points out: "The objectivity and analysis that goes into recruitment at the top of firms is often less than at lower levels, which is clearly ridiculous. Boards tend to rely on instinct rather than a well-defined process."

But chief executive candidates should not be assessed in isolation: it is their fit with the rest of the board and the challenges facing the company that should determine whether or not they are appointed. Here again internal candidates carry an advantage over outsiders. Last November, Ian Smith left publisher Reed Elsevier after just eight months in the chief executive's job. Erik Engstrom, a Reed insider initially passed over for the job, replaced him with immediate effect.

Growing numbers of boards now evaluate themselves, although only a minority open themselves up to outside assessment in order to gauge skill and capability gaps. While headhunters are taking on the mantle of external facilitators for such evaluations, corporate governance experts believe the role of search firms in succession management should be open to more scrutiny.

Sarah Wilson, managing director of Manifest, the proxy voting agency, believes headhunters are partly to blame for the continuing lack of diversity on boards and the perpetual recycling of "the usual suspects", as she calls them. "There is plenty of talent out there, but people still seem unwilling to take risks," she says. "But not having a questioning mind is risky in itself."

Tom Powdrill, head of communications at corporate governance advisory group PIRC, believes that more non-traditional appointments would generate the kind of "cognitive diversity" boards need to foster a healthy level of challenge. Most institutional investors lack the time for anything other than the most superficial involvement in the nominations process for board appointments. The veteran investor is, says Powdrill, "one of the few investors who have developed relationships with most search firms". But even he only talks to the headhunters "in terms of generalities"—believing that the boards are much better placed to make key decisions.

Others disagree. Peter Butler, chief executive of Governance for Owners, thinks that shareholders should be directly represented on nomination committees as they are in some Scandinavian countries. But they would employ people of non-executive director calibre to act on their behalf, as they have the relevant board experience to make sensible judgements.

His proposal is not popular. The veteran investor points out that the pressure Legal & General placed on ITV to appoint Tony Ball "made the board's already difficult job much more difficult". And the ABI's Jobling asks: "Why should Peter Butler's independent directors do a better job of identifying suitable candidates than the non-executives already sitting on the board?"

PIRC's Powdrill hopes that the Walker Review and the Financial Reporting Council's review of the Combined Code, both of which reported last November, might lead to "some more specific guidance" in the code on how to achieve more effective succession.

But, arguably, companies don't need further guidance. The keys to succession planning are simple: businesses should see it as a continuous process, plan ahead, create talent pools, evaluate boards on a regular basis, try to promote from within and communicate succession strategies more effectively to shareholders.

And there is another key factor—longer chief executive tenures. Bupa's Gooding and others agree that the ideal term is around 10 years because, as she says, "if you're only around for two or three years that doesn't give you time to build your bench strength".

A study of global chief executive succession patterns by consulting firm Booz & Company found that the turnover among European chief executives has fallen slightly since the onset of the economic crisis, but the average UK tenure is still only five-and-a half years.

If we assume that the acid test of a leader's success is about how well the company performs after they leave, then incumbents need to subsume their ego to that of their organisation and ensure that they neither succumb too prematurely to the siren call of headhunters, nor outstay their welcome.