In praise of bad news

We are sometimes admonished: "Don't shoot the messenger.” Since there is rarely a logical reason to shoot messengers, such advice should not be needed. But it is, because bad news hurts, and organisations find it difficult to deliver such news to the person in charge.

Andrew Rawnsley's account of Gordon Brown's premiership has received attention for its claims that Brown was abusive and physically threatening to his staff, grabbing lapels, stabbing upholstery with his pen and causing his advisers to cower for fear of violence. If true, that is disturbing – but few people will have found it surprising. High-status men sometimes do abuse that status.

I am worried not so much that Brown may be beastly, but that he is cutting himself off from good advice. Rawnsley describes Brown's fateful decision to pull back from a widely trailed snap election in late 2007. His inner circle waited until he was out of the room before agreeing that such a course would be disastrous. When the prime minister reconvened the meeting, however, this was not conveyed: "No one expressed a clear view. No one wanted responsibility for the decision.”

This is a more significant anecdote than any tale of flying spittle. Any leader needs frank advice, and the biggest obstacle to receiving it is often the leader himself. Even a polite and level-headed boss will be tempted to cut naysayers out of the loop. Knowing this, sensible juniors will avoid expressing criticism or grim tidings if at all possible.

"If you deliver bad news, you're disempowering yourself,” says Professor David Sims of Cass Business School. "You're less likely to be listened to in the future.” For some ambitious subordinates, this is a far worse fate than the threat of being thumped.

A new reality television show, Undercover Boss – which has migrated to the US after airing on Britain's Channel 4 last summer – tries to tap into the dissonance between bosses and front-line staff by filming as a senior executive works incognito in the trenches. It is a delicious premise.

When bosses must don a disguise to learn about how their organisations really work, trouble is in store. One of Friedrich Hayek's obvious-once-pointed-out observations is that society is full of local knowledge, often of a subtle nature and only fleetingly exploitable. That is one reason why decentralised market processes tend to work well. When a hierarchy has to exist, Hayek's insight is the reason why bosses should want to receive truthful assessments of what is going on the shop floor (they don't) and subordinates should be happy to provide them (they aren't).

What makes matters worse for any organisation is that the same dynamic is taking place at every level. Each middle manager is a fresh obstacle to the flow of truth up a hierarchy of wastebaskets. Sensible managers try to let information flow freely, but many are happy to reinforce the barricades for their own peace of mind.

The results of barriers to communication can be catastrophic. H.R. McMaster's influential study of decision-making during the Vietnam war, Dereliction of Duty, is packed with examples. The joint chiefs of staff were warned by their chairman, Maxwell Taylor, that Lyndon Johnson did not like "split advice”. Johnson's defence secretary, Robert McNamara, argued that government would be ineffective if department chiefs were to "express disagreement” with the president. Not disobey, but "express disagreement”. Johnson trusted McNamara implicitly and relied too heavily on the advice of a man he praised as a "can-do fellow”. Isolating himself from dissent, the president made a series of disastrous decisions.

Both the parallels and the contrasts between Vietnam and Iraq are striking. The initial decision to invade Iraq and the appalling prosecution of the war for the first three years were made in an atmosphere in which dissent was made extremely difficult by Donald Rumsfeld and his immediate subordinates at the Pentagon. When the army's chief of staff, General Eric Shinseki, argued just before the war that many more troops would be necessary, he was rebuked. He retired a few months later. Another general, John Abizaid, was a Middle East expert. He warned that "de-Ba'athification” would backfire and was flatly and repeatedly told that it was official policy. Rumsfeld would not even let his commanders use the word "insurgent”. This Orwellianism made it much harder for army officers to rely on the appropriate doctrine: a counter-insurgency strategy.

The tactics in Iraq were initially corrected from the bottom up. One colonel established stability in the city of Tal Afar only when he engaged in a rebellion against the official strategy. His requests for additional men never made it up the chain of command to the commander in Iraq, General George Casey. He went ahead and turned around Tal Afar anyway. The name of this insurgent colonel? H.R. McMaster, who made such a close study of the top-down dysfunctionality of decision-making in Vietnam.

Most organisations cannot rely on having the equivalent of an H.R. McMaster, who had – literally – a PhD in failures of leadership. So businesses try instead to encourage bottom-up feedback with "360-degree” appraisal processes, "workout” sessions such as those introduced by Jack Welch at General Electric, or anonymous suggestion cards. Older tactics include the court jester and the devil's advocate.

The other striking lesson of Iraq is that a change at the top can work wonders. General David Petraeus, who took command in Iraq in 2007, rightly relied on men such as H.R. McMaster to rewrite strategy. But Gen Petraeus and his boss, Rumsfeld's successor Robert Gates, had the advantage of not being the men who had made such awful decisions in the first place. Not many leaders can bear to perform their own U-turns.