Home / more articles - The author - Contact

Learning
Learning more from experience

by Matthew Leitch, 19 February 2004 (updated 25 February 2004)



How much difference does it make?
Learning more from experience
Learning from financial and management accounts
Some pitfalls
Don't be paralysed by analysis
Summary

How much difference does it make?

We all learn from our experiences, but with systematic application of simple methods we can learn even more. Consider the case of David Ogilvy, the advertising genius who founded the Ogilvy & Mather advertising agency in 1949 and by the 1980s had turned it into the 4th largest advertising agency in the world almost entirely by organic growth. In one period of seven years Ogilvy & Mather won every new account for which it competed. One day someone from IBM arrived unexpectedly and simply gave them the IBM account; he knew their work.

David Ogilvy had many gifts, but two things in particular distinguished his advertising. First, he could write interesting and charming copy. Second, he was totally focused on the effectiveness of his advertising (rather than its creative qualities) and used experimentation, not just to guide individual campaigns as others did, but also to build up a body of empirically supported rules of thumb about what works and what doesn't in advertisements of different kinds in different media. For example, he found that, in print advertising, pictures of human faces at larger than life size tended to repel people.

His favourite form of advertising was direct response, where consumers respond directly to advertising. It allowed him to conduct experiments with different ideas, such as headlines, and get easily measurable results.

In his book "Ogilvy on advertising" he relates how the great Stanley Resor, who by then had been head of J Walter Thompson for 45 years, told him he had begun looking at research to find factors which usually work. In two years they had already found a dozen. Ogilvy said he was "too polite" to mention that he already had 96.

Ogilvy exploited the opportunities for experimentation inherent in his business. If you are not in advertising your opportunities will be different, but there will probably be more than you are currently using.

Learning more from experience

As we come to a more realistic understanding of what we understand, can predict, and can control it becomes clear that we will often benefit from learning more about how our organisation and its environment really work, and what we can do that produces results we value.

From the organisation's perspective this learning is essential to improving performance.

For an individual within an organisation, being able to draw convincing lessons from experience as well as by reasoning is useful as a means of finding good ideas and winning support for them. If you can back up your claims with results there is less chance of your discovery being overlooked.

In some industries (e.g. design of chemical manufacturing plants) it is possible to carry out proper scientific experiments, with control conditions, or systematically vary independent variables and record the effects on dependent variables. It is even possible to build a detailed model of the connections between variables and find optimum settings for independent variables. A lot of work has been done on how to design and interpret these experiments efficiently to screen out unimportant variables and build a model.

Once such a process is in live operation it is still possible to use rigorous quantitive methods to adjust the optimum settings to meet slowly changing conditions. EVOP (Evolutionary Operation of Processes), for example, involves varying the independent variables by very small increments, so that the output of the live process is still acceptable but the slight changes in output can guide future trials towards new optimum levels.

Unfortunately, most business situations don't allow these methods to be effective. An advertising agency, hotel, car dealership, or travel agent, for example, is very different from a chemical plant. It can be difficult to vary independent variables in a controlled way. The volume of data available is often small, with many potentially confounding variables. Conditions change in many ways and often quickly. Finally, we are almost always in a situation of running a live process and it is difficult to defend not applying what appears to be the best approach wherever possible.

Despite all these difficulties we have to do something. Most often we "experiment" by simply doing something to see what happens. There's no control group or other comparison. This falls far short of standards for scientific experimentation but it's all we have. What can stop this being a futile exercise in self-deception is that we can use our knowledge of the world and the conditions surrounding our "experiment" do two things:

For example, suppose you think that offering slightly different payment terms will improve sales. You try it on the next sales lead but don't get the sale. However, the buyer tells you that the payment terms are more attractive than the usual terms but explains that a corporate decision has been taken to purchase only from another "strategic" supplier. Overall, this is slightly encouraging for your idea. The confounding factor is the corporate decision and you make an allowance for it. The intermediate effect is that the buyer finds the payment terms more attractive, even though this did not lead to the ultimate result of a sale.

If we persist in trying to quantify effects it becomes possible to make quantitive allowances for more and more factors that could not be controlled in our experiments. The benefits of this approach increase over time.

The power of experimentation can be increased by the following:

Learning from financial and management accounts

At the aggregated level of company financials it is usually difficult to see the effects of trying different approaches until they are widely used. Even then the gradual roll out and combination of many initiatives and other trends makes direct analysis of the summarised financials very difficult.

Nevertheless, we can and should try to learn from management and financial accounts. Here are some ideas:

Some pitfalls

Some things that can go wrong are:

Don't be paralysed by analysis

Paralysis by analysis is another pitfall.

It's one thing to recognise the value of more information but quite another to be unable to act at the right time because you aren't sure what to do. So what can you do when the hard evidence is inconclusive? Here are four possibilities, each more sophisticated and rational than the last:

  1. Choose a 'null hypothesis' until you're confident it is wrong: For example, you might decide to assume that schemes for motivating employees have no effect on motivation unless there is strong evidence to the contrary. Or you might assume they always increase motivation until proven otherwise. Your null hypothesis can be anything you like; there is no magic to it. If your null hypothesis happens to be the wrong place to start from this can mean you are making bad decisions and ignoring helpful evidence for longer than you really need to.

  2. Choose whatever the data point to: Suppose you've tried a scheme to improve motivation and motivation increased by a certain amount. You could assume that this is the true effect and if you use the scheme again you will get the same result. This is very responsive to the data, but can lead to extreme results if the evidence is weak. If your data base is small you are more likely to get values that are away from the true value.

  3. Go for the most likely hypothesis taking into consideration data and prior expectations: In this strategy you need to recognise that you had some expectations before you even tried the scheme, and try to clarify what they are. The evidence of an experiment is then combined with this prior view to produce a revised view. You can do this by judgement or computation.

    Doing this involves setting out all the potentially true hypotheses and attaching a probability to each (a probability density in the continuous case) that it is the true one. Once you combine evidence from the experiment the result is a revised set of probabilities. You then choose the hypothesis that is most likely to be true on your revised view. However, it could be that the most likely hypothesis is barely more likely than others.

  4. Combine all your hypotheses when making forecasts and decisions: This is the same as the previous approach except that when making forecasts and decisions you do not take the most likely hypothesis. Instead, you average the predictions/decision values of all the hypotheses, weighting each by its probability of being true. This explicitly shows the uncertainty you have about what model to use and tends to produce more widely spread distributions for future predictions. Other modelling approaches described above tend to under state uncertainty.

    This approach is called 'Bayesian model averaging' but it doesn't have to be complicated to do.

This list of possible approaches is not complete. For example, sometimes it is possible to take a decision without needing to know much about prior beliefs. It may be that one strategy is very attractive across a wide range of hypotheses so it can be chosen without having a clear idea of how likely each hypothesis is.

The approaches that take into consideration prior beliefs as well as new evidence are usually more suited to business situations because so often the data available are far from conclusive. I personally find it helpful to think about my prior beliefs, especially when the evidence of experience is weak, as it so often is.

Summary

Science has made a huge impact on the human race but there are times in business when it seems to have no relevance. This is because the circumstances in which we work often do not suit the experimental designs most of us learned at school or university. But if we adapt the principles to our circumstances we can learn more from experience and build more powerful cases for good business ideas.



About the author: Matthew Leitch's interests include risk and uncertainty management, cognitive psychology, mathematics, internal control systems, design, the Internet, and human knowledge. He is a Chartered Accountant with a BSc in psychology from University College London. Until recently he worked as a consultant in risk management and systems for a leading professional services firm. He pioneered new methods for designing internal control systems for large scale business and financial processes, through projects for internationally known clients. Matthew now works as an independent consultant.

Contact the author at: matthew@dynamicmanagement.me.uk

Words © 2004 Matthew Leitch

Home / more articles - The author - Contact