Health warning to governments: Bad numbers are bad for you

I have been wondering what it is about the prevailing obsession in international development with quantified results and league tables that really bothers me. I think that I am beginning to formulate an answer, and as I feel the passion of the number advocates, I feel like some kind of underground resistance fighter as I see what is happening in mainstream thinking on this subject.

Basically, I have come to the conclusion that many numbers and the league tables that are used to compare them are fundamentally flawed and frequently used in a way that undermines rather than promotes meaningful development. I first became aware of this when I was working at Katalyst (a project that has put huge efforts and incurred massive expenses trying to quantify its results), but now I see this trend being taken to absurd lengths. The way that this is happening needs to be challenged; dubious indicators, inadequate methodologies and bogus results should be exposed.

Even the architects of such absurdity (promulgating or guided by the work of the Donor Committee on Enterprise Development and its ridiculous attempts to come up with a methodology that produces quantified results for individual projects that can then be aggregated at a programme level) admit that results have been deliberately falsified by projects to make numbers more palatable to donors or are based on such combinations of  assumptions as to be meaningless. And remember that these very numbers then get quoted by government as evidence behind policy making!

We are living in a world where numbers have become ends in themselves, influencing much larger issues such as donor co-ordination and project design (the tail wagging the dog). One example of this is a major new DFID project where according to the published project documentation, one reason that DFID rejected the option of a multi-donor approach, because it would be easier to attribute results if they went alone. In ABIF, at one stage we were told that if we could not quantify the future impact of an unknown intervention with an unknown partner in an unknown sector at an unknown point of time, DFID would not be able to approve funding; "Just make something up!", we were told... so we did because at the end of the day, if this is the price you have to get on in life, then so be it (think Vaclav Havel's greengrocer).

My concerns about many of the league tables I see are based on three underlying reasons:

1. The design (choice and weighting of indicators) of the league table is a reflection of a particular and usually questionable world view;
2. In too many cases the results that determine a given position are inaccurate (either deliberately manipulate or incompetently calculated); and
3. Tables tend to become a substitute for thinking, it is so much easier to use simplistic devices than to think through issues.

Take the Doing Business survey, my favourite example at the moment. If you accept that the World Bank team that designed the indicators has devised the perfect business environment and that the closer every country in the world gets to this state of supposed perfection, the better things will be, then probably you will like the whole idea. It is so easy, you can design projects whose objectives are to improve the Doing Business ranking!

If on the other hand, you think that it is just possible that at different times and in different places, there could be room for local variations and variety in solutions, then you may question the one-size-fits-all approach Doing Business exemplifies. You may even be concerned by the opportunity cost of chasing this particular development rainbow.

So my proposal is that to break our addiction with numbers, we should follow a process similar to the public education and regulation that has helps people to quit smoking. First we should print health warnings on results and league tables, these could be increased in size and become more graphic with time. Then we could ban their use in enclosed public spaces (especially conference halls). Finally, we should go after the people that are peddling these highly addictive and dangerous substances, helping them to find alternative and more productive livelihoods.

More seriously, I really do argue that we should get back to thinking and looking at what are the intangible capacities that societies need to survive and develop and we should build our development objectives and results reporting around them. Put other indicators in their rightful place, as indicators, not objectives. Let's accept that human development is not a linear process, it is complicated and not everything can be quantified or monetised.

My thought is that development objectives could be better defined in terms of two key capacity dimensions relevant to human happiness in an uncertain world (neither of which I think can be quantified, but nevertheless we all know when they are present or absent):

1. The ability to withstand negative events; and
2. The ability to take advantage of positive opportunities.

These are the higher-level objectives that I have in mind as we implement ABIF. Whatever outcome or impact numbers we may have to make up in the future (and no doubt we will have to), these are the real measures that we have in mind when we use public money to improve the lot of our target groups in Afghanistan. But don't tell anyone... we could be in trouble if they knew!

Moving the focus from the superficial Doing Business-like indicator to this kind of goal would have huge implications for how development projects are designed and managed. It is this kind of thinking that underpins the market development approach, and this is why (despite my misgivings about some of the nonsensical claims I have read) I think that M4P is a very intelligent approach to development.