House of Lords report concludes attribution practically impossible

Having banged on about this subject for years (for example see this earlier post), and having long argued that the various claims of attributable quantified impact made by many projects and researchers (they know who they are and they should be ashamed of themselves) are methodologically unsound, and having consistently said that the sections of the DCED Standard related to attribution and the so-called "Universal Impact Indicators" should be dropped because they are plain wrong... it is quite gratifying that  two of the conclusions of a report published today by the House of Lords Economic Affairs Committee on the Economic Impact and Effectiveness of Development Aid are:

"We accept that accurate measurement of whether or how much aid helps promote growth is not available."

and

"32. The difficulties of accurate measurement and attribution, and of assessing what would have happened if no aid had been given, are so formidable that the evidence that aid makes a contribution to growth in recipient countries is inconclusive."

And of course, what is true at a macro-level is also true at a micro-level. It is simply not possible to measure impact attributable to a given market development project in any meaningful way, and it is certainly not possible to aggregate the impact of projects across programmes. If anyone can prove me wrong on either of these points, I would be delighted to learn about their methodologies!

So finally, can we bring some common sense to development impact reporting? Can we stop producing over-elaborate spreadsheets that calculate pseudo-scientific results? Can we stop publishing seemingly scholarly case studies that make indefensible claims that a specific intervention had a quantified poverty reduction impact? Can we stop even talking about these Universal Impact Indicators, and discard the claims that they can be calculated for each intervention and then aggregated across programmes by different donors?

In fact, let's ask about how much time and money and effort has been wasted on this subject all because some rather over-confident people made some rather foolish (but very attractive) claims about what they could do with numbers, but then didn't have the courage to admit that they were wrong...

Or maybe the noble Lords and all of the eminent folk that gave evidence to the Economic Affairs Committee have got it all wrong? In which case, can those who promote an opposite line come out and explain the methodology they propose for calculating their "robust" results?

I suspect that they won't because they can't.

So please, let's just take a step back and look to see if interventions are working, instead of getting lost in assumptions multiplied by other assumptions, adjusted for another assumption, and then claiming that we have evidence of impact!

I have always said that we should talk about the strength of the contribution of an intervention to a particular observed development outcome (for which it is possible to gather meaningful evidence), this is so much more sensible than trying to quantify that contribution.

The report says loads of other very sensible things, but this really caught my attention! For what it is worth, I say well done and thank you to the Committee. Let's hope that something good comes out of their work.

Happy New Year!

Kabul
1st Hammal 1391

One of the advantages of living in Afghanistan is having two New Year's Days each "year". Two opportunities to be buoyed up by optimism because we made it around the sun once again, and two chances to make (and no doubt break) new year resolutions!

So in this spirit of optimism, I will hope for three things this year:

1. That this year denies the doom-mongers the satisfaction of saying "I told you so";
2. That for the vast majority of Afghans who want nothing more than a quieter, better life for their children than they experienced, 1391 brings some peace and stability; and 
3. That our efforts pay off and our work makes a difference.

Not so much to ask, surely?

As for personal new year resolutions... I thought about writing something here, but I think that on the whole such thoughts are best kept to myself!

Happy New Year to everyone who reads this and to all my Afghan colleagues and friends... and just as importantly, Happy 16th Birthday to Hugo!

Thinking differently about business enabling environment reform

Incentivising investment and encouraging business enabling environment reform to follow on as a demand led process, is a much better route for donor funding to promote growth and poverty reduction than imposing reforms in the hope of "causing" investment.

In fact, we should not even talk about BEE reform feeding through into changes in the economy (increased investment and faster growth) in the way that the authors of the World Bank Doing Business survey do (e.g. see FYR Macedonia case study in DBR 2012). Such attempts to claim a causal link between BEE reform and investment led growth are dubious in the extreme. Yes, better performing economies tend to have more friendly business enabling environments, but they don't perform better because of their friendly business enabling environment. Never confuse correlation with causation!

In fact, there are very good arguments for not pushing through complex general reforms in fragile states where implementation is always problematic. Businesses on the whole prefer certainty over confusion, and if a licensing process takes a few days longer than it would otherwise, then so be it. Most people ignore the official processes in any case. But even for those who do register, better the devil you know... The only exception I would make to this general rule is where specific reforms are necessary to attract international investment (e.g. in large scale natural resource exploitation), in which case strengthening implementation capacity has to be as much a priority as the reforms themselves. And even then investors want regulatory stability as much as regulatory familiarity.

As our recent survey of Afghan investors showed (and admittedly it was a survey of successful investors) business registration and licensing issues were way down their list of problems, while corruption was much higher. Corruption is to some extent a product of complex systems, but it is much more a product of uncertain systems. And regulatory reform, especially in a fragile state, creates enormous regulatory uncertainty, so ironically could actually increase the opportunity for corruption, or at least negatively disrupt established corruption markets.

This is on top of the enormous opportunity cost of unnecessary or badly designed or prioritised reform. For example, why are donors in Afghanistan once again pouring millions of dollars into trying to set up a central credit registry? Is there solid evidence that it will unlock bank lending or is it because it will improve DB rankings? Why are they simultaneously pouring millions into alternative dispute resolution? Has demand been rigorously investigated or is it because it will increase DB rankings? These are just two cases where the reform tail could be wagging the development dog (and the development dog ends up Doing its Business). My suspicion is that this time and money could be deployed so much more effectively if only we were not so obsessed by some ranking in some table.

It would be much better to start by changing the mentality of state and private sector by building investment in Afghanistan. This would give government and private sector stakeholders a genuine reason to want to achieve the right reforms that work for them, surely much better than creating a load of expensive, ineffective and irrelevant institutions taken from the appendices of Doing Business reports. As if the World Bank has a unique insight into what would make the perfect business environment for the 180 or so countries it surveys and ranks!

(I suppose that one consolation is that any number of projects have tried to do the same things in the past and have run out of budget before they have actually implemented anything, so there is every reason to expect the same thing to happen with these latest attempts.)

This is NOT an argument to do nothing. It is an argument for throwing out the claimed causal model and the central planning approach it encourages, and acknowledging that while better regulation and investment tend to co-exist, tangible investment rather than paper reforms should be the priority. Investment should be the priority because it can be achieved in a shorter timescale, it doesn't rely on weak government to implement and it makes a real difference to real people. Quite simply, let us focus on increasing investment that reduces poverty and let reform take its own natural course when opportunities and demand coincide.

So a project like ABIF focusing on investment has three principal advantages over BEE reforms that in fragile states are rarely implemented. First, the right investment contributes to a much more immediate inclusive growth and poverty reduction impact. Secondly, investment gives the relatively rich and powerful a tangible stake in the future of the country in a way that an improved position in a league table never would. And thirdly, incentivising private sector investment is a much more effective route to meaningful BEE reform in such states than the traditional top-down (or worse still, outside-in) approach.

There are five main justifications for this third claim:

1. It is likely to result in an incremental rather than radical change process, making it easier for businesses and the implementing institutions to manage and adapt to change;
2. Reforms will be demand driven because there will be a constituency (private sector investors or their business associations) to demand reform;
3. Reforms will be designed according to real, rather than imagined needs, as there will be the same constituency able to respond to consultation opportunities;
4. Reforms will be delivered by resources managed by local institutions rather than by imposed by international agencies; and most importantly
5. There will be somebody to benefit from the reform creating pressures for implementation rather than paper reforms.

The problem of course is that it is much easier to measure paper reforms than changes in attitude, and increased local demand for the right kind of reform. So no doubt in these days of reducing all development reporting to the most simplistic numbers, however meaningless they may be, we will be stuck with the World Bank Doing Mischief report for some time.