In search of evidence

One concern that was voiced loud and clear at the recent M4P Hub Brighton conference was over the lack of evidence that this M4P thing actually works as well as it should. After 10 years or so of trying, it seems that while some interventions are working (and every case study at the conference was a success story of one kind or another), there are lingering doubts.

Ironically, it may be the ubiquitous development success-stories that have been published over the years (typically called "case studies" to maintain a fig leaf of credibility) that feed these doubts... or perhaps they are a symptom rather than a cause? Either way it seems they aren't quite working.

M4P is built around a wonderfully plausible idea. Who can argue with the assertion that through functioning markets poor people can transact a way out of poverty that is otherwise denied to them? Surely with this sound underlying theory, which has been nurtured with plenty of time and donor money, and allowed huge latitude to present its achievements in the most flattering light, these questions should have disappeared long ago.

So why were some people at Brighton still looking for evidence of actual success to match the M4P promotional hype? There are three parts to the answer. In summary it boils down to the fact that over the last few years M4P has seen an excess of:

1. Over-engineered theory;
2. Purist application; and
3. Distorted reporting.

The market-based incentives that have created this heady cocktail of confusion (among donors, implementers and observers) are obvious. If you can take any plausible idea, brand it and create an orthodoxy around your interpretation, you can promote yourself as a kind of high-priest uniquely positioned as the owner or gatekeeper of the truth. If you can then make the branded idea sound much more complicated than it really is, you can create alliances with those in power whose own interests are that development should be a complex and protected preserve, and you can advise and train novices based on your perceived wisdom. And finally, if you can define how the success of your idea is measured and get away with dressing it up with words like "robust" and "rigorous", then you place yourself beyond any effective accountability.

This is a pretty fair summary of a large part of what has happened in the world of M4P over the past 10 years. This explains the sense of dissatisfaction with how things are going and underlies the questioning about evidence.

In the same way that many other very sensible (and not-so-sensible) ideas have grown up and withered away with time, there is a risk that this will happen with M4P. This is certainly not because M4P is flawed in itself or even because it is rather fad-ish in its presentation. Rather it is because time is catching up with those who have allowed their own commercial interests to take precedence over development learning.

Evidence suggests that the development world has a pretty high nonsense tolerance level and it shouldn't be forgotten that donors have also invested a hefty amount in M4P, so it isn't a question of M4P falling off a cliff. However, if M4P is to survive and grow as it should, now is the time for a fairly rapid rethinking, based on theoretical simplicity, practical common sense and serious scrutiny. If we can get these ingredients in place, more consistent success and more robust evidence will follow a lot more easily...

Over the next few weeks, I will be posting blogs on these three themes and how they should be applied to M4P so that the practice can deliver and report the theoretical promise much more consistently and convincingly than has been the case until now.

M4P - success story?

I have recently finished reading a book by Tim Harford (the Financial Times "Undercover Economist") called "Adapt" and subtitled "why success always starts with failure". The underlying thesis of the book is that there is a parallel between biological evolution resulting from countlesss experiments, some of which have succeeded and many more have failed, and human organisations and development. In other words, to move forward, we should adapt through experiment accepting that there will be failures as well as success.

Last week I had the pleasure of attending a conference on M4P approaches to development organised by the M4P Hub in Brighton, UK. It was interesting to contrast the way that donors and development project implementers talk about their work and the arguments that Tim Harford puts forward. Three things in particular struck me from the conference:

• Donors continue to demand that projects calculate very narrowly defined attributable impact to demonstrate the success of M4P projects, without describing how these results can be credibly calculated;
• Based on the projects that were described in case studies presented during the conference, every M4P project has been a success; and
• The majority of project case studies were descriptive, there was very little analysis of why things had worked out as they did.

So, far from looking back on the last 10 years or so since M4P began to emerge armed with a credible results assessment methodology and being willing and able to talk about learning from failure as well as success, it appears that:

1. We still have no way of knowing what impact the majority of M4P interventions have had (because often we have been measuring the wrong things using the wrong tools); and 
2. We don't have an atmosphere that encourages analytical thinking and a systematic approach of putting the success stories we hear into a wider context of experimentation and failure.

This is the environment in which ABIF will be operating and in which our results will be generated and compared.

I hope that in due course we will have our fair share of success stories and that we will be able to talk about our impact in a meaningful way. However, this will mean looking at some new monitoring and evaluation tools and really trying to understand more about the context of our interventions and the evidence of results that we can gather. It also requires being willing and able to report failures so that we and others can learn from our experiences.

It is obvious from the discussions I heard in Brighton that despite the repeated (and somewhat tired) insistence by donors that projects use rigorous approaches ("start with the results you want and then work on the methodology", as if there is a credible methodology to produce any results that may be required), there is still some way to go before there is broad consensus on what M4P projects can measure with any certainty. You would have thought that given all the attention that this subject has received, if it were all so simple, somebody would have come up with a workable answer by now! Nevertheless, there clearly is room for improving on the current situation and we will do our bit to make a positive contribution to the process.

Implementation phase begins

After several months of deliberation, the ABIF design has been approved by DFID and the implementation contract has been confirmed.

We are now working on recruiting the team and setting up the office. The logistics are never simple in Afghanistan, but we hope to be fully up and running in the next few weeks.

The plan is to work with the private sector enterprises who have approached us during the design phase and to advertise to a wider audience with the intention of launching the first round as quickly as possible.

Engineered inclusive growth

The ABIF combination of an M4P strategy and an investment challenge fund machanism opens up significant potential for the project to achieve sustainable changes in the way that market systems work and impact on target beneficiaries at a level of scale that has been largely missed in Afghanistan until now.

There are two key foundations to the ABIF approach:

1. The challenge fund investment strategy that provides a market based rationale for public support to offset the public risk that impedes investment; and
2. The market development strategy that ensures that investment projects bring measurable benefits to the project's target beneficiary groups.

This approach distinguishes ABIF from much of what has gone before. While there have been projects that have successfully encouraged private sector investment (e.g. ASMED), they have not applied the same kind of project selection principles. And there are many other projects which have focused on rural development (most of ABIF's target beneficiaries live in rural areas), but they have not had the same focus on bringing sustainable change to market systems.

Through the challenge fund, ABIF has the potential to support "anchor investments" in innovative businesses and business models through the challenge fund investment that will simultaneously:

1. Contribute to the restructuring of the economy that is an essential precondition for growth; and 
2. Ensure that growth reaches out into market systems where our target beneficiaries are present. 

In other words, the investment project becomes the driver for systemic change in related markets because achieving systemic change is built into the very business model that ABIF is supporting. To give just one illustrative example, ABIF could support investment in a new food processing facility linked through a contract farming mechanism to a large number of farmers.

While at present we do not have the size of funds required, it is not so difficult to imagine how initial ABIF investments could be extended through the implementation of a carefully designed scaling up strategy. Spontaneous "crowding in" rarely happens, and so having the means to build on the anchor investment could further enhance the impact of the project. That is something for the future!

This model (it seems to me) if fully exploited, could make a significant contribution to Afghanistan's development. It provides the impetus for investment and it ensures that the nature of the growth associated with that investment is inclusive. This kind of engineered inclusive private sector led growth has not been attempted in Afghanistan until now, but just by looking around, reading the library of reports already out there and meeting with Afghan business people the potential is clear.

Two thoughts on M&E

Now that we have pretty much finalised the design phase deliverables, I wanted to share two thoughts on monitoring and evaluation that could be relevant to other projects.

Impact and outcome indicators
Over the last few years, I have watched the development of the DCED Standard for measuring and reporting results for PSD programmes quite closely. A couple of years ago, I had serious misgivings about the way that applying the Standard could distort projects due to the requirement to measure attributable results according to three "Universal Impact Indicators" (outreach, jobs and income). First, it was methodologically impossible (despite some heroic arithmetic and some very grand claims that were made at the time) to isolate the poverty reduction impact of an individual project in this way, and secondly, there were any number of legitimate interventions that did not fit into one or the other of these "universal" indicators.

It is particularly pleasing to see that the 2010 version of the Standard has dropped a lot of the prescriptive approach to calculating impact that was previously required, and most importantly, it is good to see that the Universal Impact Indicators are now in effect "Optional Outcome Indicators". While still using the same terminology, the Standard acknowledges that different programmes will want to use different indicators and also says that the indicators relate to the enterprise (outcome) rather than household (impact) level (precisely because of the methodological issues mentioned above).

It is also very encouraging to hear some of the people most closely associated with the Standard refering to it as being "more about systems than numbers". It has always been pretty solid on systems (particularly the use of results chains), but it all fell apart when it came to the numbers bit. If the thinking behind the Standard really is heading in this direction, then this is very good news.

For ABIF, we will be using enterprise outreach and enterprise income as two of our outcome indicators, but we have also added other indicators for qualitative and value for money related interventions. Based on the market analysis work that we have done, we anticipate that a fair proportion of our interventions could be non-income related, so these other indicators will capture the related outcomes.

We are not planning to quantify ABIF's contribution to changes in household incomes at the impact level. Instead we will use a combination of secondary data, small scale surveys and case studies to link the outcome level results with the observed change at the impact level. This seems to me to be the most convincing, methodologically sound and cost-effective way of describing the poverty reduction impact of our project.

Because of the diversity of investment projects, apart from knowing the total number of end-users, we won't have simple headline figures that can be aggregated for the whole project. But we will have a solid story to tell about our achievements based on an M&E system that is both workable and flexible enough to cope with the range of interventions we are likely to undertake. It also means, unlike some of the numbers that have been produced in the past (and are still repeated to this day), we will have hard evidence to back up the results that we will report.

Baselines
One of the other serious concerns that I have had with approaches to M&E in the past is with the way that baseline and the eventual intervention impact data collection was managed. It always struck me that with any intervention, trying to separate out the change in any given indicator by comparing the situation before the intervention with the situation after the intervention was fraught with problems (mostly to do with the impact of external factors that could not be controlled and contaminated samples etc). It also seemed like a lot of unnecessary work to end up with a result which at best was broadly indicative of some change.

For ABIF, we plan to use a different approach which will be cheaper and easier, but produce results that are just as good. The idea is simply to use a simultaneous baseline, comparing users of a product or service with non-users. It could be argued that our users will be self-selecting etc, but that would be true of any market based intervention. What we will be able to say though is that in identical circumstances (same weather, same third party interventions etc), the users of the product or service experienced (or didn't experience) a benefit that can somehow be quantified by comparing them with non-users. So rather than doing "before and after" type surveys, we will do "users and non-users".

Kabul Intercontinental

Just in case we needed any reminder that risk is the principal obstacle to investment in Afghanistan, we spent yesterday evening having dinner to the sound of gunfire and explosions from the Intercontinental attack...

The human tragedy of the situation is just appalling; people are needlessly dead or injured today. The wider impact of such attacks is to destabilise an already fragile transition process.

What these attacks do for investor confidence remains to be seen, but for sure, it won't have reduced the Afghan risk premium.

Design phase complete

Some personal reflections on the last day of the ABIF design phase...

It has been an intense and extremely interesting experience bringing ABIF from a few pages of draft terms of reference to a fully designed project ready to go in just 10 weeks. Since starting work at the end of March, we have come a long way in terms of developing and piecing together the DFID Business Case, the project strategy, the sector selection and market analysis, the investment strategy, the project operations, the marketing strategy, the monitoring and evaluation framework, a workplan and a budget. And on top of that, I think that we have come up with a new way to manage a challenge fund that could significantly enhance ABIF's success. No doubt we will learn and refine as we go along, but I think that we can be reasonably satisfied with what we have achieved so far.

The good news is that because of the clarity of the original objectives DFID set for this fund, we have been able to put together a project design where each element is logically aligned in order to achieve those objectives. When all is said and done, the core purpose of ABIF is very simple; we are here to incentivise investment that will make money for the project sponsors, change the way that people do business with one another and increase the incomes of poor people. A correspondingly straightforward project design around our core objective of contributing to accelerated and inclusive growth means that we have the solid foundations necessary for the implementation phase. The result (I hope) is elegant simplicity!

What has particularly surprised me during the design phase is the evident level of demand for ABIF. I guess that it is easy to get carried away, of course there is demand for grant funds. But even when we strip away investment projects that are just about making rich people richer, there is still a promising pipeline of ideas for projects that can deliver the kind of  market changes and benefits to the poor of Afghanistan that we have designed ABIF to achieve. With a bit of guidance and encouragement, these projects can get off the drawing board with the help of DFID funds used to offset the risk that stands in the way of investment.

It is always dangerous to make predictions, but I wouldn't be at all surprised if we are able to disburse the funds much faster and achieve much greater impact than anyone envisaged at the time that the initial project outline was developed.

I have lived and worked in Afghanistan for almost 5 years out of the last 10 years. Over that period I have seen many good things happen, but early optimisim often turned into disappointment as extravagant promises of progress were unfulfilled, claims of achievements did not chime with ordinary people, or money was invested in Dubai real estate rather than the future of the country.

With 2014 on the horizon, we are entering a particularly tricky period of significant change and uncertainty. Afghanistan remains largely unprepared for the future that is set out on paper; there are systemic weaknesses in almost every direction you look (economic, political and security). There has been tangible progress but it is fragile and could easily be reversed. Afghan colleagues are hopeful, but nervous about what the future holds. But what strikes me most is that despite the potential of the country and 10 years of reconstruction efforts, poverty remains widespread, particularly in rural areas. The reality is that millions of Afghans still live in the most desperate of circumstances for no good reason at all. This situation has to change.

Afghanistan is a complex environment and there is no single or easy solution, but I am absolutely convinced that ABIF can help to make some difference to the lives of poor people in Afghanistan. The implementation phase is going to be challenging and hopefully rewarding, I look forward to it very much.



If it is my place to do so, I would like to thank everyone who has been involved in the ABIF design phase; particularly the core team of Robert Smith (who worked on the challenge fund operations), Sarah Gray (who worked on sector selection and market analysis), Tamim Ahmadyar (who provided administrative support throughout) and Edriss Raha (who has introduced us to many of the project sponsors with whom we could be working in the future). Simon Foxwell and Frankie Whitwell from Landell Mills have been extremely supportive, providing constructive strategic advice and quality assurance. Additional significant contributions came from our consortium partners ITAD (M&E) and ARG (marketing). Finally, a big acknowledgement is due to the DFID Afghanistan team. Our primary contacts there were Elyas Hashemi and Doreen Broska; their feedback has been invaluable and it has been a personal and professional pleasure to work with them.