Stupidity and frustration...

Yesterday's news that copies of the Koran had been burnt at the military base at Bagram made my heart sink, a feeling compounded by ongoing reports of trouble breaking out here in Kabul and in Jalalabad. After all of these years of international military presence in Afghanistan, and so many incidents from which to learn, how could anyone be so stupid to do such a thing?

For those of us who are here for longer than the typical 6 or 12 month tour of duty and are trying to engage with our Afghan hosts in a constructive way, this kind of behaviour is the cause of immense frustration. Every such incident makes it that little bit more difficult to do the job for which we are paid.

I hope that the vast majority of foreigners in Afghanistan (including the military leadership) would absolutely condemn this latest act of desecration, and I imagine that most find it hard to understand why it happened in the first place. But a little bit more cultural sensitivity would remove the need for the General's after the event hand-wringing.

It is quite possible that somebody will die today because of what has been done, what is there to be said?

Ranking projects and allocating funds

We have now reached the stage of the first round when we have to rank shortlisted projects competing for limited funds. Our shortlist is such that the cumulative requested grant contribution is significantly higher than the total grant budget available for this round.

Of course, we can expect adjustments to grant requests once the applicants prepare their detailed business plans, we explore alternative financing options, and we have serious negotiations with applicants. Maybe some applicants will even drop out. But nevertheless, we need a rational basis for ranking projects (that deliver very different types of benefits) against one another.

Our problem was how to compare (for example) a project that delivers access to affordable genuine pharmaceuticals against a project that delivers increased fruit yields per hectare, or against a project that opens up new markets for goat owners? How to incorporate this diversity of benefit into a comparison of different grant/total investment ratios? Should we monetise everything, do we maximise leverage as some of our colleagues would argue? 

I don't think that there is an acceptable conversion rate for a child's recovery from illness to extra kgs of apples, and leverage is irrelevant when we are interested in value for money development impact. Rather, we wanted to find a methodology that is slightly more sensible (and sensitive) than these approaches. The rest of the methodology described below is pretty standard stuff, but this is the tricky bit...

We have been working on the ranking/fund allocation methodology for some time now, but now we have to apply it. Fortunately the development has now reached a point where we are pretty much ready to pilot it with our shortlist. If this methodology works (if the applicants get it and it produces the intuitively right results) then the idea is to roll it out for the second round.

One thing to note is that this approach only works when it is incorporated with the risk adjusted grant methodology we have already adopted. Otherwise you don't have a rational basis for the "cost to ABIF" column at the final step and you get into difficulties with weighted incomparable criteria and leverage/impact trade off.

So here is an overview of the methodology we intend to use to rank projects and allocate limited grant funds. We would of course be very grateful for any feedback or suggestions for improvements.

Introduction

When we are evaluating projects we are primarily concerned with four questions: 

1. Will the product or service deliver profits to the investor and development impact to the donor?
2. What is the potential outcome of the project, how many of our target beneficiaries could benefit and how much could they benefit by?
3. What are the chances of the applicant achieving the theoretical potential, do they have the necessary capacity and resources?
4. Are we giving the right amount of financial support?

The questions above relate to very different properties of the applicant and the proposed concept. It is not possible, without creating unacceptable trade-offs or forcing incompatible criteria into the same selection step to look at all of these questions together. So, the challenge we faced was to come up with a methodology that allows us to take a sequential and related criteria approach to evaluating applications against each question, regardless of the type of product or service that was being proposed... and then to come up with a ranking that "makes sense" when we take a step back from the details of the evaluation process (does the ranking tool produce similar results to our intuitive sense of what makes a good project for ABIF).

The answers to each of these questions will be obtained from the concept note in the first stage of the challenge fund competition and (in much more detail) from the application in the second stage. The approach we are taking to answering the first three questions is set out below, the way that we answer the fourth is described elsewhere - this is the risk adjusted grant methodology we have developed.

Commercial and developmental viability 

Once we have determined that a product or service is innovative within the ABIF definition, this is the most important eligibility criteria related to the concept itself.

ABIF supports private sector investment where commercial profits and development impact coincide. The Applicant should demonstrate that the business model that they are proposing will be profitable. We are not looking for projects that require ongoing subsidy, we do not want projects that are built on charity or corporate social responsibility. ABIF financial support is limited to the investment required to get a launch a product or service.

But we want more than good business ideas, we will only support projects where the investor will make a good return on his investment and there will be a tangible development impact as an integral part of the business model. So to pass this eligibility test, we need to answer two questions:

1. Is the product or service commercially viable?
2. Does the investment naturally drive the impact logic?

If the answer to both questions is positive, the application moves from the eligibility evaluation to the assessment process. In this second step we rank the projects based on the the potential development outcome (the potential outcome) of the project, which we discount by the project delivery risk (the capacity of the applicant to deliver that outcome) to arrive at the expected development return. 

It should be noted that this is not a means to quantify the expected outcome, it is simply a tool to give us a rational basis for ranking investment projects based on the development outcomes that we can reasonably expect them to achieve. As well as having internal value as a management tool, this also gives a framework for us to use to explain to applicants what we are looking for in a project and how we decided to accept or reject their application.

Potential development outcome

The potential development outcome is a product of two factors:

1. The number of target beneficiaries that could experience a benefit as a result of the new product or service; and
2. The value that the target beneficiaries would place on the benefit that they experience from adopting the product or service.

While the first factor can be projected (e.g. target market size and penetration) and measured (e.g. actual sales), to keep things manageable and avoid a spurious precision at the evaluation stage, we are restricting the estimation of numbers to orders of magnitude. 

However, the second factor is subjective and does not have a constant value within our target beneficiary population. For example, one community might value cultivation training much more highly than another, different people of different ages might value new products differently, according to their adaptability to new technology. 

So the question for us in relation to a given project is how do these two factors combine? If we look at our entire population of target beneficiaries, is there a sub-group of sufficient size that values the benefit sufficiently to incur some cost or effort to consume it, and can the product or service be delivered in such a way that the cost of delivery is less than the value that the consumer places on the benefit it generates. The question then is (to an order of magnitude, anything more precise would be nonsensical given the quality of data we have available) how big is that group?

We start off by looking at the product or service in theory, based on what we know of the relevance, affordability and accessibility of the product or service, the attitudes of the target beneficiary population and the prevailing market conditions in which they operate. This allows us to place the project somewhere in a 3 x 3 matrix and give it an associated score:

Potential development outcome of a project is a product of how many people benefit by how much.		Potential number of target beneficiaries
		100s	1,000s	10,000s
Potential value of benefit experienced by target beneficiaries	Not very valuable	0	0	1
	Quite valuable	0	1	2
	Very valuable	1	2	3

This score is essentially a means to place an evaluation value on the potential development outcome of the investment project. The values we use depend on the relative and combined importance we place on the two variables; for example, the illustrative values in the table above show that we would automatically reject any project in the top left three cells. We could adjust the value of the other cells to produce different results... but these illustrative values are just a start (and probably as good as any other values as a starting point) and we need to see how they play out in practice.

Project delivery risk

However, like any investment today that should bring returns in the future, there is a risk factor that should be used to adjust the theoretical impact downwards. The greater the risk, the greater the downward adjustment. For ABIF, which operates through the private sector, the risk associated with achieving the development potential of any given project is a function of the qualities and capacities of the investment partner and the resources available to them.

The specific factors that we are interested in are:

1. The capacity of the applicant to manage the project and the business; and
2. The capacity of the applicant to reach the potential target beneficiary group already identified.

Before, we have referred to the same questions as issues of the strength of the scale agent and the strength of the route to scale. 

The scale agent is the applicant, the entity that is responsible for managing the project and the business; creating the conditions and then achieving scale. The questions we ask here are to do with the adequacy of the resources and the timescale for delivering the investment project itself, and the capacity of the investor to manage the business resulting from the investment. We look at their financial capacity, their management capacity, and their presence and reputation in the market.

The route to scale is the physical or virtual intermediary network that links the scale agent to the target beneficiaries. This can be a wholesaler/retailer network for a manufacturer, or a provincial/district branch network of a BMO, or a network of village co-operatives for a processor, or it can be a mobile phone network or television channel for an information service provider. Whatever the form of the network, the question for us is how effective is it at linking the scale agent to the target beneficiaries? We consider questions such as:

1. Does it already exist and is already proven, or will it have to be created?
2. Has the network been used for the proposed purpose before, or is this a new use?
3. Does the scale agent own the network, control the network or does he buy the facility of the network?

Again, we construct a 3 x 3 matrix, based on our assessment criteria and the categorisation that we use and give each cell a "discount factor":

The project delivery risk is a measure of how confident we are that the applicant can deliver the theoretical development outcome		Capacity of Applicant to manage the project and business
		Weak	Borderline	Strong
Ability to reach the target beneficiaries	Weak	0	0.33	0.5
	Borderline	0.33	0.5	0.66
	Strong	0.5	0.66	1

These discount factors are subjective and we continue to refine them based on experience of comparing projected outcomes with actual outcomes. The values range from 0, which means that we lack any confidence in the applicant's ability to deliver the potential development outcome, through to 1, which means that we are confident that the applicant has the capacity to deliver the potential development outcome described in the application.

Expected development outcome

The final step in terms of ranking the projects is to discount the potential development outcome by the development project risk to give us an expected development outcome that can be used as a basis for ranking. 

So we construct a table that shows the expected development return of the group of investment projects that we want to compare:

Project	Potential development outcome	Development project risk	Expected development outcome	Ranking
A	2	1.0	2.0	1
B	1	0.5	0.5	5
C	3	0.33	1.0	3
D	3	0.66	2.0	1
E	2	0.5	1.0	3

Again, it must be stressed that the value of the expected development outcome is for ranking purposes, it is not an attempt to quantify the outcome of the project. As a ranking indicator it allows us to compare the various outcomes we might expect to see from different investment projects that are competing for limited funds.

Allocating funds

The first step in allocating funds is to use the risk based approach described elsewhere as a way of determining the grant to be offered to each investment project. This methodology gives us a rational basis for offering a specific grant to a specific investment project. As a result we do not have to trade off leverage and development outcome as a part of the evaluation process. We know how much grant a certain project will require to produce acceptable investment returns, so now we can focus (almost) exclusively on the expected development return as a basis of ranking and allocating funds. In this way we can allocate grants to maximise the expected development outcome for available budget.

Obviously things are not quite so simple as funding projects in ranked order until we run out of funds. We have to look at the individual and cumulative investment contributions and identify the optimal way of allocating funds in order to maximise the expected development outcome for the round. This may mean allocating funds to "cheaper" projects (in cash terms) that are ranked lower than a more expensive one that we do not fund.

So, for example, if we take the projects listed above and we assume that there is a maximum budget for the round of £1,000,000, we would recommend funding projects B, C and D in order to maximise the expected development outcome from the portfolio at least cost within the budget:

Ranking	Project	Expected development outcome	ABIF negotiated cash contribution	Funding recommendation
1=	A	2.0	800,000	No
1=	D	2.0	400,000	Yes
3=	C	1.0	300,000	Yes
3=	E	1.0	400,000	No
5	B	0.5	200,000	Yes

Conclusion
One last word... please don't think for one moment that we are claiming that this is some scientifically rational way to measure the development outcome of investment projects. It is just supposed to be a simple framework to structure a comparison and selection process in a way that:

1. Brings greater consistency to our project selection judgments broadly consistent with intuitive sense of what makes a good project for ABIF;
2. Makes sense to our Investment Panel and DFID who rightly ask why we preferred one project to another;
3. Allows us to explain what we are looking for to our applicants (and to justify our funding decision to them); and of course
4. Focus our efforts and limited resources on projects that stand the best chance of achieving development impact!

Using impact logics

Impact logics show when investments can change lives

Keeping the focus on impact is probably the most important discipline in development project management. It is so tempting to support projects simply because the applicant looks like they can deliver the project successfully or the business idea itself looks good. But while we want convincing partners with good projects, we have to explain time and again to our applicants that we also want a "plus". We also want to see some tangible change experienced by poor people.

It is this impact on our target beneficiaries that justifies using UK tax payers' money to support private sector investment in Afghanistan. This is what makes the difference between a project that is fundable and one that (however good in itself) we will reject. The investment project is a means to an end, not an end in itself.

The tool that we have used for guiding our evaluation of concept notes and for internal training purposes is the impact logic. Impact logics are simply a way of tracing a path from the ABIF intervention to the effect that we want the intervention to create. We haven't done anything particularly complex or time consuming (and certainly no attempts to quantify impact!), but rather we have used the impact logic in its simplest format to test whether a given investment project has the potential to create market change that in turn will lead to a real benefit for a significant number of poor people in Afghanistan.

There is no doubt that the evaluation process has been enhanced by the integration of impact logics into the concept note stage. And at the same time it has helped in terms of developing our internal capacity. I have been really impressed by the way that the team has understood and adopted this way of thinking so quickly. It makes sense to them, so they can communicate more effectively with applicants and in the end we receive better applications.

The end result in this round is that we have now finalised our shortlist with 14 projects (so I was just a little bit optimistic thinking we would have 15), and thanks to the use of impact logics the selection rationale of each would stand up to poverty reduction scrutiny. Whether they survive the next stage of the detailed business plan and financing negotiations is a different matter, but there is no question that every project on our shortlist could (if implemented in the way described in the concept note) contribute to our strategic objective.

So I would argue strongly in favour of the use of simple and graphical impact logics. There is no better tool that I have found so far for checking out whether you can justify the use of tax payers' money in any given development project.

Oh, but please forget the ridiculous calculations and the complex spreadsheets giving "results" such as the USD4.26 net monthly income increase impact that you will have on 15,287 beneficiaries. It's a complete waste of time and the results are a heap of nonsense, whatever anyone might tell you!

What the taxi driver thinks...

View from the back seat, note the in-car entertainment!

Just back from a very productive trip to Herat. Over the last two days we visited all of the Herati applicants on our provisional shortlist plus a few that didn't quite make it but offer good potential for the second round.

One of the highlights of the trip was the chance to have a talk with a local taxi driver. This kind of reality check from spending time with somebody who has no agenda with our project, and probably zero exposure to the world of donors and development (apart from the occasional inquisitive passenger) is so valuable in forming my own opinions on the necessity and effectiveness of our work.

I don't want to risk getting the poor chap into trouble, so let's just say that his name was Abdul. And advance apologies for the way that I report the conversation, I realise that I have not captured the emotion of some of the story. But nevertheless, I believe that it is an encounter worth retelling.

Abdul the taxi driver is probably around 40 years old (I didn't like to ask) and one of 8 people in his family. He is the sole bread winner for his 5 children (oldest 13, down to the youngest aged just 1), his wife and his mother. He is from Herat and apart from a period spent as a refugee in Iran, he has always lived in the city.

His memories are sobering but at the same time his views on life are inspiring.

First, we discussed how he felt about the taxi driving and how things were going generally in Herat. I asked what it was like making a living as a taxi driver, how life is in Herat. He said that "for the rich businessmen everything is fine, but for people like me, all we can do is try to make enough money to survive." Abdul explained that he works about 12 hours per day, earning something between USD25 and 30, of which more than half goes on fuel. he also has to keep his taxi running, which costs him quite a lot in spare parts. He told me that sometimes he doesn't earn enough to feed the family properly and then he gets stressed and ends up having arguments with his wife. "Being a taxi driver in Kabul would be much better than Herat, there are more people."

He said that he tries really hard to avoid trouble with the traffic police, "I know all the rules of how to drive and I don't have problems." But his big issue is with corruption, "Every day people want money for something, sometimes I have to give 3 or 4 bribes every day. If the official price is 100Afs (USD2), I have to pay 300Afs."

As we talk about different things, he suddenly tells a story about his last job. He had a job driving for a government organisation, but when his annual contract came up, the boss told him that he had to pay 20,000Afs to keep his job, "but my salary was only 8,500Afs, and I couldn't afford it." He decided to complain, and then it turned out that other people were complaining about this person as well. Apparently, there was even a prosecution pending because this individual had been taking illegal payments from the companies that used his organisation's services. According to Abdul, everything had been investigated and the case prepared.

Then the person involved "went to Kabul and instead of losing his job, he paid off the people in the ministry and came back to Herat". I could tell that although Abdul accepted that life was like this, he still had a keen sense of the injustice of losing his own job while his boss kept his by paying off his superiors.

Giving alms at the mosque

We stopped off at the mosque in the centre of the city and walked through the beautiful courtyards. He shows me various details. But, I suppose not surprisingly, even as I am admiring the beauty of the building, seeing women and children begging by the entrance is a stark reminder of the poverty that pervades this country.

Once back in the car, we talked about whether things were getting any better or any worse. Abdul compared his current life with the past, he looked back on the time when Najibullah was president, "The security was much worse then. When you went out in the morning, you didn't know if you would come home in the evening. But things were cheap and the shops had plenty of things to buy. The best times were Dr Najib and the early Karzai times. Now it is very difficult."

But during the fighting between the Mujaheddin and the Russians things were not easy for Abdul's family. As the security in Herat got worse, his father was shot and killed when he was out on the streets. The family left Herat and went to Iran, but they found things difficult, "I couldn't work, people were very unfriendly, they treated us badly and shouted at us, calling us names." Abdul told the story of when he was queuing for bread in Iran, the local people would talk against him because he was buying more bread for his family, which was larger than the typical Iranian family, "I had to buy 5 breads for my family, but they only needed two."

We moved on to happier topics. Abdul told me more about his children. He says that having children is very expensive, that he pays 1,200 Afs each month for private school for two of his children, because "when they went to the public school, after two years they could not even write their names". Now he says that they are doing much better, after a year in the new school they can read and write. He is obviously very proud of their achievements.

Herat industrial park while landing

I tell him that I have two children. He says, "You are very sensible. I have tried to stop having children, but then I make mistakes!" We laugh together at the thought.

As we reach our destination, a factory on the industrial park near to Herat airport, I say to him that listening to his story made me so thankful that I was born in a more stable country. Rather foolishly, I said that if I were him I would be quite angry. He pauses before answering and says, "What can you do? This is what life is like."

My meeting at the industrial park is with one of the wealthiest businessmen in Herat. After an hour or so of admiring his newly installed production lines, it is time to leave and he walks me back to my car. When he sees the battered taxi, he looks surprised, "You came by taxi! That makes a change, you people normally come in armoured land cruisers with body guards... That's the way to know what is going on, talk with the ordinary people." We say goodbye, and part of me is feeling quite smug and self-satisfied with my decision to travel around Herat in a 1994 Corolla with "ordinary people". And then I realise with some regret how much more relaxed I had felt talking with a rich man about his big investment plans, compared to the discomfort I had felt as I listened to the tribulations of Abdul's life.

Rarely do I get the chance to have such a prolonged and interesting conversation with a stranger in Afghanistan. But Abdul's warmth and generosity of spirit really struck me and it reminded me that at the end of the day, development is about people who are living in miserable circumstances right now, and who have to struggle with all sorts of problems that I can barely imagine each and every day. I take my hat off to people like Abdul.

Shortlist

The view across Kabul from our office

So, finally we have a provisional shortlist, and now we are working with each of the applicants left in the competition to wrap up discussions on their concept notes ahead of a public announcement of who is on the final list.

We completed our applicant due diligence and earlier this week DFID gave their no objections to the shortlist we put forward (an arrangement that is in place for the first round only while we see how things go). It is good to report that (for now at least) everything seems to be going according to schedule.

We wrote to all of the applicants on 1 February to tell them whether they had been shortlisted or not. Not surprisingly since then we have had quite a few emails and phone calls from applicants that didn't make it through this time. These communications range from requests for feedback and suggestions as to how they can strengthen their concept for the next round, to assertions that we have made a mistake and that we should meet urgently to discuss "ways" in which the concept could be reconsidered for inclusion in this round. Our policy is to be very supportive to those who ask for help, and to politely decline any requests to reconsider the evaluation. The good news is that among those that didn't make it, there is a significant number of applicants with good concepts that can be further developed for resubmission in future rounds.

But for this round, there are 19 applicants on the provisional shortlist (two have fallen by the wayside since the Investment Panel meeting) and we have now sent detailed review points to each one. This is our chance to highlight areas where we need additional information or there are details here and there that need to be corrected. The applicants have until 14 February to respond to these review points and to sign a shortlisting memorandum of understanding.

The MoU sets out all of the "terms and conditions" of shortlisting, making it clear what the obligations exist (and do not exist) for the applicant and for ABIF. The purpose of the MoU is to avoid any future misunderstandings about the basis of the shortlisting and to keep relationships on a more formal, business like basis... we want to have a solid professional relationship with the applicants, but we do not want to get too close to them.

Assuming that everything goes to plan, we then intend to announce the final shortlist on 16 February. This will be when we move on to the full proposals; mobilising our expert pool and getting down into the details of each application. We will have two months or so to prepare full proposals. I have to say that as we move on from one stage to the next in this application process, the work gets more and more interesting and I am so looking forward to watching the business plans come together in the second stage.

In the meantime, our meetings with the applicants on the provisional shortlist go on (I hope that we will get over to Herat this week to do some site visits there). It is really interesting to see how the atmosphere in such meetings has changed. The prospect of the grant is much more real and the applicants now seem to be much more focused on what it is that we are asking for.

And as we go into this final stage of the first round, it is very pleasing that first, we have some strong applicants and very good concept notes to work with, and secondly, from a budget point of view we probably have more than we ideally need in the first round. This means that even though we are in the shortlisting stage, there is still a healthy element of competition in the air. I have no doubt that this will play to our advantage as we proceed towards the full proposals...