This is a great snap shot piece Peter. Good work. I am interested in learning how can we at pManifold (www.pmanifold.com) could take up this task of MFI impact valuation using model driven RCT (Random Control Test) or other better methods. It will be great if you can describe your thoughts on what would mean to do MFI impact testing. We would be glad to take upon this as it falls very much in our 3M framework – Market Intelligence/Development, Modeling Businesses & Monitoring performances in our 3P (Profit, People and Planet) mandate.

I am attaching an excerpt from Nachiket Mor below, which I think is very good capture of ideas from Mr. Angus Deaton’s paper http://www.princeton.edu/~deaton/downloads/Instruments_of_Development.pdf:

From: Nachiket Mor [mailto:nachiket.mor@icicifoundation.org]
Sent: Sunday, July 12, 2009 9:33 AM
Subject: To RCT or Not To RCT

Sir: Thank you very much for sharing this excellent document by Professor Deaton. The main message of the paper, that way I understand it, is that before any technique there must be a theoretical model from which testable hypotheses are extracted and then, using whichever technique seems most appropriate, actually put to the test with empirical data, paying attention to mean effects, standard errors and the manner in which impact is actually distributed. I too worry greatly about the often “model-free” nature of the RCT methodology and the desire of its practitioners to move directly from experiment to policy without a serious intervening theoretical framework guiding it.

That said, I do believe that RCTs represent an important advancement in the tools needed for “signal-extraction” from “noisy” real-life data and offer an opportunities for practitioners like myself and my other colleagues in the ICICI Foundation to carefully build learning within the projects that we implement. Data needed for large scale econometric analysis are often not available and particularly when one is seeking to discover specific interventions that one can use on the ground, there is a need for specificity and precision for which RCTs offer the only way forward.

When talking to researchers on the ground I feel that there is often a needless polarisation for and against RCTs – it is not clear to me how and why this has happened – RCTs are just a tool that are sometimes most suited to analyse problems on the ground and sometimes they are inappropriate ones to use. I worry that often those opposed to RCTs are pursuing “model-free” experiments of their own without even the basic rigour of randomisation or any of the caveats that Professor Deaton recommends – for example, I have seen a number of baseline-mid-line-end-line results without any attempt to control for counterfactuals. Even in large scale econometric / regression models it is often hard to see what the underlying theory is – it often looks like pure data-mining with massive number of variables added with no attempts being made to correct for loss of degrees of freedom or to check for out-of-sample fit.

The way I see it the principal problem seems to be common to all of these empirical analyses – there is often not even a rudimentary theoretical model that is offered before large scale empirical projects are being undertaken – empirical finance is often the most guilty of this but seems to be a thriving industry.

As Professor Deaton says:

“The general lesson here is once again the ultimate futility of trying to avoid thinking about how and why things work; if we do not do so, we are left with undifferentiated heterogeneity that is likely to prevent consistent estimation of any parameter of interest”.

“The demand that experiments be theory driven is, of course, no guarantee of success, though the lack of it is close to a guarantee of failure. It is certainly not always obvious how to combine theory with experiments. Indeed, much of the interest in RCTs—and in instrumental variables and other econometric techniques that mimic random allocation—comes from a deep skepticism about much economic theory, and impatience with its ability to deliver structures that seem at all helpful in interpreting reality. The wholesale abandonment in American graduate schools of price theory in favor of infinite horizon intertemporal optimization and game theory has not been a favorable development for young empiricists. Empiricists and theorists seem further apart now than at any period in the last quarter century. Yet reintegration is hardly an option because without it there is no chance of long term scientific progress.”

“Finally, I want to return to the issue of “heterogeneity,” a running theme in this lecture. Heterogeneity of responses first appeared in Section 2 as a technical problem for instrumental variable estimation, dealt with in the literature by local average treatment estimators. Randomized controlled trials provide a method for estimating quantities of interest in the presence of heterogeneity, and can therefore be seen as another technical solution for the “heterogeneity problem.” They allow estimation of mean responses under extraordinarily weak conditions. But as soon as we deviate from ideal conditions, and try to correct the randomization for inevitable practical difficulties, heterogeneity again rears its head, biasing estimates, and making it difficult to interpret what we get. In the end, the technical fixes fail and compromise our attempts to learn from the data. What this should tell us is that the heterogeneity is not a technical problem, but a symptom of something deeper, which is the failure to specify causal models of the processes we are examining. This is the methodological message of this lecture, that technique is never a substitute for the business of doing economics.”

I feel that this is central problem underlying a lot of the empirical work that unfortunately even we are engaged in – RCT and Non-RCT. We are now trying hard to see how we can correct this.

Thanks Peter for an excellent summary and Rahul for sharing that email of mine. I right now want to express some concern about (and strong disagreement with) the pressing need for impact evalution of microfinannce. It is my belief that:

a. Financial development is strongly associated (many believe causally) with the growth and poverty reduction of nations. I am happy to take this as a given. And, in my view, if one has to test something it would have to a proposition such as this one over a twenty year or longer horizon (something that Professors Pande and Burgess have done quite conclusively).

b. If one agrees with the above proposition I fail to see why there is a need to pick up one segment of the population or one product and ask if it reduces poverty — it is like asking if credit cards reduce poverty of middle income households.

My own interest in randomised experiments such as the one we supported with Spandana is to obtain design inputs and to ask the question about how we can come up with better forms of access — we had no interest in determining if partial access produces partial reductions in poverty.

Mor's comments very perceptive. I think access to credit has a huge psychological impact on the borrowers(??). This in turn lead to inclusivity , legal compliance, better civic order.
This may sound hand wavy but independantly verifiable

Thanks Rahul and Nachiket.
@Nachiket, I agree with the proposition that access to finance is associated with economic growth and poverty reduction, so microfinance’s contribution to the broadening of financial access should be supported – recognizing its limitations and areas for improvement. I also really appreciate your comment regarding the motivations behind supporting the Spandana study.

I believe the large amount of attention paid to this single product (micro-credit), and continual attempts to specify its impact, go back to the manner in which it has been advertised or marketed to funders. To borrow from your example, if credit card companies told their investors or regulators that credit cards lift people out of poverty, then it might make sense for someone to ask credit card companies to demonstrate this, or at least show that they are not harming customers. But since they do not make such an assertion, while MFI networks and fundraisers (primarily outside of India) have continually made this claim, I think that this is why there has been such a stir about how much micro-credit can or cannot do to alleviate poverty. Donors and “social” investors can put their money in only so many places/projects to achieve their missions, and regulators must choose which industries should be given preferential treatment, so I think there is a desire to measure where they can each get the most bang (i.e. impact) for their buck. At least this is how I have interpreted the recent attention being paid to things like new social metrics, RCTs, etc.

That being said, I tend to view micro-credit in the way you have described in your post (and previously): as a relatively blunt tool that must be constantly improved/sharpened. Once we admit that the tool is imperfect and are devoted to its improvement, the results from studies/RCTs can be viewed in a non-threatening light, such as “What do these results tell us about what we can do better?” instead of “So did we have the impact we have been promising or not?” The former is clearly a more constructive attitude and also likely to lead to a greater impact in the long run anyway.

In case others have interest in reading Profs. Pande and Burgess' paper, one version is available here: http://econ.lse.ac.uk/staff/rburgess/wp/dobanksma…