I once saw a miracle. It was near a ragged wall of broken rocks near Udaipur. One side of the wall had patches of yellowing grass, not taller than an inch, with the occasional dry stalk. The other side, protected from goats by the wall, was a sea of green and yellow grass, up to a couple of feet tall. Here was something utterly simple that had the power to bring life back to these barren hillsides.

Or so I thought. It took a friend, familiar with the area, less than five minutes to set me straight.

I was forgetting, he observed, all the grass from the unwalled side that had already fed some passing goat, and all the grass that someone had cut and sold. If we added up all the grass that had grown on the unwalled side, while the grass on the other side grew tall, we would probably end up with close to what was there across the wall. We might still support building the wall on the ground that tall grass is good for soil conservation and water retention, but we should not be expecting a miracle.

It is easy, especially if you are an enthusiast, to find miracles where there are none. Assessing benefits of an intervention, as I learnt the hard way, requires thinking hard about the appropriate metric and it is easy to get it wrong. For example, as Jonathan Murdoch has shown, the Grameen Bank of Bangladesh has shown the way in profitable lending to the poor, in part by counting its fresh deposits as a part of its earning. But it has forgotten that these deposits would eventually need to be repaid.

If measuring benefits is hard, measuring costs can be a nightmare. You have an innovative programme that involves the community in managing the local greenery and it is quite apparent that your programme is doing wonderful things for the environment. But you also know that you have a lot of charisma and suspect that it might have a lot to do with the way the community has involved itself in your programme. How do you, short of cloning yourself, even start to come up with the cost of reproducing the programme elsewhere? Making the programme scaleable from the beginning is probably the best way to avoid this problem, but it does impose constraints on the process of innovation.

Once you have solved the problem of measuring costs and benefits, you hit the really hard problem — that of figuring what would have happened had you not implemented the programme. For example, you have a novel scheme for teaching children mathematics. At the end of one year of teaching, the children are clearly doing much better. You want to congratulate yourself, announce a miracle, except that you realise that the children are also a year older and have spent an additional year in school. That itself would make them better at solving problems. Indeed in the area of education it is only a rare programme, whose effect outweighs just what you would get by letting the child spend another year at school. How then are you supposed to figure out how much of the improvement in test scores is your miracle, and how much comes from the normal process of growing older?

You can try to find a group of kids who look and feel like the kids in your programme and use their improvement as a benchmark, but how do you determine that these kids were just like your kids? The only really reliable way to solve this problem is to do what drug companies do when they test new products: randomly assign the programme to some locations and not to others, and then compare results. This is actually easier than it might seem: many non-governmental organisations, such as pratham in Mumbai and Seva Mandir in Udaipur, have shown us the use of this technique.

All this takes commitment and forward planning, as well as painful care to ensure that costs and benefits are accurately measured.

Miracles in social interventions are almost always announced without being subjected to this crucial test. This is perhaps why, after so many miracles, the world’s problems look as daunting as ever.

Abhijit Vinayak Banerjee is Ford Foundation International professor of economics at Massachusetts Institute of Technology, Cambridge, USA