Announcing the Handbook on Using Administrative Data for Research and Evidence-based Policy

We are excited to announce J-PAL’s new Innovations in Data and Experiments for Action (IDEA) Initiative.

IDEA’s mission is to support governments, firms, and non-profits (“data providers”) who want to make their administrative data accessible; analyze it to improve decision-making; and partner with researchers in using this data to design innovative programs, evaluate program impact through randomized evaluations, and scale up successful programs. IDEA will create long-term institutional partnerships with data providers who are interested in partnering with J-PAL at each step along this pathway. 

Around the world, vast amounts of data are now being digitally collected or stored, which creates tremendous opportunities to transform lives through better social policies. Using these data sets in creative and innovative ways to evaluate programs and therefore improve outcomes is a vital step toward making significant progress in the fight against poverty worldwide.

Yet so little administrative data is accessed, analyzed, or used in impact evaluations to improve decision-making. 

The core challenges we aim to address with IDEA include: cumbersome or non-existent data use policies and access modalities; low usability of data stored in outmoded formats and disconnected databases; unclear value proposition for data providers who have little understanding of what research is possible or who perceive high risks; and a lack of advanced technical skills for analysis and experiments.

J-PAL is already working at the intersection of existing administrative data and critical social problems, but the majority of the evaluations conducted by our affiliated researchers still rely on primary data collection and surveys, some of which can be expensive and lengthy. Experimental evaluations using administrative data can be cheaper and faster.

IDEA will enable us to greatly expand the use of administrative data in randomized evaluations, and do so in a much more systematic way. Examples of some initial projects include:

In the United States: J-PAL North America has collaborated with many city and state governments to find innovative uses for administrative data.

For example, in Philadelphia, J-PAL affiliated researchers are using administrative data to follow the impacts of a summer jobs program on crime, employment, and educational outcomes in low-income neighborhoods.

Several free and publicly accessible guides and catalogues of administrative data use have also been created.

In India: As part of J-PAL South Asia’s institutional partnership with the government of the state of Tamil Nadu, J-PAL has provided technical advice to help establish a Data Analytics Unit in the state’s Economics and Statistics department. Initial projects involved working with the state police to link a list of unidentified bodies to a data set of missing people, bringing closure to grieving families. 

Additionally, in some states, a current project includes an experiment to reduce tax fraud through machine learning as a way to increase revenue available for development.

In Brazil: J-PAL Latin America and the Caribbean collaborated with the Institute of Public Safety in the state of Rio de Janeiro to create a unified map-based platform to improve the effectiveness of police deployment to prevent crime. Data sets on crime, police presence, and environmental factors informed an improved system for optimal and efficient police response, helping to make communities safer.

In South Africa: J-PAL Africa is creating a long-term partnership with the City of Cape Town that involves a data access portal allowing researchers to use administrative records. So far, ideas for experiments have included research on electricity metering and water use during the 2018 drought to optimize use of scarce resources during critical periods of use.

In each of the above examples we draw insights from research using large administrative data sets—insights with important implications for social policy that may not have been discovered without access to and proficiency in the use of administrative data for experiments.

We believe this data can—and should—be used far more widely and creatively to improve lives than it currently is, and IDEA hopes to overcome some of the barriers described above to help make that happen. 

Following initial data exploration, IDEA aims to support the design, analysis, and evaluation of interventions that can address social problems.

For instance, once we can observe water consumption and understand use patterns during a drought, the next step might be to evaluate the impact of strategies that can reduce water use without disproportionately burdening poor families. Randomized evaluations or experiments can provide key insights to policymakers to tackle these kinds of problems and help design the scale up of successful programs.

This initiative will draw on the deep expertise of J-PAL’s affiliate network and staff in running and synthesizing the results of randomized evaluations. We will also make use of the local knowledge of our regional staff and strong partnerships with implementing organizations and government bodies.

Experiments that use administrative data can be:

Faster: Time-consuming surveys (both pre- and post-intervention) can often be replaced by existing administrative data sets.

Bigger: Administrative data sets often cover entire cities, states, or even countries allowing representative sampling and the simultaneous testing of multiple treatment arms—it would be infeasible for individual researchers to collect data manually from this number of people.

Less expensive: Data collection is often extremely expensive, but such costs can be mitigated by taking advantage of data sets that already exist. 

More precise: Impact estimates can be much more precise through the use of bigger sample sizes often included in administrative data sets.

More accurate: Attrition is less of a problem when using administrative data sets than when attempting follow-up in person, resulting in more accurate results.

The IDEA Initiative launched this month, seeded with startup support from the Alfred P. Sloan Foundation. IDEA will be led by J-PAL’s co-chair of Research, Education, and Training, Shawn Cole (HBS); Executive Director Iqbal Dhaliwal; Director of Research, Education, and Training Anja Sautmann; and co-chair Lars Vilhuber (Cornell). 

Initiative staff will include Sam Friedlander and additional positions for which we are currently hiring.

Later this year IDEA will convene a group of experts on the use of administrative data for experiments and decision-making. These experts will contribute to a new Handbook of Administrative Data Access.

The handbook will provide general guidance and case studies demonstrating the use of administrative data for research, experiments, and evidence-based decisions. This includes defining best practices, ensuring that research conforms to local regulations, streamlining processes, and assisting with knowledge transfer. It will be widely available for use by the public.

As IDEA continues to grow we are seeking funding for an expanded set of activities, including:

Setting up IDEA Labs at our regional offices to work with data providers who want to better analyze their data and use it for experiments to evaluate innovative programs.

Issuing competitive RFPs that will provide funding to researchers interested in using administrative data for experiments (for more information, see the IDEA website).

Conducting policy outreach to promote wider access to and use of administrative data in experiments.

At J-PAL, we believe in the power of evidence-based policy to work toward alleviating poverty worldwide—and we believe in the power of administrative data to help us get there. A vital step toward reducing poverty through policy is taking advantage of all that administrative data has to offer.

We are excited to begin to do so more systematically in the coming months and years, and we hope this will be the beginning of a trend toward more instances of existing data sets and experiments informing social policy around the world.

IDEA’s first pilot RFP will be issued in 2020 and will focus on the United States. Stay tuned for information about additional RFPs, projects coming out of IDEA Labs, and more.

For more information about IDEA, including opportunities to support this work, please contact Sam Friedlander at [email protected].