J-PAL State and Local Innovation Initiative – Q&A
Who can apply to this competition?
Phase I of the competition, which provides support to selected jurisdictions to design a randomized evaluation, is open to representatives of U.S. state, county, municipal, and tribal governments. Representatives of public school systems or other state or local government organizations are also welcome to apply.
In reviewing Letters of Interest, J-PAL North America looks for a demonstrated commitment from senior leaders within the government to use evidence to inform decision-making. In addition to providing the names and titles of the main contacts for the application, applicants are encouraged to provide a letter of support from the Governor, Mayor, City Manager, County Commissioner or Supervisor, or other relevant government leader.
What kind of support will selected state and local governments receive?
State and local governments selected in Phase I of the competition receive up to one year of ongoing support from J-PAL North America staff, who will provide project management, connect them to other research resources and technical experts at J-PAL, and facilitate partnerships with J-PAL’s network of leading academic researchers. Selected state and local governments can also access up to $100,000 in flexible pilot funds to help get evaluations off the ground.
In Phase II, state and local governments that have partnered with a researcher from J-PAL’s network to design a high-quality randomized evaluation can apply for funding, typically in the range of $250,000-500,000, to implement the evaluation.
Over the course of the five-year initiative, selected state and local governments will have the opportunity to participate in custom trainings and convenings to showcase them as leaders in creating and using evidence to inform policy, to facilitate lasting connections between policymakers and researchers, and to help advance the spread of innovative ideas.
When are applications for this round of the competition due?
The deadline to submit letters of interest for the current round of the competition is March 2, 2018. Read the instructions for submitting letters of interest here.
How many state or local governments will be selected?
Five state and local governments were selected in the first round of the competition: Pennsylvania; Puerto Rico; South Carolina; the City of Philadelphia, Pennsylvania; and the City of Rochester, New York. Three state and local governments were selected in the second round of the competition: the City of Baltimore, Maryland; King County, Washington; and Santa Clara County, California. Read more about the projects here.
Can a single jurisdiction submit applications from more than one agency or department?
Multiple agencies or departments from the same jurisdiction may apply, either together or separately. Where feasible, we encourage agencies or departments from the same jurisdiction to apply together and to describe either (a) how they would coordinate on a single project or (b) how they would use the resources available through the competition to support multiple projects.
Can a service provider apply?
Only state or local governments are eligible to apply. However, state or local governments may identify a service provider or other organization that would serve as a partner in delivering the program or policy they are interested in testing.
Does my proposal have to involve a randomized evaluation?
In short, yes. We understand that there are many ways to generate evidence to inform policy and that randomized evaluations are not always feasible or optimal. However, the specific focus of this initiative is to support policymaker and researcher partnerships to design and implement randomized evaluations. Thus, applicants to the competition must be willing to consider incorporating some element of random assignment into the program or policy they are interested in testing. Selected state and local governments will receive technical support to help them determine the number of people (or schools, hospitals, neighborhoods, etc.) who could potentially participate and a plan for fairly allocating services among them.
Is this initiative focused on particular policy areas?
This initiative is open to a wide range of policy areas, including but not limited to children and families, crime, education, employment, energy and environment, health, household finance, housing and homelessness, and governance.
What is J-PAL North America looking for in a letter of interest?
J-PAL North America will consider the following general criteria in reviewing letters of interest:
- Clear description of an important policy question or challenge
- Promising approach to addressing the policy challenge
- Feasible opportunity for a randomized evaluation
- Commitment to using evidence to inform decision-making
- Potential contribution to the evidence base
For more details, please see our Request for Letters of Interest.
Can I submit a letter of interest with multiple programs that I’m interested in evaluating?
Yes. J-PAL North America staff will support state and local governments in determining which programs are most suitable for a randomized evaluation.
Is it possible to conduct a randomized evaluation of an entitlement program?
If not everyone who is eligible participates in an entitlement program, it may be possible to evaluate the impact of the program by randomly encouraging some individuals to participate. It may also be possible to use a randomized evaluation to study changes to the program’s eligibility requirements (e.g. randomly offering the program to some individuals who would not be eligible under current rules) or benefits (e.g. randomly offering some individuals a more generous version of the program or the program combined with some other benefit or service).
Is it possible to use this initiative to support a retrospective evaluation of a program?
Probably not. Randomization needs to be incorporated from the start; participation in a program cannot be randomly assigned after the fact. Retrospective analysis may be possible for programs that already implemented random assignment (e.g., programs in which a lottery was used to determine who would participate).
What day-to-day activities might I expect if selected through this initiative?
J-PAL staff provide project management and technical support to help government partners and researchers move the proposed evaluation from concept to design.
J-PAL North America staff:
- Arrange calls and meetings between government partners and researchers in J-PAL’s network to develop a research question of mutual interest.
- Prioritize the key questions for assessing the feasibility of the evaluation.
- Arrange site visits for J-PAL North America staff and researchers to gather information about the jurisdiction’s priorities, the program or policy of interest, and the local context.
- Develop an overall project timeline and set agendas for weekly or bi-weekly calls to discuss progress and next steps.
- Communicate different options for random assignment developed by researchers through calls, memos, and presentations.
- Answer questions from J-PAL North America staff and researchers and provide regular updates on the program’s implementation, enrollment procedures, and any planned program changes.
- Introduce J-PAL North America staff and researchers to key contacts, such as the relevant data management staff, service providers, and senior-level decision makers.
- Define the target population and eligibility criteria for the program or policy.
- Assess service provider buy-in and capacity to implement an evaluation.
- Identify potential sources of administrative data for measuring key outcomes and provide aggregate or de-identified historical program data.
- Develop the research question and procedures for enrollment, consent, and randomization.
- Assess prior evidence, identify knowledge gaps, and specify outcomes of interest.
- Determine appropriate sample size to detect meaningful impacts.
- Assess data quality, analyze historical data, and draft data sharing requests and agreements.
- Obtain Institutional Review Board approval.
- Develop grant proposal for full evaluation funding.
What are common uses of the pilot funding?
Jurisdictions have proposed to use or used the flexible pilot funds in many ways, including:
- Personnel costs for existing program staff, project managers, and data analysts who will contribute to the evaluation. Jurisdictions sometimes budget a small portion of time for a more senior decision maker, who can provide momentum on the project, and a large portion of staff time for a day-to-day project manager, who allocates a significant percentage of their time to the project.
- Personnel costs for temporary staff or consultants hired for specific purposes (for example, program enrollment specialists who will help service providers implement the random assignment or a short-term research assistant to link administrative datasets).
- Travel and lodging for research partners to conduct site visits.
- Costs from administrative data providers of matching individual records to outcomes data.
- Costs to develop and field test survey instruments.
- Costs for enumerators to conduct surveys or interviews.
- Compensation for participants to complete surveys, focus groups, or interviews.
- Laptops or tablets to use for survey data collection and user fees for survey data collection software.
In some cases, pilot funding may be sufficient to pay for a very low-cost service (e.g., sending letters). In most cases, however, we anticipate that governments will use traditional sources to pay for the program or policy they wish to test, and the pilot funding available through this initiative will be used to pay for the additional costs associated with designing an evaluation.
If a jurisdiction is selected during Phase I of the initiative, does it automatically receive Phase II funding?
Jurisdictions selected in Phase I that have partnered with a researcher from J-PAL’s network to design a proposal for a high-quality and feasible randomized evaluation are eligible to apply for Phase II funding, and we expect that the likelihood of these proposals being funded in Phase II will be very high. We also understand that there are many reasons why a state or local government selected in Phase I might be unable to proceed to Phase II (e.g., resources become available to serve everyone who is eligible; randomization proves to be infeasible after a pilot period of implementation; analysis reveals that the sample size is too small to detect a meaningful effect).
What is the timeline for evaluations to begin and end?
The timelines for evaluations are flexible, but we expect that most evaluations will begin within a year of the jurisdiction being selected in Phase I of the competition. End dates will depend on the specific details of the program or policy being evaluated.
Do interested jurisdictions need to partner with academic researchers before applying?
No. As part of the State and Local Innovation Initiative, J-PAL North America will connect selected jurisdictions with academic researchers from J-PAL’s network.
Do the researchers have to be J-PAL affiliates?
In order to apply for funding in Phase II, the proposed research team must include at least one J-PAL affiliate or invited researcher. As part of the assistance offered through this initiative, J-PAL North America will help connect selected jurisdictions with academic researchers in J-PAL’s network.
Who are the researchers participating in this initiative?
J-PAL is a network of affiliated professors, who work at more than 40 universities across the world. J-PAL’s affiliated professors are united by their use of randomized evaluations to test and improve the effectiveness of programs and policies aimed at reducing poverty. You can find a current list of J-PAL’s affiliated professors here.
Is the funding available through this initiative sufficient to cover the costs of an evaluation?
If an evaluation makes use of administrative data that is already being collected, we anticipate that the funding available will be sufficient to cover the full evaluation cost. Evaluations that rely on data that can only be collected through surveys will likely be more expensive and require additional sources of funding.
Should government agencies that focus on health care delivery apply to this competition or to the J-PAL Health Care Delivery Innovation Competition?
State and local government agencies focused on health care delivery may be eligible for both the J-PAL Health Care Delivery Innovation competition and this competition. If an applicant is not sure which competition is the right fit, we encourage them to contact Anna Spier ([email protected]) for help determining which competition is most appropriate..
Who is funding the initiative?
The Laura and John Arnold Foundation (LJAF) is providing funding for the initiative. LJAF is a private foundation that is working to address our nation’s most pressing and persistent challenges using evidence-based, multi-disciplinary approaches. LJAF has offices in Houston, New York City, and Washington, DC.
Whom should I contact with questions?
For questions about the State and Local Innovation Initiative, please contact Initiative Manager Julia Chabrier at [email protected].