Achieving Academic Success After School: A Randomized Evaluation of the Higher Achievement Program
- Students
- Empowerment
- Student learning
- Training
Policy issue
Improving the quality of educational instruction has the potential to increase children’s test scores, graduation rates, and subsequent earnings, particularly for disadvantaged and minority children. However, many children living in poor neighborhoods attend under-funded, low-quality public schools. While improving these schools is a desirable goal, the complexity of urban school reform and the slow nature of such change have compelled many policymakers and parents to consider educational interventions outside of the school system, such as out-of-school (OST) academic enrichment programs. Academically focused OST programs, in particular, aim to improve students’ academic attitudes, behaviors and performance by increasing youths’ access to high-quality academic supports and opportunities. But these programs range widely in their focus, structure and intensity—variations that likely affect their ability to make such improvements. Indeed, the other rigorous evaluations of after school programs have not found strong positive impacts on academics.
This multi-year study focuses on investigating the impact of a “Cadillac” version of academic enrichment OST programs by analyzing the effects of the Higher Achievement Program, a mature OST program with a rigorous and intensive academic curriculum. If this comprehensive program fails to generate long-term improvements in academic outcomes, then OST programs in general may not be capable of doing so, but if it succeeds, then it will demonstrate that such programs can have lasting impacts on students’ lives.
Context of the evaluation
Washington D.C. is an ethnically diverse, midsized city. The D.C. public school system struggles with a number of structural and functional problems and as a result has failed to adequately educate numerous students. With 59% of students receiving a high school diploma in 2001, Washington D.C. held the 4th lowest graduation rate in the nation1. D.C., like other struggling school systems in the country, has been experimenting with reforms; closing under-populated schools, fixing up decrepit buildings, sweeping out underperforming principals and hiring new young teachers. Yet if school reforms are ultimately effective, it may be a number of years before changes in student learning are seen. OST academic enrichment programs such as the Higher Achievement Program may supplement student learning in the meantime.
Details of the intervention
Higher Achievement is an established OST program that strives to develop positive academic behaviors and attitudes in academically motivated but underserved fifth- through eighth-grade youth. Its objective is to improve participants’ grades, standardized test scores, and school attendance in order to increase their educational opportunities, specifically through acceptance and scholarships to attend private, parochial and public magnet high school programs.
The program is extremely time-intensive, offering approximately 650 hours a year of academic instruction, enrichment activities and mentoring during the after-school and summer hours. During the school year, participants, or “scholars”, attend the “After-School Academy” three days a week. This 25-week program, which runs from 3:30 to 8:00 p.m., includes homework help, dinner, an elective, a 15-minute group meeting and two hours of academic instruction in small groups of about two to three scholars—one day a week in mathematics, one day in literature and one in technology. Volunteer mentors lead these groups. Scholars participate in monthly field trips and community service projects. During the summer, the six-week “Summer Academy” operates from 8:00 a.m. to 4:00 p.m., five days a week. Students attend four classes a day, taught by trained faculty, in mathematics, science, social studies and literature as well as two electives. Scholars take weekly field trips and participate in a three-day university trip during which they experience college life by attending classes, sleeping in dorms, going to lectures and eating in dining halls. In both the After-School and Summer Academies, students receive high school placement services.
The intensity of the program and HA’s strict admission policies screen for motivated parents and what the program defines as “academically motivated” students. While academic performance is not a criteria for admission to the program, it is of course possible that such families may succeed academically even without access to the HA program. To identify the effects of attending HA above and beyond what these families would have achieved without the program, we built the study around an oversubscription randomized controlled trial.
During recruitment for the study, Higher Achievement recruited more youths than were required to fill the available positions; slots were then randomly allocated among eligible students. Students were evaluated using a short version of the Stanford Achievement Test and a student survey containing psychometric scales designed to assess the effects of the program on students’ perceptions of themselves, their peers, and the options available to them in the future. Parents also complete brief surveys.
Results and policy lessons
Test Scores: The program had little effect in the first year, but by the second year, student scores had improved by 0.09 standard deviations in reading comprehension and 0.12 standard deviations in problem solving.
Academic Attitudes: Surprisingly, rather than improving students’ perceptions of their academic abilities, the program seems to have negatively impacted students’ academic attitudes in the first year. However, by the second year, there are no differences between the treatment and comparison students’ attitudes. These changes in attitude may have been driven by the transition into a more competitive environment. Students entering the Higher Achievement program in fifth grade experience a decline in academic confidence upon entrance. Students in the comparison group have relatively higher confidence as fifth graders, but then experience a similar decline as they make the transition to middle school in sixth grade.
Behavior: Participation in the program increases the rate at which children report engaging in many forms of misconduct. For example, the probability of taking something that did not belong to the child increases by 14 percentage points in the first year and 19 in the second year. Though this could reflect an increase in self-reporting, rather than an increase in actual misconduct, this is clearly an important area for future investigation.
High School Preparation: Students in the program were more likely to visit a high school, get information about a high school, talk with non-parental adults and peers about high schools, and decide where to apply.
Change over the summer: The youth in the program academic skills do not increase further rather to the comparison group over second or three summers. Treatment youth’s high school preferences, however, do show a marked change. In the spring, 9 percentage points more students in the treatment group express a desire to attend a competitive area high school compared to the control group. In the fall, this increases to 17 percentage points; and 10 percentage points fewer students express a desire to attend their local public school.
1 Manhattan Institute for Policy Research (MI), “Civic Report - High School Graduation Rates in the United States,” https://www.manhattan-institute.org/html/cr_baeo_t2.htm.