Guiding College Students’ AI use Improves Learning: Experimental Evidence from Chile

Notably, AI use is accelerating much faster than researchers are accumulating rigorous evidence on its impact. In the meantime, universities are faced with deciding whether and how to restrict, regulate, or integrate AI tools. I recently conducted a study in Chile to address two important questions: Can universities influence whether and how students use AI tools? And does that have causal effects on student perceptions and learning outcomes? 

The study designed and deployed a course-specific AI assistant (called GPT-U.AI) for undergraduate econometrics students at a selective university in Chile. I implemented two randomized interventions to assess it's effects on adoption, usage quality, and learning outcomes.

The key finding: Student adoption of AI is not just about access, but also how institutions guide its use.

Encouraging AI adoption increased usage but not academic performance

The first intervention randomly encouraged students to adopt the AI tool before a midterm exam. Students in the treatment group received three emails introducing GPT-U.AI, showing how it could be used, and sharing a short instructional video. The messages invited students to ask questions about concepts and use the tool for review before the exam. Students in the comparison group had access to the tool but did not receive the instructional messages.

The intervention increased awareness and adoption of the tool. Students assigned to treatment were more likely to know about GPT-U.AI and to report using it. Reported usage intensity also increased substantially. 

However, these gains in adoption did not translate into improvements in academic performance in the midterm exam. One potential mechanism behind these null effects is that students may not have perceived GPT-UAI as particularly useful, especially at a time when many were still experimenting with AI tools. I asked the students about their perceptions of the tool’s value and found no differences by treatment status. Therefore, the intervention increased awareness and usage but did not increase perceived value, which is consistent with no effects on learning. Many students may have viewed GPT-U.AI as simply one more study input among several existing resources.

The findings also reflect a broader reality in higher education today: students already have widespread access to generative AI tools. In this context, increasing exposure alone may not substantially change behavior or learning.

Guidance on how to use AI improved final exam performance

The second intervention randomly shifted attention from adoption to usage quality. Before the final exam, students in the treatment group received three additional emails encouraging “learning-oriented” use of GPT-U.AI. Rather than simply recommending the tool, the messages instructed students to use it as a tutor.

The guidance emails included prompts asking the GPT to:

• act as an econometrics tutor;

• guide students through concepts step-by-step instead of giving final answers immediately;

• generate practice questions and provide feedback; and

• help students verify assumptions and identify mistakes in their reasoning.

In this case, the results were quite different. Students assigned to this intervention improved their final exam performance by 0.22 standard deviations. The gains were concentrated in open-response sections requiring structured reasoning and problem solving, rather than multiple-choice questions. This distinction is important because the intervention emphasized interactive reasoning processes rather than memorization or answer retrieval. The study suggests even larger effects among students whose exposure to the guidance was driven by the intervention. 

The study also documents important changes in how students interacted with the AI assistant. Students exposed to the guidance intervention were substantially more likely to use GPT-U.AI in “tutor mode,” engaging in step-by-step interaction rather than simply requesting direct answers. The intervention also increased the perceived usefulness of the assistant by 0.38 standard deviations.

Importantly, the intervention did not increase overall use of generic AI tools such as ChatGPT, Gemini, or Claude, which were already nearly universal among students. Rather, the intervention changed how students used a course-aligned AI assistant.

What do these findings imply for universities?

While many schools may focus exclusively on restricting or preventing student use of AI, these efforts may miss an important opportunity to support learning: rather than substituting for cognitive effort, AI may complement structured reasoning. But to achieve that, education institutions should consider how AI is offered to students.

In the Chile study, the interventions themselves were relatively low-cost. The university did not require specialized infrastructure, paid licenses, or major curricular redesigns. The guidance treatment consisted primarily of structured prompts delivered via the existing course platform and standard email. This makes the findings especially relevant for universities attempting to respond quickly to the rise of generative AI use while operating under limited budgets and institutional constraints.

My findings contribute to a rapidly growing literature examining the role of generative AI in education. The evaluation in Chile suggests that institutional guidance may play a central role in determining whether AI tools support or hinder acquisition of skills and knowledge. However, the results may not generalize automatically to other contexts, disciplines, or student populations. Therefore, we need more evidence to understand how AI guidance interacts with different learning environments, pedagogical models, and assessment structures.

As universities around the world continue adapting to the arrival of large language models, the evidence from Chile points toward a simple but important lesson: access alone may not be enough. The educational value of AI may depend critically on whether institutions help students learn how to use it well.

This blog post is based on a working paper, "Guidance Over Adoption: Experimental Evidence on AI-Assisted Learning in Higher Education,” by Sebastian Gallegos, associate professor of economics at Universidad Adolfo Ibañez and invited researcher at J-PAL.