The challenge of managing school systems remotely arises in times of disasters, war, and health shocks. Online courses for students were the solution during school closures in many countries due to the COVID-19 pandemic. A key issue has been how to ensure student participation and learning outcomes. Unlike voluntary learners in massive open online courses, school education includes a larger set of actors. Thus, we wanted to know on which level lightweight behavioral interventions are effective: Student? Teacher? System?  Nowadays, rapid testing of generative artificial intelligence (AI) in education further emphasizes the need to understand the role of digital systems in education and the role of behavioral interventions in digital education on different levels.

To improve the participation of high school students in online courses on the national scale in Ecuador we tested lightweight, behavioral science-motivated interventions at three levels: the system level, the teacher level, and the student level. The student-level treatment included a lottery ticket with cash prizes for each lesson completed, encouraging messages, and asking students to plan their study process. The teacher-level treatment consisted of performance feedback of their classes compared to other classes of the same course type, reminder messages, and encouragement emails that included a video showing the experiences of students and teachers who had previously completed the course.

The most effective intervention at the school system level consisted of a real-time centralized management system: Ministry of Education staff had access to and received weekly reports on attendance at each treated school through this system. We built this intervention on a set of insights from behavioral economics:

• Reduced hassle. Ministry personnel, e.g., heads of the educational zones and personnel of central office, could monitor the study activity on the fly by simply entering the online platform.
• Real-time feedback. Ministry personnel could utilize this information to nudge technical staff and teachers whose students are falling behind in an agile manner using ministry channels, e.g., official social networks.
• Reminders. For example, ministry personnel reminded teachers that this course was a part of their responsibilities they were paid for.
• The salience of monitoring. Teachers could observe on the platform that Ministry personnel are enrolled in each class’s monitoring dashboard.

We tested in randomized control trials rapid behavioral interventions at the student, teacher, and system level on 45,000 Ecuadorian high school students’ performance in online learning during one year (from December 2019 to December 2020). A low-cost (less than 60 cents per student) centralized online management system of schools showed the largest impact.

Students covered by the centralized online management system accessible to Ministry personnel improved overall subject knowledge by 0.13 SD relative to self-management schools, in which only teachers receive information from the management system about their class (Figure 1). To put it in perspective, this is equivalent to how much students learn in ten months of business as usual.

Figure 1: Impact of Centralized Management on overall subjects’ knowledge test scores (p<.05)

Centrally monitored students had more regular study patterns (Figure 2) that explained better educational outcomes among them. Centrally monitored students had 0.21 SD more learning time on the online learning platform already on week 8, which means an average of 125 minutes more time for online learning and 2.1 more completed lessons two weeks before the end of the program. Students in self-managed schools had to exert additional effort to catch up with the centrally monitored students during the last weeks of the educational program.

Figure 2: Student study patterns by groups

These results highlight the need to develop behavioral interventions beyond the student level to improve participation in online learning.

Inspired to implement this design in your own work? Here are some things to think about before you get started:

• Are the behavioral drivers to the problem you are trying to solve similar to the ones described in the challenge section of this project?
• Is it feasible to adapt the design to address your problem?
• Could there be structural barriers at play that might keep the design from having the desired effect?
• Finally, we encourage you to make sure you monitor, test and take steps to iterate on designs often when either adapting them to a new context or scaling up to make sure they’re effective.

Additionally, consider the following insights from the design’s researchers:

• The system-level approach allows various active actors to improve educational outcomes. Implementers should consider the various actors in education, e.g., heads of the educational zones, personnel of central office, to achieve better student outcomes.
• The advantage of the centralized management system was providing information on the fly and weekly to the ministry personnel on a larger number of metrics (number of enrolled students, number of lessons completed, students’ learning activity e.g. time spent on educational material on the platform) by student, class, and educational zones.