PublicDisclosureAuthorizedPublicDisclosureAuthorized

PolicyResearchWorkingPaper11125

FromChalkboardstoChatbots

EvaluatingtheImpactofGenerativeAIonLearningOutcomesinNigeria

MartínDeSimone

FedericoTiberti

MariaBarronRodriguez

FedericoManolio

WuraolaMosuro

EliotJolomiDikoru

WORLDBANKGROUP

EducationGlobalDepartmentMay2025

PolicyResearchWorkingPaper11125

Abstract

ThisstudyevaluatestheimpactofaprogramleveraginglargelanguagemodelsforvirtualtutoringinsecondaryeducationinNigeria.Usingarandomizedcontrolledtrial,theprogramdeployedMicrosoftCopilot(poweredbyGPT-4)tosupportfirst-yearseniorsecondarystudentsinEnglishlanguagelearningoversixweeks.Theinterventiondemonstratedasignificantimprovementof0.31standarddeviationonanassessmentthatincludedEnglishtopicsalignedwiththeNigeriancurriculum,knowledgeofartifi-cialintelligenceanddigitalskills.TheeffectonEnglish,themainoutcomeofinterest,wasof0.23standarddeviations.

Cost-effectivenessanalysisrevealedsubstantiallearninggains,equatingto1.5to2yearsof’business-as-usual’schooling,situatingtheinterventionamongsomeofthemostcost-effectiveprogramstoimprovelearningoutcomes.Ananalysisofheterogeneouseffectsshowsthatwhiletheprogrambenefitsstudentsacrossthebaselineabilitydis-tribution,thelargesteffectsareforfemalestudents,andthosewithhigherinitialacademicperformance.Thefind-ingshighlightthatartificialintelligence-poweredtutoring,whendesignedandusedproperly,canhavetransformativeimpactsintheeducationsectorinlow-resourcesettings.

ThispaperisaproductoftheEducationGlobalDepartment.ItispartofalargereffortbytheWorldBanktoprovideopenaccesstoitsresearchandmakeacontributiontodevelopmentpolicydiscussionsaroundtheworld.PolicyResearchWorkingPapersarealsopostedontheWebat

/prwp.Theauthorsmaybecontactedat

desimone@

,ftiberti@,mbarronrodriguez@,fmanolio@,wmosuro@,andedikoru@.

ThePolicyResearchWorkingPaperSeriesdisseminatesthefindingsofworkinprogresstoencouragetheexchangeofideasaboutdevelopmentissues.Anobjectiveoftheseriesistogetthefindingsoutquickly,evenifthepresentationsarelessthanfullypolished.Thepaperscarrythenamesoftheauthorsandshouldbecitedaccordingly.Thefindings,interpretations,andconclusionsexpressedinthispaperareentirelythoseoftheauthors.TheydonotnecessarilyrepresenttheviewsoftheInternationalBankforReconstructionandDevelopment/WorldBankanditsaffiliatedorganizations,orthoseoftheExecutiveDirectorsoftheWorldBankorthegovernmentstheyrepresent.

ProducedbytheResearchSupportTeam

FromChalkboardstoChatbots:Evaluatingthe

ImpactofGenerativeAIonLearningOutcomesin

Nigeria*

MartnDeSimone,FedericoTiberti,MariaBarronRodriguez,FedericoManolio,WuraolaMosuro,EliotJolomiDikoru?

Keywords:large-languagemodels,adaptivelearning,artificialintelligence,educa-tiontechnology,secondaryeducation,teachingattherightlevel.

JELClassification:C93,I21,J24,O15,O33.

*TheteamwouldliketothankScherezadLatifandHalilDundar,EducationPracticeManagers,WorldBank.TheteamextendsitsappreciationtoDr.JoanOsaOviaweandJenniferAisuan,fortheircollaborationthroughouttheimplementationofthepilot,aswellasAlexTwinomugisha,RobertHawkins,andCristbalCobofortheirsupportwiththeintervention.Theteamthanksthosewhoprovidedcommentstoapreviousversionofthispaper,includingDavidEvans,HalseyRogers,CarolinaLopez,FranciscoHaimovich,DanielRodriguez-Segura,NoahYarrow,JuanBarn,andLucasGortazar.TheteamacknowledgesthefinancialsupportreceivedfromtheMastercardFoundation.

?DeSimone:TheWorldBank.E-mail:mdesimone@.Tiberti:TheWorldBank.E-mail:ftiberti@.Barron:TheWorldBank.E-mail:mbarronrodriguez@.Manolio:TheWorldBank.E-mail:fmanolio@.Mosuro:TheWorldBank.E-mail:wmo-suro@.Dikoru:TheWorldBank.E-mail:edikoru@.

2

1Introduction

Theglobaleducationsectorisgrapplingwithalearningcrisis.AccordingtotheLearningPovertyIndex,approximately70%of10-year-oldsinlow-andmiddle-incomecountriescannotreadandunderstandanage-appropriatetext(WorldBank,2022).Thesedeficitsinlearningaccumulateandbecomeparticularlyacuteatthesecondaryschoollevel,asevidencedbynumerousinternational,regional,andnationalassessments.

Inhisseminal1984study,Bloomdemonstratedthatstudentsreceivingone-on-onetu-toringoutperformedtheirpeersintraditionalclassroomsettingsbyanaverageoftwostandarddeviations(Bloom,1984).Subsequentstudieshaveconsistentlyconfirmedthesignificantbenefitsofone-on-onetutoring(Nickowetal.,2020).Thechallenge,however,isthatimplementingone-on-onetutoringatscaleiscostlyandunaffordableformosted-ucationsystems.Bloomreferredtothischallengeasthe“two-sigmaproblem”:howtoreplicatethegainsofpersonalizedtutoringatscaleinacost-effectivemanner.

Thispaperexamineswhethergenerativeartificialintelligence,specificallylargelanguagemodels(LLMs),canhelpsolvethatproblem.Weevaluateasix-weekafter-schooltutoringprograminNigeriathatusedapubliclyavailableLLM(ChatGPT-4)tosupportstudentsinlearningEnglish.First-yearsecondarystudentsfromninepublicschoolsinBeninCitywereinvitedtoparticipate;fromthispool,52%ofeligiblestudentsexpressedinterest,andparticipantswererandomlyselectedfromamongthem.Thoseassignedtotheinter-ventionattendedtwelve90-minutesessionsincomputerlabs,engagingincurriculum-alignedactivitiesguidedbyteachers.Weusearandomizedcontrolledtrial(RCT)designtoestimatethecausalimpactoftheprogramonlearningoutcomes.

Wepresentthreemainsetsofresults.First,weshowthatstudentsselectedtoparticipateintheprogramscore0.31standarddeviationhigherinthefinalassessmentthatwasdeliv-eredattheendoftheintervention.Wefindstrongstatistically-significantintent-to-treat(ITT)effectsonallsectionsofthatassessment:Englishskills(whichincludedthemajorityofquestions,0.24σ),digitalskills(0.14σ),AIskills(0.31σ)andanItemResponseTheory(IRT)compositescoreofeachstudent’sexam(0.26σ).WealsoshowthattheinterventionyieldedstrongpositiveresultsontheregularEnglishcurricularexamofthethirdterm.Thisresultisimportantbecausethecontentevaluatedinthatexamwasbroaderthantheonecoveredduringthesixweeksoftheinterventionandincludedthecontentoftheen-tireyear.WecalculateanITTeffectofbeingselectedfortheprogramontheperformanceinthethird-termexamof0.21standarddeviations.

Second,weexamineheterogeneityoftheeffectsbycertainpre-treatmentcharacteristics.

3

Treatmenteffectswerepositiveandstatisticallysignificantacrossalllevelsofbaselineper-formance,butstrongeramongstudentswithbetterpriorperformance.Similarly,treat-menteffectswerepositiveandstatisticallysignificantovertheentiredistributionofaproxyforsocioeconomicstatus,butstrongeramongstudentswithahigherone.Lastly,treatmenteffectswerestrongeramongfemalestudents,compensatingforadeficitintheirbaselineperformance.

Third,weconductdose-responseanalysis.WeestimateLocalAverageTreatmentEffect(LATE)estimates,focusingontheimpactofactualattendancetotheinterventionsessions,whichaveraged72%amongthetreatmentgroup.Usingattendancedata,weestimateadose-responserelationship,findingastronglinearassociationbetweendaysattendedandimprovedlearningoutcomes,withaneffectsizeofapproximately0.031standarddeviationperadditionaldayofattendance.Furtheranalysispredictssubstantialgainswithextendedprogramduration,estimatinganincreaseofbetween1.2and2.2standarddeviationsforafullacademicyearofparticipation,dependingonattendancerates.

Thefindings,combinedwithacostanalysis,seemtoindicatethattheprogramwashighlycost-effective.Thesix-weekpilotgeneratedlearninggainsthattakebetween1.5and2yearsinabusiness-as-usualscenario.Theprogramachieved3.2equivalentyearsofschooling(EYOS)per$100invested,surpassingmanycomparableinterventions.UsingLearning-adjustedyearsofschooling(LAYS)asthemetricfortheanalysis,theprogramgeneratesupto0.9yearofhigh-performanceeducation.Whenbenchmarkedagainstevidencefrombothlow-andmiddle-incomecountries,thepilotprogramranksamongthemostcost-effectivesolutionsforaddressinglearningcrises.

Ourstudycontributestodifferentstrandsoftheliteraturethataimtoidentifytheef-fectofprogramsthattrytocustomizeinstructiontothelevelofstudents,bothwithandwithouttechnology.Effortstoaddressthischallengehaveincludedthedevelopmentofthe”TeachingattheRightLevel”(TaRL)approach,whichhasshowntoimprovelearningoutcomesincontextssuchasIndia,Kenya,Ghana,andZambia(Banerjeeetal.,2016).Im-plementationmodalitiesofTaRLhavevaried,rangingfrompullingstudentsoutofclass(Banerjeeetal.,2007),trackingclassrooms(Dufloetal.,2011),providingextrainstruc-tionaltimeoutsideofschool(Banerjeeetal.,2016),andemployingvolunteersinsteadofteachers(Banerjeeetal.,2008).However,scalingTaRLprogramsremainsdifficultduetotheirlabor-intensivenature.Thischallengeisparticularlypronouncedgiventheglobalshortageofteachers,whichisparticularlypronouncedinSub-SaharanAfrica.Recentestimatessuggestthatby2040,countriesintheregionwillneed21%moresecondaryschoolteachersperyear(EvansandMendezAcosta,forthcoming).Teachershortagesare

4

furthercompoundedbyhighattritionrates,andtheneedforspecializedknowledgeatthesecondarylevelmakesTaRLprogramsevenmoredifficulttoimplement.

Inrecentyears,adaptivelearningsoftwarehasemergedasapotentialsolutiontothescal-abilityoftutoringprogramsbyusingtechnologytomimicone-on-onetutoring.Evidencesuggeststhatcomputer-adaptivelearningsystemscanimprovelearningoutcomes.Forexample,astudyofpersonalized,technology-aidedafter-schoolinstructionformiddleschoolstudentsinIndiareportedgainsof0.37standarddeviationinmathand0.23stan-darddeviationinHindiovera4.5-monthperiod(Muralidharanetal.,2019).AstudyinCambodiathatfocusedonmathinstructionforprimaryschoolstudentsfoundimpactsoncognitiveskillsduetostudents’increasedlearningproductivityperhour(Itoetal.,2021).InElSalvador,theuseofsoftwareforadaptivelearningprovedeffectiveinanen-vironmentwithheterogeneousclassesandpoorlyqualifiedteachers(Bcheletal.,2022).ExperimentsinChinahavealsofoundpositiveeffectsonstandardizedmathscores(Laietal.,2015a)andonMandarin(Laietal.,2015b),includingwhenimplementedduringregularschoolhours(Moetal.,2014).InEcuador,thepossibilitytouseanadaptive-learningsoftwarefor4monthsledtolargepositiveimpactonstandardizedtestscoresinmath(Angel-Urdinolaetal.,2023).Otherstudiesthatdonotfollowexperimentalap-proacheshavealsoestimatedpositiveeffectsofsimilarsoftwareprograms,suchasapro-graminUruguaythatshowedgainsof0.2standarddeviationonmathematicstestscores(PereraandAboal,2019).

Despitethesesuccesses,adaptivelearningprogramsfaceseveralchallenges.First,mostarenotdeployedintheworld’smostchallengingeducationalcontexts,particularlyinSub-SaharanAfrica,raisingquestionsaboutexternalvalidity.Second,theseprogramsoftenrelyonproprietarysoftware,whichtypicallyinvolvesbothfixedandper-studentcosts,makingthemdifficulttoscaleinresource-constrainedenvironments.

Someadaptive-learningoptionsaredevelopedusingartificialintelligence(AI)toadjusttothelevelofthestudents,buttheyprimarilyrelyonpatternrecognitionandpredictivealgorithms,toprovidestudentswithexercisesadjustedtotheirlevelbasedonapoolofthousandsofitems.Therecentadvancesingenerativeartificialintelligenceofferapromisingavenuetousesoftwaretoteachstudentswhilemaintainingamorehuman-likeinteractionwithstudentsthroughtheuseofnaturallanguage.

MostofthestudiesthathaveexaminedgenerativeAIineducationhavebeenconductedindevelopedcountriesandlabsettings,assessingtheshort-termeffectsofbriefinterac-tions(Kumaretal.,2023).InItaly,studieshavefoundpositiveeffectsofLargeLanguageModels(LLMs)onlearningoutcomesthroughhomeworksupport(Vanzoetal.,2024).In

5

theUnitedStates,ahuman-AIapproachwithexpertguidancethroughlanguagemod-elssupportstutorsinsteadofprovidingdirecthelptostudents,andfoundthatstudentsworkingonmathematicswithtutorsrandomlyassignedtohaveaccesstoatutorco-pilotare4percentagepointsmorelikelytomastertopics(Wangetal.,2024).AstudycarriedoutamongundergraduatestudentsatHarvardUniversityshowedthatthosewhoben-efitedfromanAI-poweredtutorathomeperformedbetterthanthoseexposedonlytoactivelearningclasses(Kestinetal.,2024).

OnlyafewstudiesevaluatetheeffectofgenerativeAItosupportstudentsthroughtutor-ing.InGhana,studentswhoweregivenaccesstoaphoneforonehouraweekandwereallowedtouseanAI-poweredmathtutorviaamessagingapptoindependentlystudymathimprovedtheirscoresmuchmorethanthosewithoutaccess,withaneffectsizeof0.36(Henkeletal.,2024).ArecentstudyinTu…rkiyeofaninterventionthatincludedonlyfoursessionsshowedthatwhileLLMscanimprovemathematicslearningoutcomes,theycanalsobedetrimentaltolearninginthelongtermiftheyareusedas”crutches”ratherthanastutors(Bastanietal.,2024).Asimilareffectwasfoundforcodingclassesinalabsetting(Lehmannetal.,2024).ThisstudyshowedmorepositiveimpactswiththeLLMusedwithpromptstosafeguardlearning(Bastanietal.,2024).

Thus,thispapercontributestothisrecentliteraturebyexaminingtheimpactofoneofthefirstprogramstoleverageLLMsforeducationalpurposesinadevelopingcountrycon-textusingarealexperimentaldesigninSub-SaharanAfrica.ItalsoaimstoaddresssomeofthechallengesidentifiedinrecentreviewsofemergingstudiesontheeffectofLLMsonlearning:thelackofobjectivemeasurestocomplementsubjectiveassessmentsofim-pact,weaknessesinthedefinitionofthecontrolandtreatmentgroups(Weidlichetal.,2025),andthelackofpoweranalysistodetermineadequatesamplesizes(Dengetal.,2024).Furthermore,theinterventionusedafree,off-the-shelfmodel,requiringminimalcustomizationandnopre-builtquestionbanks,whichmightfacilitateitsscalability.

Thefindingsofthisinterventionunderscoreseveralcriticalpolicyimplicationsforad-dressingthelearningcrisisindevelopingcountries,particularlyinSub-SaharanAfrica.Theprogramdemonstratedsignificantimpactsonlearningoutcomes,evenamidchal-lengessuchasinternetdisruptionsandpoweroutages,highlightingitspotentialincon-textswithsevereteachershortagesandresourceconstraints.AI-poweredtutoringpro-gramsusingLLMscancomplementtraditionalteachingbyenhancingteacherproduc-tivityanddeliveringpersonalizedlearningexperiences,particularlywhenpairedwithguidedprompts,teacheroversight,andalignmentwiththecurriculum.Theinterven-tion’scost-effectivenessandscalabilityarepromising,leveraginglocalstaffandfreetools

6

tominimizecostswhileeliminatingtheneedforextensivequestionbanksrequiredbytraditionaladaptivesoftware.However,policymakersmustaddresspotentialinequitiesarisingfromdisparitiesindigitalliteracyandaccesstotechnology.Investmentsininfras-tructure,teachertraining,andinclusivedigitaleducationareessentialtoensureequitableaccessandmitigatetheriskofexacerbatinginequalities.GiventhenascentapplicationofLLMsineducation,numerousquestionsremainunanswered,underscoringtheimpor-tanceofreplicatingthisstudy,includingwithsmallvariations.

Therestofthispaperisorganizedasfollows.Section2describestheinterventionandtheexperimentaldesign,includingthedataused.Section3presentsourmainresults,includingaheterogeneityanddosageanalysis,aswellasarobustnessanalysis.Section4discussescosteffectiveness,proposesfutureresearchdirections,andpresentspolicyimplications.

2InterventionandStudyDesign

2.1TheIntervention

Thestudyanalyzestheeffectsofanafter-schoolprograminwhichstudentsinteractedwithalargelanguagemodeltwiceperweektoimprovetheirEnglishskills,followingthenationalcurriculum.TheinterventionwasimplementedinBeninCity,Nigeria,us-ingCopilot,anLLMpoweredbytheGPT-4modelatthetimeofimplementation.1Theprogramwasimplementedoverasix-weekperiodbetweenJuneandJuly2024,targetingfirst-yearseniorsecondaryschoolstudents,whoaretypically15yearsold.2Theinterven-tionaimedtoimprovelearningoutcomesinEnglishlanguageclassesusinganAIchatbotasavirtualtutor.TheselectedtoolwasMicrosoftCopilot,poweredbyChatGPT-4,whichwasfreelyavailableandrequiredonlystudentregistration.Theprogramwasconductedinnineschoolsandthestudentsweregroupedaccordingtothenumberofcomputersineachschoollab,withanaverageof30studentspersession.Eachstudentwasallowedtoparticipateinamaximumoftwo1.5-hourafter-schoolsessionsperweek.

Theselectionofschoolswasbasedontheavailabilityofcomputerlabs.Theselabsvariedinthetypesofdevicestheyused,rangingfromlaptopstodesktopcomputers.Internetaccess,essentialforreal-timeinteractionwiththeLLM,wasprovidedthroughrouters

1GPT-4exhibitshuman-levelperformanceonvariousprofessionalandacademicbenchmarks,includingpassingasimulatedbarexamwithascorearoundthetop10percentoftesttakers(Achiametal.,2023).

2AdetailedimplementationtimelinecanbefoundinTable14.

7

andmobiletelephonesignals.However,internetdisruptionsandpoweroutageswerecommonchallengesfacedduringtheintervention.Despitetheseissues,studentswereabletointeractwiththechatbotforthemajorityofthesessions.

Allstudents’guardianssignedconsentforms,agreeingtotheirchildren’sparticipationinthepilotprogram.Studentsworkedinpairs,sharingacomputer,andengagedindialoguewiththeAItooltoenhancetheirlearning.Teachers,whoplayedacriticalroleinguidingthestudentsbutdidnotprovidedirectinstruction,participatedinasinglethree-daytrainingprogramwithonecohort.ThistrainingintroducedteacherstothefunctionalitiesoftheLLMandequippedthemwithpedagogicaltechniquestoensuretheirresponsibleuseandsupervisestudentsduringthesessions.Italsomadethemawareofpotentialrisks,suchashallucinationsandbiases,thattheLLMcouldhave.

Inthefirstsession,teachersfamiliarizedstudentswithMicrosoftCopilot,emphasizingbothitseducationalbenefitsandpotentialrisks,suchasover-relianceonthemodelandthepossibilityofhallucinationsandbiasedoutputs.Thegoalwastofosterresponsibleus-age,encouragingstudentstocomplementtheirlearningwiththeAItoolwhileretainingcriticalthinkingskills.

Eachsubsequentsessionfocusedonatopicfromthefirst-yearEnglishlanguagecurricu-lum,alignedwiththematerialthatstudentscoveredduringtheirregularclasses.Thesessionsbeganwithateacher-providedprompt,followedbyfreeinteractionbetweenthestudentpairsandtheAItool.Teacherscirculatedtheclassroom,ensuringstudents’in-teractionsremainedrelevantandontask.Eachteacherwasprovidedwithathree-partimplementationtoolkitwhichincluded:a)curatedonlinelearningresourcesontheuseofCopilotandLLMs;b)ahandbookfocusedonAIliteracyandpotentialrisksandbenefits;andc)sessionguidelines,includingsuggestedinitialpromptsandpotentialfollow-upquestionstoassiststudentsifneeded.Teacherswerealsoprovidedwithcontactsincasetheyfacedanyproblemswiththeprogramimplementation,andagroup-chatwascre-atedtostreamlinecommunications.Thestudentsalsohadacustomizedguide,whichincludedtheinitialprompts.

ThelessonguidesandtheirpromptswerecarefullycraftedtopositiontheLLMasatu-tor,focusingonfacilitatinglearningratherthansimplyprovidingdirectanswers.ThesepromptswereinformedbyprinciplesfromthescienceoflearningandweretailoredtotheculturalcontextofsouthernNigeria,incorporatingfamiliarnamesandcustomstoresonatewithstudents.3SomeofthepromptstructureswerederivedfromMollickand

3OneofthestrategiesemployedtoenhancelearningthroughpromptingwastoencouragetheLLMtoleverage”desirabledifficulties”ratherthansimplyprovidingdirectanswers.Theseareconditionsthat,

8

Mollick(2023a).ThisdesignaimedtoencouragetheLLMtoadapttoeachstudent’sindividuallearninglevel,providingpedagogicalsupportthroughcontextuallyrelevantexamplesanddiverseteachingtechniques.StudentsinteractedwiththeLLMbyaskingquestions,completingexercises,andreceivingpersonalizedfeedback.Attheendofeachsession,thestudentswereencouragedtoreflectanddiscusslessonslearnedandchal-lengesencounteredduringsessiontofacilitateknowledgesharingamongthegroup.

Toensurethefidelityofprogramimplementation,monitorswerefirsttrained,providedwithmonitoringguidelines,andthenassignedtotrackstudentattendanceandgatherinformationabouteachsessionusingKoboToolbox.4Thissystemallowedforreal-timedatacollection,ensuringthattheinterventionwascarriedoutasintendedineachschoolandofferedtheopportunitytorespondpromptlytoanychallenges.5

2.2SampleandRandomization

Therandomizationforthepilotprogramwasconductedatthestudentlevelinthenineselectedschools.Allfirst-yearseniorsecondaryschoolstudentsintheseschoolswereinformedabouttheprogramthroughinformationsessionsandgivenawindowoftendaystoexpresstheirinterestinparticipating.Onlystudentswhovoluntarilyexpressedinterestwithinthisperiodwereincludedintherandomizationpool.

Toassesswhetherstudentswhoexpressedinterestintheafter-schoolprogramdifferedsystematicallyfromthosewhodidnot,wecomparepre-programexamscoresbetweenstudentswhowereeligibleforthelottery(i.e.,thosewholaterexpressedinterest)andthosewhowerenot.Table12reportsestimatesfromregressionsofbaselineacademicoutcomesoneligibilitystatus.Inthefirstterm,studentswhowouldlaterexpressinterestscored0.085standarddeviationshigherthantheirpeers(p?0.1)(seeFigure6).However,bythesecondterm—stillpriortothelottery—thisrelationshipreverses:studentswho

whileseeminglychallenging,fostermoredurableandflexiblelearning(Bjork,1994).Forexample,theinitialandsuggestedpromptsincorporatedevidence-basedprinciplessuchasretrievalpractice—showntobeeffectiveforuppersecondarystudentswhenimplementedthroughmultiple-choiceandshort-answerquizzes(McDermottetal.,2014)—elaborativeinterrogation(Dunloskyetal.,2013),andtheuseofconcreteexamples(Weinsteinetal.,2018).However,webelievethereissignificantpotentialforfutureiterationsoftheinterventiontomorefullyexploitevidence-basedstrategiesforimprovinglearningoutcomes.Forinstance,whileinourprogram,eachsessionwasdedicatedtoasinglecurriculumtopic,futureprogramscouldexperimentwithvariations,suchasincorporatinginterleaving(Weinsteinetal.,2018)andspacingpractices(Kang,2016).Theseapproacheswouldallowforthecoverageofmultipletopicswithinasinglesession,revisitingandreinforcingthemovertimetoenhancelong-termretentionandunderstanding.

4Fordetailsonthistool,seeDas(2024).

5Themonitoringdataincludedteacherandstudentattendance,punctuality,powerandinternetcondi-tions,aswellasparticipants’engagement,amongotherfactors

9

didnotexpressinterestscored0.147standarddeviationshigher(pi0.01)(seeFigure7).Theabsenceofaconsistentdirectionalpatternacrosstermssuggeststhatselectionintotheprogramwasnotstronglyorsystematicallycorrelatedwithacademicperformance.Whileouranalysisfocusesontreatmenteffectsamongthosewhoexpressedinterest,thelackofclearacademicselectionimpliesthatresultsmaygeneralizebeyondthisgroup.Nevertheless,welackdemographicdataonnon-interestedstudents,whichlimitsourabilitytoassessrepresentativenessalongotherdimensions.

Oncetheperiodtoexpressinterestclosed,therandomizationwascarriedoutusingsim-plerandomsamplingwithoutreplacement6amonginterestedstudentstoassignthemeithertothetreatmentgroup,whichparticipatedintheprogram,ortothecontrolgroup,whichdidnotreceiveanyinterventionbutcontinuedtheirregularlearningintheclass-room.Thestudentscompletedabaselinesurveyandanend-linesurveywithsociodemo-graphicinformation.Initially,657studentswereassignedtothetreatmentgroupand671tothecontrolgroup.However,only422studentsinthetreatmentgroupand337inthecontrolgroupcompletedthefinalassessment,whichconstitutesthefinalsampleusedfortheanalysis.

Table1providessummarystatisticsandbalancetestsforkeyobservablecharacteristicsofthetwogroups.Demographicvariablesincludegender,age,andasocio-economicstatus(SES)index.Thisindexwasderivedfromaprincipalcomponentsanalysisofhouseholdcharacteristics,suchasaccesstogoods(computers,phones),services(internetconnec-tion),studyspaces,andparentaleducation.7TheSESindex,aswellasothervariablessuchastheproportionoffemalestudentsandage,showsthatthesampleisbalancedacrossthetreatmentandcontrolgroups,withdifferencesth