TheLearning-TransferEvaluationModel:

SendingMessagestoEnableLearningEffectiveness

byWillThalheimer,PhD

PAGE

TheLearning-TransferEvaluationModel:

SendingMessagestoEnableLearningEffectiveness

ByWillThalheimer,PhD

Citation:

Thalheimer,W.(2018).Thelearning-transferevaluationmodel:

Sendingmessagestoenablelearningeffectiveness.

Availableathttps://WorkL/CatalogThisreportisalignedwithLTEMVersion12.

LTEMVersion12incorporatestwosignificantchanges:(1)themodelisinverted,makingbetterevaluationapproachesappearatthetop,notatthebottomaswasdisplayedinpreviousversions,and

(2)theterm“Level”(asin“Level7”)hasbeenchangedto“Tier”(asin“Tier7”)—todistinguishLTEMterminologyfromtheterminologyusedintheoriginalfour-levelmodel.Notethatbecausethemodelisalreadybeingusedwithsuccessandbecausetheselasttwochangesaretheonlyremainingchangesconsistentlysurfacedbythemodel’susers,thistwelfthiterationwillbethelastiterationfortheforeseeablefuture.

?Copyright2018byWillThalheimer.Allrightsreserved.

Iwelcomecommentsandsuggestionsatinfo@!

SpecialThanks:

ToJulieDirksen,ClarkQuinn,RoyPollock,AdamNeaman,YvonDalat,EmmaWeber,ScottWeersing,MarkJenkins,IngridGuerra-Lopez,RobBrinkerhoff,TrudyMandeville,MikeRusticiandvariousanonymousaudiencemembersattalksandworkshopswhereIdiscussedthemodel,especiallyISPI’s2017DesignThinkingConferenceandtheLearningTechnologies2018conferenceinLondon.WithoutthethoughtfulandextensivefeedbackIreceived,neitherthemodelnorthisreportwouldbeanygood.

TheLearning-TransferEvaluationModel:

SendingMessagestoEnableLearningEffectiveness

byWillThalheimer,PhD

Oneofthemostinsidiousandnefariouspropertiesofscientificmodelsistheirtendencytotakeover,andsometimessupplant,reality.

ErwinChargaff

Whileseeinganynumberofblackcrowsdoesnotproveallthecrowsareblack,seeingonewhitecrowdisprovesit.Thusscienceproceedsnotbyprovingmodelscorrectbutbydiscardingfalseonesorimprovingincompleteones.

ByronK.Jennings

TheGist

TheKirkpatrick-Katzellfour-levelmodel—conceivedbyRaymondKatzellinthe1950sandenhancedandpopularizedbyDonaldKirkpatrickoverthenextsixdecades—hasleftanindeliblemarkontheworkplacelearningfield.Thefour-levelmodelwasinfluentialintransformingthetraining-and-developmentfieldintothelearning-to-performancefield.WeareindebtedtoRaymondKatzellandDonaldKirkpatrickforthemodel’semphasisontheinstrumentalityoflearning,animatingtheideathatlearningisintendedtoenablebehaviorchangeandorganizationalresults.Unfortunately,thefour-levelmodelalsosendsmessages—unintendedbyitsauthors—thatundermineoureffortsaslearningprofessionalstoimproveourlearninginitiatives.Almost60yearsaftertheKirkpatrick-Katzellmodelwasintroduced,theworkplacelearningfieldislongoverdueforanimprovedmodeloflearningevaluation—onethatspursustogreaterlearningeffectiveness.

ThisreportintroducesaseriouspracticalalternativetotheKirkpatrick-Katzellmodel.TheproposednewmodelisalignedwiththescienceoflearningandisintentionallydesignedtocatalogamorerobustsetofrequirementsthantheKirkpatrick-Katzellmodel—requirementstargetedtoovercomethemosturgentfailuresinlearningpractice.

Inshort,thenewmodelprovidesmoreappropriateguideposts,enablingus,aslearningprofessionals,tocreatevirtuouscyclesofcontinuousimprovement.Thenewmodelis

relevanttoeducationalcontextsaswell,wherelearningdoesn’tnecessarilyfocusonjobperformanceperse,butonotherfutureapplicationsoflearning.

MindsforModels

Wehumansbuildmodels—notonlybecauseit’sfunandprofitable,butbecauseweneedmodelstohelpusmakesenseofourcomplexworld.Intruth,weneedmodelsbecauseourmindsaresmall.Oh,certainly,withexhaustiveconcentrationwecandiscerneventhemostpuzzlingphenomena.Butallofus(iftruthbetoldanymore)arepre-inclinedtosimplicity—totheleastenergeticusesofourminds.Welikethingssimple.Nothingwrongwiththat!It’snature.It’snatural.Anditactuallymakessenseifyouthinkaboutit.Who’sgottimetowastewhenweneedtofocusonwhatourbosswants,whatshowtobingewatch,whatmonstrousoutrageourpoliticalleaderswillbecaughtemoting?

Inthelearningfield,webuildallkindsofmodels.We’vegotthefourlevelsofKirkpatrick,theFiveMomentsofNeed,thefivephasesofADDIE,theNineEventsofInstruction,theDecisiveDozen.Weusethesemodelsbecauseweneedthem.Theyhelpusknowwhattodo.Butwhatiftheydidn’t?Whatiftheypushedustowardpoorpractices?Shouldn’twethendiscardthemorfixthemorreplacethem?Ofcourse!

Amodelisonlyasgoodasthemessagesitsends.Ifthemessagesarehelpful,it’sagoodmodel.Ifthemessagesareharmful,it’stimeforabettermodel.Messagespushustowardsomeactionsandawayfromothers.

Modelscreatemessages.Messagesnudgeactions.Actionsproduceresults.Ifourmodelsconveyunhelpfulmessages,weendupwithpoorresults.Themoreourmodelsconveyhelpfulmessages,thebetterourlearningresults.

Thebottomline—thethingI’mgettingtointhisreport—isthatit’stimeforabetterlearning-evaluationmodel.Thefour-levelKirkpatrick-Katzellmodelisaclearandpresentdanger.Itsendsnumerousharmfulmessages.Ithaspushedustowherewearetoday—enmeshedinanindustrywheretoofewofusaredoinggoodlearningevaluation,andhardlyanyofusaregettinggoodfeedbackabouttheeffectivenessofourlearning.

Mostly,wemeasureLevel1.Level1.Level1.Level1.It’sinsanity!Yes,Iwroteabookonhowtogetfeedbackfromlearners—howtodoaKirkpatrick-KatzellLevel1—butgettinggoodfeedbackfromlearnersistheleastweshouldbedoing.

Let’sthinkaboutthis.

Learning-EvaluationMessages

Inmybook,Performance-FocusedSmileSheets,Iintroducedtheconceptofstealthmessagestoshowhowlearner-feedbackquestionscansendmessagesaboutfactorsthatarecriticaltolearningeffectiveness.Forexample,aquestionaboutafter-trainingsupportsendsastealthmessagetolearners,trainers,andinstructionaldesignersthatafter-trainingsupportiscriticalforlearningtransfer.Aspecificfocusonmanagementsupportcansendamessagethatinvolvinglearners’managersisvitaltolearningapplication.

Stealthmessagesaremessageswesendthataren’tperceivedasbeingattemptsatpersuasion.Theyarepotentfortworeasons—firstbecausetheysendmessagesandsecondbecausetheirmessagesaremorelikelytobeacceptedasgivens;lesslikelytobesubjectedtoextendedconsciousdeliberationsinthemindsofthosereceivingthemessages.Stealthmessagescanbedeliveredwithverbalcommunications,buttheycanalsobedeliveredorstrengthenedthroughtheuseofsymbolic,visual,ormetaphoricnudges.Thecolorredprovidesastealthmessageofwarningordangerinmanyhumancultures.TheADDIEmodel,withthe“E”standingforevaluation,sendsastealthmessagethatevaluationshouldbedoneattheendofthelearningdevelopmentprocess.ADDIE,then,usesahybridofverbalandsymbolicmessaging.Theword“Evaluation”isverbal.Itsplacementattheendofthemodelissymbolic,visual,andmetaphoric.

I’mspendingtimeonthestealthmessagingnotionbecauseitexplainsalmosteverythingaboutthesorrystateofourlearningevaluationpractices.Italsogivesusawayout—awaytowardaninfinitelybettermodeloflearningevaluation.

Here’sthebottomline:Ifourmodelspushustowardbadevaluationpractices,we’relikelytocreatepoorlearningevaluations.Ifourmodelspushustowardgoodevaluationpractices,we’relikelytocreateeffectivelearningevaluations.So,letusstartbyexaminingthemessageswemightwanttosend.

Whatmessagesshouldwesend—ifwecan—whenweconsiderlearningevaluation?Here’smyshortlist:

JustbecauselearnersENGAGEINLEARNINGdoesn’tmeantheywillhavelearned.Therefore,measuringattendanceisaninadequatewayofevaluatinglearning.

JustbecauselearnersCOMPLETEALEARNINGEVENTdoesn’tmeantheylearned.Therefore,measuringcoursecompletionisaninadequatewayofevaluatinglearning.

JustbecauselearnersPAYATTENTIONdoesn’tmeantheylearned.Therefore,measuringattentionisaninadequatewayofevaluatinglearning.

JustbecauselearnersSHOWINTERESTinlearningdoesn’tmeantheylearned.Therefore,measuringinterestisaninadequatewayofevaluatinglearning.

JustbecauselearnersACTIVELYPARTICIPATEinlearningdoesn’tmeantheylearned.Therefore,measuringparticipationisaninadequatewayofevaluatinglearning.

JustbecauselearnerssaytheyLIKEALEARNINGEVENTdoesn’tmeantheylearned.Therefore,surveyinglearnersontheirgeneralsatisfaction—andonotherfactorsnotrelatedtolearningeffectiveness—isaninadequatewayofevaluatinglearning.

JustbecauselearnersREPORTTHEYHAVEEXPERIENCEDEFFECTIVELEARNINGMETHODSdoesn’tguaranteetheylearned.Therefore,surveyinglearnersontheirexperiencewithlearningmethods,whileitcanprovidesomegoodinformationonlearningdesign,mustbeaugmentedwithobjectivemeasuresoflearning.

JustbecauselearnersCANRECITEFACTSANDTERMINOLOGYdoesn’tmeantheyknowwhattodo.Therefore,measuringknowledgerecitationisaninadequatewayofevaluatinglearning.

JustbecauselearnersCOMPREHENDACONCEPTdoesn’tmeantheywillbeabletousethatconceptinaworksituation.Therefore,measuringknowledgeretentionisaninadequatewayofevaluatinglearning.

JustbecauselearnersDEMONSTRATEASKILLORCOMPETENCYduringalearningeventdoesn’tmeanthey’llrememberhowtousetheskillorcompetencylater.Therefore,measuringskillsorcompetenciesduringorsoonafteralearningeventisaninadequatewayofevaluatinglearning.

JustbecauselearnerscanMAKERELEVANTDECISIONSdoesn’tmeantheycanperformsuccessfully—theyalsohavetobefluent,appropriate,andtimelyinthewaytheyimplementthosedecisions.Therefore,measuringfortaskcompetence(thatis,measuringbothdecisionmakingandactiontaking)isbetterthanmeasuringfordecisionmakingcompetencealone.

MeasuringDECISIONMAKINGCOMPETENCEISABETTERMETRICthan

measuringknowledgerecitationorretention.Measuringdecisionmakingisalsobetterthangauginglearnerperceptions.

ThereareaNUMBEROFGOALSWESHOULDHAVEaslearningdesigners,includingsupportingourlearnersinbuilding:comprehension,remembering,decisionmakingcompetence,taskcompetence,andperseveranceinapplyingwhatthey’velearnedtotheirjoborotherperformancesituations.

ItisNOTSUFFICIENTTOSUPPORTLEARNERSONLYINCOMPREHENSION,

whichiswhattoooftenpassesforlearning.Ataminimum,wealsoneedtosupportlearnersinrememberingandindecisionmaking.

MEASURINGONLYTHEULTIMATEGOALSofalearninginterventionISINSUFFICIENTbecausewewillbeleftblindtofactorsthatenablethoseultimategoals.

1

It’simportanttomeasureBOTHTHEPOSITIVEANDNEGATIVEimpactsofourlearninginterventions.

Inmeasuringtheeffectsoflearning,it’sCRITICALTOSEPARATEOUTFACTORSthatarenotrelatedtothelearning.

Agoodevaluationmodelwillsendmessageshighlightingtheseissues—oratleasthintingatmostoftheseissues.Let’stakealookatthemessagessentintheKirkpatrick-Katzellmodel.

TheKirkpatrick-KatzellModel’sSuccessesandFailuresinMessaging

Thefour-levelKirkpatrick-Katzellmodelsendsmanyimportantmessages.

2

Ittellsusweshouldfocusnotonlyonlearning,butoncreatingon-the-jobbehaviorandorganizationalresults.Ithintsthatlearneropinionsarenotsufficientandshouldbealowerprioritythantheotherthreelevels.Ittellsusthefourlevelsareinterconnected.Thesearecriticallyimportantideas—especiallytheconceptthatourendgoalisnotlearning;thatlearningis,instead,ameanstoanend.WeoweadebtofgratitudetoDonaldKirkpatrickandRaymondKatzellforthisstealthmessage—thatlearningisnotourendgoal.

3

Thisisoneofthemostimportantideaseverconveyedintheworkplacelearningfield.TheKirkpatrick-Katzellfour-levelmodelwasinstrumentalingettingtheindustrytoacceptthistruth.

Unfortunately,themodelalsoconveys—orisoftenunderstoodassending—othermessagesthatareproblematic.Itsendsthemessagethatlearnerfeedbackisrelatedtolearningresults,jobperformance,andorganizationalresults—butresearchshowsthatthisisnot

1So,forexample,ifourultimategoalistoproduceimprovedcustomersatisfaction,it’snotenoughtomeasurehowlearningimpactedthatsatisfaction.Wealsoneedtoknowwhetherthelearnerscouldmakegooddecisions,whethertheycouldrememberwhattheylearned,andwhethertheycomprehendedwhattheylearnedinthefirstplace.Toseewhythisisessential,supposewedidourhomeworkandwedeterminedthatcustomerserviceisworsethanitshouldbebecauseemployeeshaveaknowledgeproblem—onethatcanbehelpedwithaneffectivelearningintervention.Supposewetraintheemployeesbutcustomerservicedoesnotimprove.Ifwehaven’tevaluateddecisionmaking,wewon’tknowwhetherourtrainingwasdeficientinhelpinglearnersmakegoodcustomerservicedecisions.Ifwehaven’tevaluatedremembering,wewon’tknowwhetherourtrainingwasdeficientinsupportingremembering.Ifwehaven’tevaluatedcomprehension,wewon’tknowifourtrainingwasdeficientinsupportingcomprehension.Giventhatthereisacausalchainfromcomprehensionanddecisionmakingtorememberingtoon-the-jobapplicationtocustomerservice,wehavetoevaluatethefullcausalchaintoknowwhathappened.

2RaymondKatzelloriginatedtheideaofseparatinglearningevaluationintofourparts(Kirkpatrick,1956).DonaldKirkpatrickenhancedKatzell’sformulationbycreatinglabelsforeachofthefourlevels.Healso,forfivedecades,tenaciouslypopularizedtheuseofafour-partmodel.

3ItwillcertainlysurprisemanythatRaymondKatzellformulatedthefour-partnotionoflearningevaluation.Youcanreadmorehere:

https://wp.me/p9hau4-1bj,

whichcanbefoundonmyblog:

/wills-blog/)

true.Traditionalsmilesheetshavebeenshowntobevirtuallyuncorrelatedwithlearningresultsandworkperformance.

4

TheKirkpatrick-KatzellmodelalsosupportstheconnotationthatmeasuringLevel3(jobperformance)andLevel4(organizationalresults)ismoreimportantthanmeasuringLevel2(learningresults)—eventhoughwelearningprofessionalshavemuchmoreleverageoverlearningresultsthanwedooverjobperformanceandorganizationaloutcomes.ThisideaisespeciallyproblematicwhenweinterpretthemodeltomeanweoughttomeasureonlyLevel3jobperformanceorLevel4organizationalresults.It’sproblematicbecauseitignoresthecausalpathwaystartingatlearningandmovingtojobperformanceandthenorganizationalresults.IfwemeasureonlyLevel3and4andgetpoorresults,wewillhavenoideahowourlearningdesigncontributedtothesefailures.

5

Thefour-levelmodeliscenteredontraining—hidingtherolepromptingmechanisms(jobaids,performancesupport,etc.)andon-the-joblearningmightplayinlearningandperformance.Ifwhatgetsmeasuredgetsmanaged,thenwe’renotlikelytoevenconsiderpromptingmechanismsoron-the-joblearningastargetsforourevaluations.Indeed,untilrecentlythesenon-trainingmodalitieswerelargelyignoredintheworkplacelearningindustry.

TheKirkpatrick-Katzellmodelignoresthecausalchainfromlearningtorememberingtoperformancetoresults.Morespecifically,ithasabsolutelyzeromessagingaboutthecriticalimportanceofremembering—orthatlearninginterventionscanbegoodatsupportingcomprehensionandpooratminimizingforgetting.Ideally,weneedamodelthatpushesustouselearningdesignsthatsupportlong-termremembering.

TheKirkpatrick-Katzellmodelpushesustofocusonweighingoutcomes,whileitislargelysilentaboutmeasuringthevariouslearningsupportscriticaltogoodlearningoutcomes.Mostpointedly,themodelignorestherolenon-learningprofessionals—especiallymanagers—canplaytosupportthelearning-to-performancecycle.

Thefour-levelmodelprovidesnoleveragepointsforlearning-designimprovements.Indeed,withitsfocusonlearnerperceptionsasacausalfactorinlearningresults,ithasdistractedusfromlearningfactorsthatreallymatter.Ideally,we’dwantourlearningevaluationmodel

4Alliger,Tannenbaum,Bennett,Traver,&Shotland,1997;Sitzmann,Brown,Casper,Ely,&Zimmerman,2008.

5IamsomewhatintriguedbyRoyPollock’sidea(personalcommunicationNovember2017)thatmaybeweshouldmeasureworkresultsroutinely—andlookatotherlearningresultsonlyifsomethinggoeswrong.I’mstilluncomfortablewiththisasIthinkweneedmorefeedbackonoureverydayperformanceaslearningprofessionals,notless.Butmaybethereisacompromisethatmightwork—perhapsaperformance-focusedlearnerfeedbacksurveyandavalidmeasureofworkperformanceasRoysuggests.

toencouragethoughtfulreflectiononhowtoimproveourlearningdesigns—pointingustotheimportanceofrememberinganddecisionmakingcompetence,forexample.

TheKirkpatrick-Katzellmodelalsofailstowarnusaboutsomeoftheworstevaluationpractices—commonpracticeslikegivinglearninginterventionsrecognitionofachievementwhenlearnersattend,payattention,complete,orparticipateinlearning.Fortoolong,manyinthelearningprofessionhaveusedtheseirrelevantmetricsasindicatorsoflearning.Agoodlearningevaluationmodelwoulddisavowtheseineffectualevaluationpractices.

Nomodeliseverperfect,and—giventhegargantuancomplexityofhumanlearning—nolearningevaluationmodelislikelytocontainallthemessagingwemightwant.Inthenewmodel,TheLearning-TransferEvaluationModel,someoftheabovelimitationsareovercome,butmanyareleftforustoconsideroutsidetheparametersofthemodel.

RaymondKatzellandDonaldKirkpatrick,workinginthe1950s,werenotprivytotoday’sadvancesinthescienceoflearning,norcouldtheyhaveforeseentheevaluationpracticesthatwouldbepopularinthenextcentury.Theybuilttheirmodelinresponsetopracticesofthe1950s.ThekeyinsightsbakedintotheKirkpatrick-Katzellmodelmustberetainedinanynewmodel,inappropriatemessagesmustberemoved,andcriticalmissingmessagesmustbeaddedwherepossible.

ThefollowingtwolistssummarizethebenefitsandharmsoftheKirkpatrick-Katzellmodel:

6

BeneficialMessagingInherentintheKirkpatrick-KatzellFour-LevelModel:

Tellsusweshouldfocusnotjustonlearning,butweshouldalsoaimforon-the-jobperformanceandorganizationalresults.

Impliesthatlearningisinstrumentalinachievingimprovedbehaviorandresults.

Hintsthatlearneropinionsofthelearningarenotsufficientontheirown,andshouldbeviewedasalowerprioritythantheotheroutcomemeasures.

Impliesthatlearneropinions,learningresults,on-the-jobbehavior,andorganizationalresultsareinterdependent.

7

6TheKirkpatrickNewWorldModelintroducessomeminortweaks,butgenerallyconveysthesamemessagingastheoriginal.Indeed,withitsrecommendationtothinkfirstaboutresultsandworkbackward,it’snotsomuchanevaluationmodelasalearningdevelopmentmodel.

7Thisismostlyabeneficialmessageexceptthatlearnerperceptionshavebeenshowninpracticetobevirtuallyuncorrelatedwithlearningresultsandon-the-jobbehavior.Tobespecific,thebeneficialmessageisthatLevel2(learningresults),Level3(on-the-jobbehavior),andLevel4(organizationalresults)areinterrelated.Ontheotherhand,perhapsnewmethods—liketheperformance-focusedlearnerfeedbackapproach—mayenableareliablepositiverelationshipbetweenlearnerperceptionsandtheotherlearningoutcomes.

HarmfulMessagingInherentintheKirkpatrick-KatzellFour-LevelModel:

Itfailstodisabuseusofineffectivelearningevaluationpracticessuchasgivingcreditforattendance,completion,attention,andparticipation.

Itignoresthecausalchainfromlearningtorememberingtoperformanceandresults.

Itignoresthecriticalimportanceofsupportingrememberingandminimizingforgettinginourlearningdesigns.

Itignoresandfailstohighlighttheadvancedcompetenciesofdecisionmakingandtaskperformance.

Itpromptsustovaluelearnerratingsaspredictiveoflearning,workperformance,andorganizationalresults—somethingproventobefalseastypicallypracticed.

8

ItimpliesthatLevel3andLevel4(jobperformanceandorganizationalresults)aremoreimportantthanmeasuringlearning—eventhoughwe,aslearningprofessionals,havegreaterleverageoverlearning.

Itisalsotooeasilyinterpretedtomeanwedon’tneedtomeasurelearningresultsbecausejobperformanceandorganizationalresultsaremoreimportant—thusignoringthefactthatlearningisonthecausalpathwaytojobperformanceandorganizationalresults.

Itpushesustofocusonlearningoutcomesandignoresthemeasurementoflearningsupports,includingsuchthingsassituationalcues,resourcesandtime,culturalsupport,andmanagementsupport.

Themodelistrainingcentric,ignoringpromptingmechanisms(likejobaidsandperformancesupport)andon-the-joblearning.

Itconveystheideathattheonlygoaloflearningshouldbeorganizationalresults—ignoringlearningoutcomesthataffectlearnersdirectlyandignoringotherpotentialbeneficiariesandinjuredparties.

8Butseethecaveatinthepreviousfootnote.

TheNewModel

Thenewmodel,TheLearning-TransferEvaluationModel(whichwemightpronounce“L-tem”andwriteLTEM),isdesignedspecificallytohelporganizationsandlearning

professionalsdeterminewhethertheirevaluationmethodsareeffectiveinprovidingvalidfeedback.Previousevaluationmodelshavenotdonethissufficiently.

LTEMiscomposedofeighttiers

9

—startingatthetopwiththeeffectsoflearningtransferandmovingdownthepagetowardcompletelyinadequatemethodsoflearningevaluation.Themodelismadeavailableinasingle,tabularformattoeasecomprehension.Itisannotatedandcolorcodedtoenhancetheclarityofitsmessaging.Thecolorcodingutilizesredtoindicatepoororinadequatemethods,yellowtoindicatemediocremethods,andgreentoindicategoodorusefulevaluationmethods.

10

Themodelisdesignedtoberelevantforalllearninginterventions,includingclassroomlearning,elearning,mobilelearning,on-the-joblearning,self-studylearning,etc.Wherelearningoccurs,wecanevaluateit.Wemayhavetochangeterminologyormodifyourevaluationpractices,butthenewmodelisdesignedtohelpbringwisdomtoourefforts.

Themodel,whileanexponentialimprovementovertheKirkpatrick-Katzellfour-levelmodel,stillleavessomeimportantmessagesunstated.Forexample,itdoesnotfocusonevaluatingpromptingmechanismslikejobaidsorperformancesupport.Thisisintentional.Whenmodelsgettoocomplicated,peopledon’tusethem.

Eachofthetiersofthemodelwillnowbedescribed.

Tier1—Attendance

Believeitornot,manyorganizationswillcertifylearningbycheckingattendance.Thisisaterribleideabecauseattendancedoesnotguaranteethatlearningoccurred.Learnershavebeengivencreditforsigningupforcourses,startingcourses,sittingthroughlearningevents,andcompletinglearningexperiences.Noneoftheseattendancemetricsisacceptableasameasureoflearning.

Tier2—Activity

Toooften,weuselearneractivityasagaugeoflearning.Wedothisformallyandinformally.Forexample,wemayformallymeasurethenumberofwordsalearnerusesinanonline

9Iusetheword“Tiers”insteadof“Levels”todistinguishLTEMfromtheKirkpatrick-KatzellFour-LevelModel.ThischangewasmadeinthetwelfthversionoftheLTEMmodelbecauseinpracticeIfoundthatpeoplewantedtodiscussLTEMincomparisontotheFour-LevelModel.Saying“Level4”couldgetpeoplethinkingabouttheKirkpatrick-KatzelllevelfourwhentheintentionwastotalkaboutLTEMlevelfour—orviceversa.

10Thecolorcodingisalignedwithsomeculturalexpectationsbutmisalignedwithothers.So,forexample,redcanconnotebadorgooddependingonone’sexperiencesorculturalinfluences.Colorexpectationsaretoovariedtoworkuniversally,soI’veleftthecolorsasoriginallydesigned,probablyreflectingmylifetimeofexperiencelivingintheUnitedStates.

forum.Wemayinformallygaugelearningsuccessbythe“energyintheroom”whenlearnersareinvolvedinsmall-groupdiscussions.

Therearethreemainwayswemeasurelearneractivity:(1)attention,(2)interest,and

(3)participation.Allthreeareunacceptableasmeasuresoflearningbecauseourlearnersmaypayattention,showinterest,andparticipateactively—yetnotlearn.

Whataresomeexamples?Someorganizationsgivecreditinelearningprogramswhenlearnersareperiodicallypresentedwithabuttontoclicktoshowtheyarepayingattention.Otherelearningprogramsmeasureattentionbymonitoringtimingofmouseclicks.

Measuringattentionisanunacceptablemetricbecauseattentiondoesnotguaranteelearning.Instructors,whetheronlineorinaclassroom,oftengivecreditforclassparticipation.Sometimespeersareaskedtoevaluateeachotheronvariousactivitycriteria.Rarely,butsometimes,outsideobserversevaluatelearneractivities.Notethatactivitymetricsgobeyondmereattendance;instead,theyattempttogaugesuchthingsashowactive,howengaged,howhelpful,orhowcooperativealearnerappearstobeduringlearning.Whilelearneractivitymayseemtobeindicativeofgoodlearningdesign,itisinadequatebecauseactivitydoesnotdemonstratelearningresults.Learnersmayfailtolearn.Theymaylearnthewrongthings.Theymaylearnthingspoorly.Theymayfocusonlow-priorityideasinsteadofcriticalconstructs.Theymayfailtobepreparedtousewhatthey’velearned.Someactivitiesareuseful.Someareawasteoftime—eventhoughtheyseemuseful.Someactivitiesareharmful.

Withtoday’sdigitaltechnologies,itistemptingtocountlearneractivityusingobjectivemetrics(forexample,thenumberofwordswrittenonadiscussionboard,numberofsocial-mediaposts