UnravelingMeta-Learning:UnderstandingFeature

RepresentationsforFew-ShotTasks

HarichandanaVejendla

(50478049)

1

2

Definitions

?Meta-Learning:Meta-learningdescribesmachinelearningalgorithmsthatacquireknowledgeandunderstandingfromtheoutcomeofothermachinelearningalgorithms.Theylearnhowtobest

combinethepredictionsfromothermachine-learningalgorithms.

?Few-shotLearning:Few-ShotLearningisaMachineLearningframeworkthatenablesapre-trainedmodeltogeneralizeovernewcategoriesofdatausingonlyafewlabeledsamplesperclass.

?FeatureExtraction:Featureextractionisaprocessofdimensionalityreductionthatinvolvestransformingrawdataintonumericalfeaturesthatcanbeprocessed.

?Featureclustering:Featureclusteringaggregatespointfeaturesintogroupswhosemembersaresimilartoeachotherandnotsimilartomembersofothergroups.

?FeatureRepresentation:RepresentationLearningorfeaturelearningisthesubdisciplineofthe

machinelearningspacethatdealswithextractingfeaturesorunderstandingtherepresentationofadataset.

3

Introduction

?TransferLearning:Pre-trainingamodelonlargeauxiliarydatasetsandthenfine-tuningtheresultingmodelsonthetargettask.Thisisusedforfew-shotlearningsinceonlyafewdatasamplesareavailableinthetarget

domain.

?Transferlearningfromclassicallytrainedmodelsyieldspoorperformanceforfew-shotlearning.Recently,few-shotlearninghasbeenrapidlyimprovedusingmeta-learningmethods.

?Thissuggeststhatthefeaturerepresentationslearnedbymeta-learningmustbefundamentallydifferentfromfeaturerepresentationslearnedthroughconventionaltraining.

?Thispaperunderstandsthedifferencesbetweenfeatureslearnedbymeta-learningandclassicaltraining.

?Basedonthis,thepaperproposessimpleregularizersthatboostfew-shotperformanceappreciably.

4

Meta-LearningFramework

?Inthecontextoffew-shotlearning,theobjectiveofmeta-learningalgorithmsistoproduceanetworkthatquicklyadaptstonewclassesusinglittledata.

?Meta-learningalgorithmsfindparametersthatcanbefine-tunedinafewoptimizationstepsandonafewdatapointsinordertoachievegoodgeneralization.

?Thetaskischaracterizedasn-way,k-shotifthemeta-learningalgorithmmustadapttoclassifydatafromTiafterseeingkexamplesfromeachofthenclassesinTi.

Algorithm

5

6

AlgorithmDescription

?Meta-learningschemestypicallyrelyonbi-leveloptimizationproblemswithaninnerloopandanouterloop.

?Aniterationoftheouterloopinvolvesfirstsamplinga“task,”whichcomprisestwosetsoflabeleddata:thesupportdata,Tis,andthequerydata,Tiq.

?Intheinnerloop,themodelbeingtrainedisfine-tunedusingthesupportdata.

?Fine-tuningproducesnewparametersθi,thatareafunctionoftheoriginalparametersandsupportdata.

?Weevaluatethelossonthequerydataandcomputethegradientsw.r.ttheoriginalparametersθ.Weneedtounrollthefine-tuningstepsandbackpropagatethroughthemtocomputethegradients.

?Finally,theroutinemovesbacktotheouterloop,wherethemeta-learningalgorithmminimizeslossonthequerydatawithrespecttothepre-fine-tunedweights.Basemodelparametersareupdatedusingthe

gradients.

7

Meta-LearningAlgorithms

Avarietyofmeta-learningalgorithmsexist,mostlydifferinginhowtheyarefine-tunedusingthesupportdataduringtheinnerloop:

?MAML:Updatesallnetworkparametersusinggradientdescentduringfine-tuning.

?R2-D2andMetaOptNet:Last-layermeta-learningmethods(onlytrainthelastlayer).Theyfreezethefeatureextractionlayers(featureextractor’sparametersarefrozen)duringtheinnerloop.Onlythelinearclassifierlayeristrainedduringfine-tuning.

?ProtoNet:Last-layermeta-learningmethod.Itclassifiesexamplesbytheproximityoftheirfeaturestothoseofclasscentroids.Theextractedfeaturesareusedtocreateclasscentroidswhichthen

determinethenetwork’sclassboundaries.

8

Few-ShotDatasets

?Mini-ImageNet:ItisaprunedanddownsizedversionoftheImageNetclassificationdataset,

consistingof60,000,84×84RGBcolorimagesfrom100.These100classesaresplitinto64,16,and20classesfortraining,validation,andtestingsets,respectively.

?CIFAR-FSdataset:samplesimagesfromCIFAR-100.CIFAR-FSissplitinthesamewayasmini-ImageNetwith60,00032×32RGBcolorimagesfrom100classesdividedinto64,16,and20

classesfortraining,validation,andtestingsets,respectively.

ComparisonbetweenMeta-LearningandClassicalTrainingModels

?DatasetUsed:1-shotmini-ImageNet

?Classicallytrainedmodelsaretrainedusingcross-entropylossandSGD.

?Commonfine-tuningproceduresareusedforbothmeta-learnedandclassically-trainedmodelsforafaircomparison

?Resultsshowthatmeta-learningmodelsperformbetterthanclassicaltrainingmodelsonfew-shotclassification.

?Thisperformanceadvantageacrosstheboardsuggeststhatmeta-learnedfeaturesarequalitativelydifferentfromconventionalfeaturesandfundamentallysuperiorforfew-shotlearning.

9

10

ClassClusteringinFeatureSpace

MeasuringClusteringinFeatureSpace:

Tomeasurefeatureclustering(FC),weconsidertheintra-classtointer-classvarianceratio:

φi,j-featurevectorcorrespondingtodatapointinclassiintrainingdata

μi-meanoffeaturevectorsinclassi

μ-meanacrossallfeaturevectors

C-numberofclasses

N-numberofdatapointsperclass

Where,fθ(xi,j)=φi,jfθ-featureextractor

xi,j-trainingdatainclassi

Lowvaluesofthisfractioncorrespondtocollectionsoffeaturessuchthatclassesarewell-separatedandahyperplaneformedbychoosingapointfromeachoftwoclassesdoesnotvarydramaticallywiththechoiceofsamples.

WhyClusteringisimportant?

?Asfeaturesinaclassbecomespreadoutandtheclassesarebroughtclosertogether,theclassificationboundariesformedbysamplingone-shotdataoftenmisclassifylargeregions.

?Asfeaturesinaclassarecompactedandclassesmovefarapartfromeachother,theintra-classtointer-classvarianceratiodrops,andthedependenceoftheclassboundaryonthechoiceofone-shotsamplesbecomesweaker.

11

ComparingFeatureRepresentationsofMeta-LearningandClassicallyTrainedModels

?Threeclassesarerandomlychosenfromthetestset,and100samplesaretakenfromeachclass.Thesamplesarethenpassedthroughthefeatureextractor,andtheresultingvectorsareplotted.

?Becausefeaturespaceishigh-dimensional,weperformalinearprojectionontothefirsttwocomponentvectorsdeterminedbyLDA.

?Lineardiscriminantanalysis(LDA)projectsdataontodirectionsthatminimizetheintra-classtointer-classvarianceratio.

?Theclassicallytrainedmodelmashesfeaturestogether,whilethemeta-learnedmodelsdrawtheclassesfartherapart.

12

13

HyperplaneInvariance

Thisregularizerwithonethatpenalizesvariationsinthemaximum-marginhyperplaneseparatingfeaturevectorsin

oppositeclasses

HyperplaneVariationRegularizer:

DatpointsinclassA:x1,x2

DatapointsinclassB:y1,y2

fθ-featureextractor

fθ(x1)-fθ(y1):determinesthedirectionofthemaximum

marginhyperplaneseparatingthetwopointsinthefeaturespace

?Thisfunctionmeasuresthedistancebetweendistancevectorsx1?y1andx2?y2relativetotheirsize.

?Inpractice,duringabatchoftraining,wesamplemanypairsofclassesandtwosamplesfromeachclass.Then,wecomputeRHVonallclasspairsandaddthesetermstothecross-entropyloss.

?WefindthatthisregularizerperformsalmostaswellasFeatureClusteringRegularizerandconclusivelyoutperformsnon-regularizedclassicaltraining.

14

Experiments

?FeatureclusteringandHyperplanevariationvaluesarecomputed.

?Thesetwoquantitiesmeasuretheintra-classtointer-classvarianceratioandinvarianceofseparatinghyperplanes.

?Lowervaluesofeachmeasurementcorrespondtobetterclassseparation.

?OnbothCIFAR-FSandmini-ImageNet,themeta-learnedmodelsattainlowervalues,indicatingthatfeaturespaceclusteringplaysaroleintheeffectivenessofmeta-learning.

15

Experiments

?Weincorporatetheseregularizersintoastandardtrainingroutineoftheclassicaltrainingmodel.

?Inallexperiments,featureclusteringimprovestheperformanceoftransferlearningandsometimesevenachieveshigherperformancethanmeta-learning

16

WeightClustering:FindingClustersofLocalMinimaforTaskLossesinParameterSpace

?SinceReptiledoesnotfixthefeatureextractorduringfine-tuning,itmustfindparametersthatadapteasilytonewtasks.

?WehypothesizethatReptilefindsparametersthatlieveryclosetogoodminimaformanytasksandis,therefore,abletoperformwellonthesetasksafterverylittlefine-tuning.

?ThishypothesisisfurthermotivatedbythecloserelationshipbetweenReptileandconsensusoptimization.

?Inaconsensusmethod,anumberofmodelsareindependentlyoptimizedwiththeirowntask-specificparameters,andthetaskscommunicateviaapenaltythatencouragesalltheindividualsolutionsto

convergearoundacommonvalue.

17

ConsensusFormulation:

?Reptilecanbeinterpretedasapproximatelyminimizingtheconsensusformulation

?Reptiledivergesfromatraditionalconsensusoptimizeronlyinthatitdoesnotexplicitlyconsiderthequadraticpenaltytermwhenminimizingfor?θp.

18

ConsensusOptimizationImprovesReptile

?WemodifyReptiletoexplicitlyenforceparameterclusteringaroundaconsensusvalue.

?Wefindthatdirectlyoptimizingtheconsensusformulationleadstoimprovedperformance.

?duringeachinnerloopupdatestepinReptile,wepenalizethesquareddistancefromtheparametersforthecurrenttasktotheaverageoftheparametersacrossalltasksinthecurrentbatch.

?ThisisequivalenttotheoriginalReptilewhenα=0.Wecallthismethod“Weight-Clustering.

ReptilewithWeightClusteringRegularizer

n-numberofmeta-trainingsteps

k-numberofiterationsorstepstoperformwithineachmeta-trainingstep

19

20

Resultsofweightclustering

?WecomparetheperformanceofourregularizedReptilealgorithmtothatoftheoriginalReptilemethodaswellasfirst-orderMAML(FOMAML)andaclassicallytrainedmodelofthesamearchitecture.We

testthesemethodsonasampleof100,0005-way1-shotand5-shotmini-ImageNettasks

?ReptilewithWeight-Clusteringachieveshigherperformance.

21

Resultsofweightclustering

?ParametersofnetworkstrainedusingourregularizedversionofReptiledonottravelasfarduringfine-tuningatinferenceasthosetrainedusingvanillaReptile

?Fromthese,weconcludethatourregularizerdoesindeedmovemo