軟件工程德克薩斯州奧斯汀市78712亞當·波特(AdamA.Porter)：馬里蘭大學計算機科學專馬里蘭大學帕克分校勞倫斯·沃塔(LawrenceG.Votta)：摩托羅拉公司伊利諾伊州阿靈頓高地，1但是在軟件工程的研究中，并不成功。與它們在其他科學領域的廣泛運用我們認為這些文章多以實現(xiàn)為導向，也就是說它們認為運用最大的在于實比如，NormanFenton等人[1]許多的數(shù)據(jù)設計很失敗，不能與大系統(tǒng)相匹配，而且實施時間過短。VictorBasil[2]。PhilipJohnson也提到實踐者被測試或量化[3]誠然，這樣那樣的原因影響著的運用，但是我們相信即使所有這些問題都我們嘗試用改變創(chuàng)建軟件方式的經(jīng)驗讓我們相信要達到目的，必須還要以在試著提高朗訊開發(fā)系統(tǒng)里的軟件檢驗流程中，我們得出這一結(jié)論，最大的不2、為什么用們的理論。所以有以下步驟：3、的狀前面提到是將我們相信的運用到我們所觀察到的事物上。理想情況下，這目前的優(yōu)的平均質(zhì)量進步很大?，F(xiàn)在研究普遍受過良好的教育，懂得程的[4]。我們已經(jīng)和目前活躍的學者有過多次交談，這些人已對或開始對感。當然，的形因有很多。很多研究和實踐者已經(jīng)解決了提高應有一些有、被廣泛的文章試著提高我們對現(xiàn)狀的關(guān)注。Tichy等人[5]以及Wallace和Zelkowitz[6]均認為相對于工程學的其他領域，在軟件工的，Kitchenham和Pfleeger為ACMSigSoftSoftwareEngineeringNotes寫過一系列文章，于行業(yè)數(shù)據(jù)、對行業(yè)經(jīng)驗有重要且詳細記錄的。這一的先驅(qū)包括國家航天局的軟件工程，計算機科學公司和馬里蘭大學[7]。體系上的盡管有上述優(yōu)勢，但問題依然嚴重。這源于對是什么和為什么做 通常當人們說我們需要做軟件工程的時，他們的意思是研究結(jié)果應該舉個例子，在編譯程序最優(yōu)化研究中，已經(jīng)辨認了通用代碼使用模式。例分、的用。太多的關(guān)注一些顯而易見的東西。但有些明顯的東西并非如此，所以我們么靠所做的論證錯在哪兒了？我們清楚地認識到正確的事情并非是上明顯的，許多的唯一賣點就是大量的數(shù)據(jù)，但僅僅有數(shù)據(jù)是遠遠不夠的。僅僅展示數(shù)據(jù)問題更基本的一面是很多缺乏假設。研究者沒有提出問題，研究沒有明確4、未來不僅是，所有研究的目標都是提高研究和實踐的現(xiàn)狀。如果我們想讓實證創(chuàng)造更好的研究很少是明確的。除了用一個研究試著解決一些大問題，須做多個研幫助他人我們結(jié)論的方法。可信的解 設計一個5、的結(jié)關(guān)于的文章也應討論它們，這些成分即：研究背經(jīng)做了哪些研究，還有哪些問題尚待回答，以及關(guān)注哪些問題。假假設是必不可少的。它們陳述了我們研究問題。有時人們對這一概念并不清組找出問題”。實驗設效度的因數(shù)據(jù)分析和展假設檢驗決定原假設可被的置信水平。置信水平是原假設被錯誤的可能性定性分析則是運用量化程度不高的數(shù)據(jù)，如觀察、采訪、等。當我們要理解人[8。軟件工程研究中，定量分析要比定性分析用得多，但我們在未來會看到的定性來檢驗定型數(shù)據(jù)的，兩者是互補品、用來相互驗證、是同一里不同形式的數(shù)據(jù)”。結(jié)果和結(jié)討論結(jié)果的實際意義。如果結(jié)果是普遍通用的，管理者和開發(fā)能怎樣用它？確保你提供了足夠的信息讓他人研究。6、具體步設計研相反，須縮小問題，問能引出重要答案的問題。Knight和Leveson對N-VersionProgramming的研究是一個說明問題的好例子[9]靠度可能沒有那么高。相互關(guān)聯(lián)的，進而N-VersionProgramming不能實現(xiàn)它有關(guān)高可靠度的承諾。，的不是所有問題都能像N-VersionProgramming一樣引出一個單一的。對于很們那么有用。些研究，減輕開發(fā)的負擔。這是一個很有力的優(yōu)勢，因為我們行業(yè)伙伴不想?yún)⒅R來解決，這樣吸納軟件工程專業(yè)以外的將十分有用。碼的一個例子來自于基于朗訊科技公司的代碼項目（CodeDecayProject）[11]。機構(gòu)的歷史、發(fā)展政策和編程標準。這個項目的目標是定義反應變量，證碼的 獲取數(shù)我們尚未關(guān)注的一個資源是版本控制系統(tǒng)（VCS。很多對不同流程和工具的長期影響的分析依賴于重建軟件在不同時間點的片段的能力。版本控制系統(tǒng)研發(fā)者對系統(tǒng)做出的每一個改變，作為結(jié)果，可以重建續(xù)的片段。版本控制系統(tǒng)的例子包括修訂控制系統(tǒng)（RCS）[13]和源代碼控制系統(tǒng)（SCCS）[14]中發(fā)生了什么，有時比研發(fā)的更可靠。同時，版本控制系統(tǒng)的數(shù)據(jù)經(jīng)得起自動望將模擬和建模結(jié)合起來指導研究。一個有趣的例子是Solheim和Rowland[15]做的有關(guān)系統(tǒng)的研究。其中，其他例子包括用數(shù)學建模檢驗更改某些性的成本收益[16]，用實驗設計理論產(chǎn)生測試囊括其教育這種教學模塊可以以教育的形式展開，教育作為自然科學教育的一個。要建立這樣的，研究者要把打包進一本手冊，手冊里包含講7本文提供了關(guān)于目前的概述，了它的優(yōu)勢和不足，討論了在創(chuàng)造為軟要改進現(xiàn)狀，須有更好的設計，并從中汲取可靠的解釋。作為未來結(jié)尾：設計更好的研究，獲取數(shù)據(jù)，在中獲得他人的幫助。盡管與其他科學和工程學領域相比，實證科學尚不成熟，但在進步，我們8、參考文[1]N.Fenton,S.L.Pfleeger,andR.Glass,ScienceandSubstance:AChallengetoSoftwareEngineers.IEEESoftware,1994.11(4):p.86-95.[2].V.Basili,Editorial.EmpiricalSoftwareEngineeringJournal,1996.[3].P.M.Johnson,ProjectLEAP:Lightweight,Empirical,asurementdysfunction,andPortableSoftwareDeveloperImprovement,inDepartmentofInformationandComputerSciences.1997,UniversityofHawaii,Honolulu.[4].D.Pregibon,etal.,StatisticalSoftwareEngineering,.1996,NationalAcademyofSciences:Washington,D.C.[5].W.F.Tichy,P.Lukowicz,L.Prechelt,andE.A.Heinz,ExperimentalEvaluationinComputerScience:AtativeStudy.JournalofSystemsandSoftware,1995.28(1):p.[6].M.V.ZelkowitzandD.Wallace,ExperimentalvalidationinsoftwareInformationandSoftwareTechnology,1997.39(11):p.735-[7].V.R.Basili,etal.TheSoftwareEngineeringLaboratory--AnOperationalSoftwareExperienceFactory.in14thInternationalConferenceonSoftwareEngineering.1992.Melbourne,Australia.[8].B.GlasserandA.Strauss,Thediscoveryofgroundedtheory:Strategiesforqualitativeresearch.1977,Chicago:AldinePublishing.[9].J.KnightandN.Leveson,AnExperimentalEvaluationoftheAssumptionofIndependenceinMulti-VersionProgramming.IEEETransactionsonSoftwareEngineering,1986.SE-12(1):p.96-109.[10].B.Schneiderman,R.Mayer,D.McKay,andP.er,ExperimentalInvestigationsoftheUtilityofdetailedFlowchartsinProgramming.CommunicationsoftheACM,1977.20(6):p.373-381.[11].S.G.Eick,etal.,DoesCodeDecay?AssessingtheEvidencefromChangeManagementData.IEEETransactionsonSoftwareEngineering,(toappear).[12].C.M.Judd,E.R.Smith,andL.H.Kidder,ResearethodsinSocialRelations.1991,FortWorth,TX:Holt,RinehartandWinston,Inc.[13].W.F.Tichy,Design,Implementation,andEvaluationofaRevisionControlSystem,inProceedingsoftheSixthInternationalConferenceonSoftwareEngineering.1982:Tokyo,Japan.p.58—67.[14].M.J.Rochkind,TheSourceCodeControlSystem.{IEEE}TransactionsonSoftwareEngineering,1975.1(4):p.364—370.[15].J.A.SolheimandJ.H.Rowland,AnEmpiricalStudyofTestingandIntegrationStrategiesUsingArtificialSoftwareSystems.IEEETransactionsonSoftwareEngineering,1993.19(10):p.941-949.[16].W.Harrison.Change-ProneModules,LimitedResources,andMaintenance.inwess.1996.Monterey,CA.[17].S.R.DalalandC.L.Mallows,Factor-coveringdesignsfortestingTechnometrics,1998.40:p.234-[18].G.V.Glass,B.McGaw,andM.L.Smith,Meta vsisinsocialresearch.1981,BeverlyHills,CA:Sage.[19].A.A.PorterandP.M.Johnson,AssessingSoftwareReviewMeetings:ResultsofaComparativeysisofTwoExperimentalStudies.IEEETransactionsonSoftwareEngineering,1997.23(3):p.129-145.EmpiricalStudiesofSoftwareEngineering:ADewayneE.AdamA.LawrenceG.ElectricalandComputerComputerUniversityofTexasatUniversityof1501W.ShureAustin,TXCollegePark,MDArlingtonHeights,ILInthisarticlewesummarizethestrengthsandweaknessesofempiricalresearchinsoftwareengineering.Wearguethatinordertoimprovethecurrentsituationwemustcreatebetterstudiesanddrawmorecredibleinterpretationsfromthem.Wefinallypresentaforthisimprovement,whichincludesageneralstructureforsoftwareempiricalstudiesandconcretestepsforachievingthesegoals:designingbetterstudies,collectingdatamoreeffectively,andinvolvingothersinourempiricalenterprises.EmpiricalStudies,SoftwareAnempiricalstudyisreallyjustatestthatcompareswhatwebelievetowhatweobserve.Nevertheless,suchtests,whenwiselyconstructedandexecutedandwhenusedtosupportthescientificmethod,playafundamentalroleinmodernscience.Specifically,theyhelpusunderstandhowandwhythingswork,andallowustousethisunderstandingtomateriallyalterourworld.Yetinsoftwareengineeringresearch,empiricalstudieshavenothadthesamesuccess.Thisseemsoddgiventheirwideuseinothersciences.Thisproblemhasbeenwidelydiscussedandmanyarticleshavepointedoutpossiblecauses.Weargue,however,thatmanyofthesearticlesare“implementation-oriented”.Thatis,theysuggestthatthebiggestbarrierstousingempiricalstudieslieinthedetailsofconductingthem.Forexample,NormanFentonetal.[1]pointoutthatmanyempiricalstudieshavepoorstatisticaldesigns,don’tscaleuptolargesystems,andareconductedovertooshortatime.VictorBasili[2]suggeststhatthemanydifferences

betweenindividualsoftwareprojectsmakecomparisondifficult.PhilipJohnsonalsoremarksthatpractitionersmayresistbeingmeasured.[3].Surely,theseandmanyotherfactorsaffecttheuseofempiricalstudies.Nevertheless,webelievethatevenifalltheseissuesdisappeared,empiricalstudieswouldstillfailtohavetheimpacttheyhavehadinotherfields.Thisisbecausethereisagapbetweenthestudiesweactuallydoandthegoalswewantthosestudiestoachieve.Ourexperienceinattemptingtouseempiricalstudiestochangehowadevelopmentgroupbuildssoftwarehasconvincedusthatwemustalsotakea“requirements-oriented”view.Thatis,thatwemustthinkharderaboutwhatexperimentsreallyareandhowtheycanbemosteffectivelyusedtoimprovesoftwaredevelopment.WecametothisconclusionwhiletryingtoimprovethesoftwareinspectionprocessusedinaLucentdevelopmentsetting.Wefoundthatourgreatestdifficultieswerenotindesigningandconductingindividualstudies(whichwasbynomeanseasy).Ourgreatestdifficultieswereinconceptualizingandorganizingabodyofworkthatcouldbereliedonasthebasisforchanginganorganization’slong-practiceddevelopmentprocesses.Moreover,webelievethatthisproblem–definingandexecutingstudiesthatchangehowsoftwaredevelopmentisdone-isthegreatestchallengefacingempiricalresearchers.Therefore,inthisessaywewillexaminethenatureandpurposeofempiricalstudies,discusshowtheyarecurrentlyused,andoffersomesuggestionsforimprovingtheminthefuture.WHYEMPIRICALAlllargesoftwareprojectsfollowsomeunderlyingdevelopmentprocessthatincludesstagessuchasrequirementsdefinition,functionaldesign,unitimplementation,integration,andsoon.Thewayinwhichthesestagesareconducted,thetoolsthatareusedtosupportthemandtherationaleforngso,however,varieswidely.Somecompanieshaverigidprocessesthatallprojectsfollow.Othersallowindividualmanagerstomakedecisionsbasedontheir alexpertise.Otherssimplyfollowinstitutionaltraditionsforlackofsuitablealternatives.Nomatterwhichapproachistaken,inalmostallcases,thereislittlehardevidencetoinformthesedecisions,andtheircostsandbenefitsarerarelyunderstood.OnereasonforthisisthatsoftwareengineeringresearchhasfailedtoproducethedeepmodelsyticaltoolsthatarecommoninotherThesituationindicatesaseriousproblemwithresearchandpracticeinsoftwareengineering.Wedon’tknowthefundamentalmechanismsthatdrivethecostsandbenefitsofsoftwaretoolsandmethods.Withoutthisinformation,wecan’tlwhetherwearebasingouractionsonfaultyassumptions,evaluatingnewmethodsproperly,orinadvertentlyfocusingonloyoffimprovements.Infact,unlessweunderstandthespecificfactorsthatcausetoolsandmethodstobemoreorlesscost-effective,thedevelopmentanduseofaparticulartechnologywillessentiallybearandomact.Empiricalstudiesareakeywaytogetthisinformationandmovetowardswell-foundedEmpiricalstudiestakemanyforms.Theyarerealizednotonlyasformalexperiments,butalsoascasestudies,surveys,andprototyexercisesaswell.Nomatterwhatitsformis,theessenceofanempiricalstudyistheattempttolearnsomethingusefulbycomparingtheorytorealityandtoimproveourtheoriesasaresult.Therefore,empiricalstudiesinvolvethefollowingsteps:formulatinganhypothesisorquestiontoobservingaingobservationsintoyzingthedata,drawingconclusionswithrespecttothetestedOfthese,thelaststep–drawingconclusions-isthemostimportantandtoooftentheleastwelldone.It’simportantbecauseit’sherethatwegettheinformationthatwillenableustoguide,tochangeandtopushourfield.It’sherethatwepinpointinefficiencies,identifywherelargeimprovementscanbemade,anddeterminewhetherourstill-formingideasareon-track.It’sthereasonwhywedoempiricalstudies.Theothersteps,howeverindispensable,areonlyprologue.Ofcourse, ngallofthesestepswellisdifficult.Donewell,however,thepayoffswillbelarge,includingthat:knowledgeisencodedmore yofforerroneousresearchideasarediscarded

high-payoffareasarerecognizedandcorrectlyvalued,importantpracticalissuesareTHESTATEOFEMPIRICALWehavesaidthatempiricalstudiesareusedtocomparewhatwebelievetowhatwesee.Ideally,thesetestsshouldallowustopositivelyaffectthepracticeofsoftwaredevelopment.Inthissectionwewillexploretowhatdegreewe,asaresearchcommunity,arelivinguptothisideal.CurrentEmpiricalsoftwareengineeringhasmaturedconsiderablyoverthelast10-20years.Considerforexample:Insomesoftwareengineeringsub-fieldsempiricalvalidationisconsidered,ifnotastandardpart,thenapowerfuladditiontoresearchpapers.Thishasbeenespeciallynotableinthetestingcommunity.Thequalityoftheaverageempiricalstudyisrising.Researchersare ingbettereducatedaboutempiricalstudiesandhowtoconductthem.Consequently,weareseeingincreasinglymorecomprehensivestudiesconductedonincreasinglyrealisticprogramsandprocesses.Fundingagenciesarerecognizingthevalueofempiricalstudies.IntheU.S.forexample,NationalScienceFoundation(NSF)programssuchastheExperimentalandIntegrativeActivitiesprogramsupportsresearchwithadecidedlyexperimentalflavor.TherecentlyproposedInformationTechnologyResearch(ITR)programalsostressesthatproposalsincludeastrongvalidationcomponent.OtherexamplesincludeNationalAcademyofSciencessponsoredworkshoponthetopicofstatisticsandsoftwareengineering[4].We’vehadmanytalkswithcurrentlyactiveresearcherswhohave einterestedinandarebeginningtodoempiricalstudies.Andfinally,therehavebeenseveralempiricalstudies-relatedtutorials,panelsandstate-of-theartpresentationsatmajorsoftwareengineeringconferencessuchasICSE,FSE,ICSMandothers.Ofcoursemanyfactorscontributetothissituation.Manyresearchersandpractitionershavetackledtheproblemofincreasingtheuseandeffectivenessofempiricalstudies.Forexample:Therehavebeenseveralinfluentialandwidelyquotedarticlesattemptingtoraiseourconsciousnessaboutthestateofempiricalstudiesinsoftwareengineering.Tichyetal.[5]andWallaceandZelkowitz[6]botharguethatempiricalstudiesareunderusedinsoftwareengineeringrelativetootherareasofengineering.Bothferociouslycondemnsoftwareengineeringresearchersfornotvalidatingtheirresearchideasandbothhavebeeninvaluablemakingthisahighprofileissue.Thereisagrowingawarenessthatsoftwareengineeringresearchersmustbeeducatedaboutconductingempiricalstudies.Tothisend,KitchenhamandPfleegerwroteaseriesofarticlesforACMSigSoftSoftwareEngineeringNotes.Thesearticlescoveredavarietyoftopicsincludingthelogicalfoundationsanddesignofempiricalstudies,theiroperation,andtechniquesforcollecting,yzingandinterpretingdata.Severalresearchgroupswereinstrumentalinincreasingresearcheraccesstoindustrialdata.Todaywefindmanypaperswithsignificant,detailedaccountsofindustrialexperiencebasedonindustrialdata.OneoftheforerunnersofthisapproachwastheSoftwareEngineeringLaboratoryofNASA,theComputerSciencesCorporation,andtheUniversityofMaryland[7].Finally,manyfineresearchershavewadedinanddonetheirownempiricalstudies.SystemicDespite,ormaybebecauseof,thestrengthslistedabovetherearesomeseriousproblems.Thesestemfrommisunderstandingsaboutwhatempiricalstudiesareandwhywedothem.Beforewecanimproveouruseofempiricalstudieswehavetoeliminatesomeproblematicpracticesandbeliefs.Oftenwhensomeonesaysthatweneedmoreempiricalstudiesinsoftwareengineering,theyreallymeanthatresearchresultsshouldbeempiricallyvalidated.Theywantresearcherstodemonstratethevalueoftheirnewideasasearlyaspossible.Thisisagoodideaformanyreasons.Webelieve,however,thatitisimportanttorememberthatempiricalstudiescanbeusednotonlyretrospectivelytovalidateideasafterthey’vebeencreated,butalsoproactivelytodirectourresearch.Forexample,incompileroptimizationresearchempiricalstudieshaveidentifiedcommoncodeusagepatterns.Knowing,forinstance,thatbranchingbehaviorisnotusuallyrandom,helpsidentifyandjustifythepotentialvalueofresearchonbranchprediction,aggressivepre-fetching,etc.Inshort,weshoulduseempiricalstudiesalsotodriveourresearchInprogramcommitteemeetingsweoftenhearlengthydiscussionsovertheexactstatisticaltestsusedinastudyorwhetheritwouldn’thavebeenbettertohavedoneonethingoranother.Thesediscussionsreflectavainsearchfortheperfectstudy.Well,we’vedonemanystudiesandwe’veneverdoneoneperfectly!Ofcourse,wewanttoseeproperstatisticsused.Butaswewilldiscussshortly,what’simportantisnotwhetherthestudyistextbookperfect,butwhetherthestudyanditsconclusionstakenasawholeareToomanyempiricalstudiesstudytheobvious.Asthissometimesshowsthattheobviousisn’tsoobvious,we

wouldn’tdiscourageanyonefromngsuchwork.Nevertheless,itmakesuswonder,“ifempiricalstudiesmostlyjustconfirmtheintuitivelyobvious,thenwhat’swrongwithargumentbyintuition”.Clearly,webelievethattherearethingsthataretrue,butthatarenotintuitivelyobvious.Furthermore,webelievethatsomeofthesefindingswillbevaluabletosoftwareresearchandpractice.Therefore,weneedtothinkmuchharderaboutthequestionswearestudyingempirically.Therearetoomanypaperswhoseonlysellingpointisthattheyhavelotsofdata.Dataisnotenough.Justpresentingdataorsimplyapplyingcurve-fittingalgorithmstothemmaybeuseful.Buttheydon’tusuallyhelpusunderstandwhythedataisasitis.Ourdatashouldbeusedtoanswerquestions,notjusttofillgraphs.Amorefundamentalaspectofthisproblemisthatmanyempiricalstudiessimplylackhypotheses.Theyposenoquestions,theyservenowell-definedend.Thusatofthestudytheresearchercanonlypresentobservationsaboutthedata.Allstudies,evencasestudies,shouldbedesignedtoanswersomequestion.Aswesaidearlier,themostimportantpartofnganempiricalstudyisdrawingconclusions.Manypapersfailtodoanythingwiththeirresults.WeneedtolearnsomethingfromeverystudyandrelatethesethingstotheoryandSincemanyresearchersarereluctanttodrawconclusionsfromtheirdata,it’seasytoimaginethattheyaren’ttoohappytogeneralizethemeither.Insteadofspeakingthoughtfullyabouttheirworktheycloaktheresultsin“weaselwords”.Somuchsothat,often,intheend,theysaynothing.There’sobviouslyabalancetobereachedherebecausewedon’twantresearcherstoover-generalize,Butontheotherhand,ifwecan’tdiscusswhatastudy’sresultsmightmeanthenit’shardtomakeprogress. Thegoalofallresearch,notjustempiricalstudies,istoimprovethestateofresearchandpractice.Ifwewanttoempiricalstudiestoimprovesoftwareengineeringresearchandpractice,thentherearetwothingsthatweneedtodobetterinthefuture.Saidsimply,weneedtocreatebetterstudiesandweneedtodrawmorecredibleconclusionsfromthem.CreatingBetterEmpiricalCreatingbetterstudiesmeansngstudiesthathavesomechanceofdirectingourresearch.Itimpliesthatwemustbeclearaboutthegoalsofourstudies,designthemmoreeffectively,and izetheinformationwegetoutofTodothisweshouldconsideratleastthefollowingOurstudiesshouldstrivetoestablishprinciplesthatarecausal,actionableandgeneral.ForafactorAtocause eBit’snecessarythatAandBarecorrelated,thatAprecedesBintimeandthatthereisaconstructive,testabletheoryexplaininghowAaffectsB.WithoutcausalityyouhavenoabilitytocontrolyourAprincipleisactionableifthecausalagentAcanbeeffectivelycontrolled.Forexample,knowingthatlargersystemsnormallyhavemorebugsmaynotbeanactionableprincipleifthedevelopercan’tmakethesystemsmaller.Theprinciplesshouldbeapplicableinaswideavarietyofcircumstancesaspossible.Whenwehaveacausalrelationshipweknowwhysomethinghappens.Ifntisactionable,thenwehaveaknobthatcanbeturnedtocontrolthe e.Ifitisgeneralitwillbeusefultoawiderangeofpeopleinawidesetofcontexts.Ourstudiesshouldtrytoaddressimportantquestions.Therearemanyquestionstoanswer.Answeringsomeofthemwillbecheaperthanansweringothers;usingthoseanswerswillhavemoresignificanceinsomecasesthaninothers.Thisconsiderationimpliesthatweneedtospendagooddealoftimeunderstandingwhywe’rengourstudiesandwhatresultsmightcomefromthem.Individualstudiesarerarely,ifever,unequivocal.Insteadoftryingtosolvelargeissueswithasinglestudywemustattackitwithseveral;eachexaminingdifferent,butcomplementaryaspects.Herethecriticalissueistouseeachnewstudytogenerateandrefineourhypotheses.Empiricalstudiesareexpensiveandtaketime.Ifwemustdomultiplestudies,thenwehavetofindwaystogettheinformationweneedatalowcost.Thismayalsomeanthatwehavetotakesomeshortcutsinourexperimentaldesignsortacklesmaller,morefocusedproblems.Wewillalsoneedtoenlistthehelpofothers.Empiricalstudiesgaincredibilitywhentheyareredoneandrechecked.Weneedtofindwaystohelpotherstoreproduceourresults.CredibleThecredibilityofastudyreferstothedegreeofconfidencewehaveinitsconclusions.Ifstudiesaren’tcredible,thenthetimespentngthemwaswasted.Toimprovethecredibilityofourstudieswemustconsiderseveralissues.Ifwearetryingtoestablishtheexistenceofcausalrelationships,weneedtodesignexperimentswithhighvalidity.Validity,aswewillexplainlater,isacharacteristicofanempiricalstudyandisthebasisofestablishingcredibleconclusions.Therearethreetypesofvaliditythatareparticularlyimportant:internal,external,andconstructvalidity.

Ourstudies(nomatterhowtheyaredone)shouldalwayshavehypotheses.Witheverystudywemustdefinewhatwearecomparingandwhy.Oftenastudywon’tbepowerfulenoughtoshowacausalrelationship.Still,inmanycaseswecanpositseveralalternativeexplanationsforthedataandthenuseotherdatatodiscreditthem.Thisstilldoesn’tshowcausality,butitcanatleastremoveobviousalternativeexplanationsfromWeshouldavoidthetemptationtomeasureeverythingtothefinestpossibleprecision.Sometimesitwillbeenoughtoidentifyanupperandlowerbound;othertimesitwillbeenoughtomeasureatagrossresolution.Thedefinitionofadequateprecisionwilldependontheproblem,butusingcoarsemeasurementsmaybeonewaytolimitstudycosts,whilestillgettingimportantinformation.Ourdataandproceduresneedtobemadepublicsothatotherscanunderstand,yzeandpossiblyreplicateourstudies.Frankly,thiscanbereallydifficult,andwehaven’talwaysmanagedtokeepupourselves,butwebelieveit’sworththeeffort.Inourcareerswe’vedesignedandconductedanumberofstudies.Nonehavebeenwithoutflaws.Ourconclusionisthatnostudyisperfectandthattherealchallengeistocreate,designandconducthigh-impact,credibleThisinvolvesmanagingtrade-offsinsuchawaythatweaccuracyofinterpretation-theresultsweseearenotreallytheresultofsomeunknowninfluence,relevance-ourresults lussomethingimportantaboutsoftwareengineering,andimpact-ourresultsaffectthepracticeoforresearchintosoftwareengineeringsubjectresourceconstraints-studiesareexpensive;wemustworkwithinresourcelimitations,andrisk-studies,especiallythosedoneinindustry,candisruptorputatriskindustrialpartners;wemustminimizetheseproblems.THESTRUCTUREOFANEMPIRICALInthissectionwediscussthestructureandcomponentsofempiricalstudies.Weexpectthatgoodempiricalstudieswillhaveeachofthesecomponentsandthatpaperswrittenaboutthestudieswilldiscussthemaswell.Thesecomponentsare:researchthreatsto ysisandpresentation,resultsandResearchAllstudiesfocusonaproblem.Heretheproblemisdefinedanditsterminologyexplained.Thissectionlinksthestudygoalstowhat’scurrentlyunderstoodabouttheproblem.Thissectionhastwoparts.ProblemDefinition:Wedefinetheproblemandexplainit'simportantterminology.ResearchReview.Weprovidethehistoricalcontextsurroundingtheproblem.Wedescribewhatweknowabouttheproblem,whathasbeendonepreviously,whatquestionsstillremaintobeansweredandwhatquestionswillwebefocusingon.Hypothesesareessential.Theystatetheresearchquestionsweareasking.Sometimesthereisconfusionsurroundingthetermhypothesis.Infacttherearereallytwokindsofhypotheses.Thetrickistothinkofastudyasaprocedureformakingacomparison.Therefore,westartatwithhigh- questionsandrefinethemintolow-level,concretequestions.hypothesesarehigh-level,naturallanguagestatementsthatareusuallystatedineverydayterms.Theysaythingslike,“meetingsareanindispensablepartoftheinspectionprocess”.Concretehypothesesarestatedintermsofthestudy’sdesign.Theymaysaythingslike,“teamswhodoinspectionswithmeetingsfindmoredefectsthanteamswhonspectionswithoutthem.”Webeginbystatingourhypothesesfirstineverydayterms.Thenwetranslatethemtotermsthatexistinthestudy’sdesign.Tothedegreethatthismapisdonewell,comparisonsmadeatthelevelofconcretehypothesescanbemappedbacktothecomparisonsmadeatthelevelStudyAstudy’sdesignisadetailedplanforcreatingthedatathatwillbeusedtotestitshypotheses.IthasseveralOnecomponentisasetofvariablesthatlinkcausesandeffects.Typically,therearetwokindsofvariables:dependentandindependent.Independentvariablesareattributesthatdefinethestudysetting.Insomecases,especiallywhencomparingtwosituations,thesevariablesareactivelymanipulated.

Dependentvariablesareend-processoutputswhosevaluesareexpectedtovarypredictablywhenthevaluesofindependentvariableschange.Thestudydesignmayalsoincludeaplanforsystematicallymanipulatingtheindependentvariableswhileobservingthedependentvariables.Thefinalcomponentistheoperationalcontextofthestudy.Thisisadescriptionofthephysical,inlectualandculturalsurroundingsinwhichthestudytakesplace.Itisincludedsothatthestudy’suserscanbetterinterprettheThreatstoThreatstovalidityareinfluencesthatmaylimitourabilitytointerpretordrawconclusionsfromthestudy’sdata.Thereareatleastthreekindsofvaliditythatmustbeprotectedfromsuchthreats.Constructvaliditymeansthattheindependentand accuray Internalvaliditymeansthatchangesinthedependentvariablescanbesafelyattributedtochangesintheindependentvariables.Externalvaliditymeansthatthestudy’sresultsgeneralizetosettingsoutsidethestudy.DataysisandTwogeneralapproachestopresentingandyzingdataarecalledtativeandQualitativeysis.tativeyses,asthenamesuggests,dealmainlywithcomparingnumericdata.Thecomparisonsaretypicallyaimedatrejectingornotrejectinganullhypothesis.Twoofthetoolsusedintativeysisarehypothesistestingandpowerysis.Hypothesistestingdeterminestheconfidencelevelatwhichthenullhypothesiscanberejected.Theconfidencelevelisameasureoftheprobabilitythatthenullhypothesiswillbeerroneouslyrejected.Somepeoplebelievethatthisconfidencelevelmustbelessthan1in20or0.05foraresulttobesignificant.Itdoesn’thavetobe.Insituationswheredataisplentifulandmeasurementsprecise,higherconfidencelevelsmaybecalledfor.Sincedataisoftenlimitedandmeasurementimpreciseinstudiesofsoftwareengineering,lowerconfidencelevelsmaybejustified.Inanyevent,wesuggestthatresearchersreportth