PerformancecomparisonofdynamicwebplatformsVarshaAptea,1,TonyHansenb,*,PaulReeserbaDepartmentofComputerScienceandEngineering,IndianInstituteofTechnology-Bombay,Mumbai400076,IndiabAT&TLabs,200LaurelAve,Middletown,NJ07748,USAReceived7August2002；accepted7August2002AbstractOverthelastfewyears,theWorldWideWebhastransformeditselffromastaticcontent-distributionmediumtoaninteractive,dynamicmedium.TheWebisnowwidelyusedasthepresentationlayerforahostofon-lineservicessuchase-mailandaddressbooks,e-cards,e-calendar,shopping,banking,andstocktrading.Asaconsequence(HyperTextMarkupLanguage)HTMLfilesarenowtypicallygenerateddynamicallyaftertheserverreceivestherequest.FromtheWeb-siteproviderspointofview,dynamicgenerationofHTMLpagesimpliesalesserunderstandingoftherealcapacityandperformanceoftheirWebservers.FromtheWebdeveloperspointofview,dynamiccontentimpliesanadditionaltechnologydecision:theWebprogrammingtechnologytobeemployedincreatingaWeb-basedservice.SincetheWebisinherentlyinteractive,performanceisakeyrequirement,andoftendemandscarefulanalysisofthesystems.Inthispaper,wecomparefourdynamicWebprogrammingtechnologiesfromthepointofviewofperformance.Thecomparisonisbasedontestingandmeasurementoftwocases:oneisacasestudyofarealapplicationthatwasdeployedinanactualWeb-basedservice；theotherisatrivialapplication.Thetwocasesprovideuswithanopportunitytocomparetheperformanceofthesetechnologiesattwoendsofthespectrumintermsofcomplexity.Ourfocusinthispaperisonhowcomplexvs.simpleapplicationsperformwhenimplementedusingdifferentWebprogrammingtechnologies.Thepaperdrawscomparisonsandinsightsbasedonthisdevelopmentandperformancemeasurementeffort.q2002PublishedbyElsevierScienceB.V.Keywords:Dynamic；Web；CommonGatewayInterface；FastCGI；C；Java；Servlets；JavaServerPages；Performance；Comparison1.IntroductionTheWorldWideWeb(WWW)firstemergedadecadeagoasamediumtorenderhypertextdocumentsthatwerestoredontheInternet,onauserscomputer,usingspecialsoftware(thebrowser)andanewprotocol(HTTP:HyperTextTransferProtocol).Forthefirstfewyears,theWWWgrewprimarilyasanewmediuminwhichstaticcontentcouldbepublished,andinformationshared.ThecontentwaspublishedintheformofHTML(HyperTextMarkupLanguage)files,whichwereservedbyWebservers,onrequestsfrombrowsers.However,overthelastfewyearstheWWWhastransformeditselffromastaticcontent-distributionmediumtoaninteractive,dynamicmedium.ContentontheWebisnowoftenpersonalized,andthereforedynamicallygenerated.TheWebisnowwidelyusedasthepresentationlayerforahostofon-lineservicessuchase-mail,e-cards,e-calendar,andaddressbooks,shopping,banking,andstocktrading.Asaconsequence,theHTMLfilesthatarerenderedbytheclientsbrowserarenowtypicallygenerateddynamicallyaftertheWebserverhasprocessedtheusersrequest.ThisdynamicgenerationofHTMLfileshasnothappenedwithoutanassociatedperformancecost.JustwhenInternetusersweregettingaccustomedtoclick-and-waitondial-uplinesduetographics-richWebsites,dynamicallygeneratedcontentstartedproliferatingontheWeb.Nowusersmustwaitnotonlyforthenetworkdelaybutalsofortheserver-sideprocessingdelayassociatedwithservingarequestdynamically.Inmanycases,thisisturningouttobethelargestcomponentofthedelay.FromtheWeb-siteproviderspointofview,dynamicgenerationofHTMLpagesimpliesalesserunderstandingoftherealcapacityoftheirWebservers.Thevendor-providedhits-per-secondcapacityoftheWebserverisnolongerenough,asthisonlypertainstostaticHTMLfiles.ComputerCommunications26(2003)888898/locate/comcom0140-3664/03/$-seefrontmatterq2002PublishedbyElsevierScienceB.V.PII:S0140-3664(02)00221-91ThisworkwasdonewhilethisauthorwaswiththeNetworkDesignandPerformanceAnalysisDepartment,AT&TLabs,Middletown,NJ07748,USA.ThisauthorhaspreviouslypublishedunderhermaidennameofVarshaMainkar.*Correspondingauthor.E-mailaddress:(T.Hansen).FromtheWebdeveloperspointofview,dynamicWebcontentimpliesanadditionaltechnologydecision:theWebprogrammingtechnologytobeemployedincreatingaWeb-basedserviceorproduct.Thisdecisionisbasedonseveralfactors.Amongthefactorsconsideredareeaseofprogramming,richnessoffeatures,maintain-ability,reliability,andperformance.SincetheWebisinherentlyinteractive,performanceisakeyrequirement,andoftendemandscarefulanalysisofthesystems.Inthispaper,wecomparetheperformanceoffourWebprogrammingtechnologies,namelyJavaServlets,JavaServerPages,CGI/CandFastCGI/C(asmallerversionofthisstudywaspresentedinRef.1).ThestudywasmotivatedbyarealneedtomakeatechnologychoicefordevelopingsoftwarethatwouldsupportaWeb-basedservice.Astudyofexistingliteratureshowedvaryingconclusionsabouttheperformancesuperiorityofonetechnologyovertheother.ProponentsoftheJavaplatformconsistentlyclaimsuperiorperformanceofservlettechnologymainlyduetotheeliminationoftheoverheadofprocesscreation2,3.Ontheotherhand,astudydoneinRef.4thatcomparesCGI,FastCGIandservletsshowedservletstobetheslowestofthethree.AsimilarcomparativestudyhasbeendoneinRef.5,focusingonWeb-to-databaseapplications,whichshowedJavaservletstobebetterthanCGIprograms.Further-more,although4brieflyprovidessomeexplanationabouttheresults,anin-depthanalysisandinsightabouttheresultswasnotavailableintheliteraturethatwesurveyed.Oneconclusionfromthevariabilityoftheresultswasalsothattheperformanceofthetechnologydependsontheapplication.Inthispaper,thecomparisonisbasedontwocases:oneisacasestudyofacomplexapplicationthatwasdeployedinanactualWeb-basedservice；theotherisatrivialapplication.Themethodologyofperformanceanalysiswasstresstestingandmeasurement.Performancemeasure-ment(ratherthanmodeling)mademostsenseinthiseffort,sinceaquickturnaroundofresultswasnecessaryandtheaccuracyofresultswasrequiredtobehigh.Thetwocases(i.e.thecomplexandthetrivial)provideduswithanopportunitytocompareperformanceofthesetechnologiesattwoendsofthespectrumofapplications,intermsofcomplexity.Theperformanceorderofdifferenttechnologiesisseldomabsoluteitdependsgreatlyonthenatureoftheapplication.Ourfocusinthispaperisonhowcomplexvs.simpleapplicationsperformwhenimplementedusingdifferentWebprogrammingtechnologies.Amongthepaperssurveyed,webelievethisistheonlypaperpresentingasystematicbottleneckanalysisofeachofthetechnol-ogies,andtheonlyonethatdemonstratesthattheperformancerankingcouldbereversed,basedontheapplication,whileprovidinginsightonwhythishappens.Themainobservationsfromthisworkwereasfollows:Ingeneral,FastCGIoutperformedCGI,whileJSPoutperformedJavaservlets.Inthecaseofacomplexapplication,theCGI-basedtechnologiesoutperformedtheJava-basedtechnologiesbyafactorof34,withJavaperformancelimitedbysoftwarebottlenecksintheJVM.Inthecaseofatrivialapplication,therelativeperformancewasreversed,asJavaoutperformedCGIbyafactorof23.Therestofthepaperisorganizedasfollows:inSection2weprovidethemotivationforconductingsuchacomparativestudy,andinSection3wedescribebrieflythetechnologiesthatwecompared.Section4describestheperformancemeasurementandanalysismethodology,Section5describesthecase-studytestingresultsindetail,andSection6describestheresultsoftestingatrivialapplication.Finally,Section7summarizesourresults,andSection8providessomeconcludingremarks.2.MotivationTheapplicationcontextforthiscasestudywasanewWeb-basedmessagingservice.Suchaservicewouldinvolveaback-endthatwouldincludethecoremessa-ging-relatedservers(IMAP,POP,LDAP,SMTP),andafront-end,orapresentationlayerthatwouldserveasmiddlewarebetweentheWebbrowserandthemessagingserver.Thecoreserverswerechosentobeestablishedmessagingproducts.ThepresentationlayerwouldconsistofaWebserver(off-the-shelf)andanewdynamicpagegenerationenginetobedeveloped,whichwouldReaddatasentbytheuser,throughtheHTTPprotocolDothenecessaryprotocolconversions(toIMAP,LDAP,)CarryouttheactionencodedintheHTTPrequest,usingtheappropriateprotocolwiththeback-endserversFormattheresultasaWeb-page,andReturntheresulttotheusersWebbrowserthroughtheWeb-serverOnewaytowritesuchaprogramthatgeneratesWebpagesdynamicallyistocodethelogic,andthenembedprintstatementsintheprogramthatwriteoutstaticHTMLcode,alongwithprintingotherstringvariablesthathavebeenpopulateddynamically.Abetterandmorepopularwayisusingtemplates.Themiddlewarewestudiedusedatemplate-baseddesignforproducingdynamicWebpages.ThismethodinvolveswritingWebpagetemplateswhichhavestaticHTMLcodethatspecifythedesignoftheWebpage,interspersedwithscriptingtagsthatareinterpretedbyatemplateserverprogram.Thesetagsinstructthetemplateserverprogramontheactionthatistobetaken(e.g.populatingafieldwiththesubjectlineofamessage)onreadingthetag.Thetemplateserverprogram,therefore,V.Apteetal./ComputerCommunications26(2003)888898889ParsesthespecifictemplateaccessedbyauserrequestInterpretsthetagsandcommunicateswiththeback-endserversPopulatesthefieldsthataretobydynamicallypopulatedTheresultisadynamicallygeneratedWebpage.ThearchitectureofsuchasystemisshowninFig.1.Giventheacceleratedtime-to-marketgoalsandlimiteddevelopmenttime,thenaturalchoiceoftechnologyforthetemplateserverwastheoneperceivedtobepowerful,feature-richandyeteasytouseanddeploynamely,Javaservlets.Aninitialeffortwasdonetoquantifytheperformanceofthispagegenerationengine,toseewhetheritcouldmeettheexpectedperformancerequirements.Weconductedaseriesofstresstestsusingacommercialloaddrivertogeneraterepeatedrequeststotheservletengineatvariouslevelsofconcurrency(simulatedusers).ThetestconfigurationconsistedofaWindowsNTserverrunningtheloadgenerationscripts(driver),aSolarisserverrunningthefront-endsoftware,andaSolarisserverrunningtheback-endapplication(forthistest,aPOP3/IMAP4mailserverandanLDAPdirectoryserver).Hardware(CPU,memory,disk,I/O)andsoftwareresourceconsumptionsweremeasuredonallmachines.Inaddition,end-to-enduser-perceivedresponsetimesweremeasured.Thedriverscriptsemulatedaprescribednumberofconcurrentusersrepeatedlygeneratingthesamerequest(e.g.readmessage,sendmessage,etc.).Thenumberofconcurrentsimulateduserswasvariedfrom1to20.Thenumberofrepeatedrequestsperuserateachconcurrencylevel(typically2000)wassufficienttoachievestatisticalstability.Thetestswereruninstressmode；thatis,assoonasauserreceivesaresponse,itimmediatelysubmitsthenextrequest(i.e.withnegligibleclientdelay).EachoftheNsimulatedusersdoessoindependentlyandinparallel.Asaresult,thecon-currencylevel(i.e.thenumberofrequestsinthesystem)equalsNatalltimes.Theresultsofthestresstestsforaparticularrequesttype(reada20KBmessage)areshowninFigs.2and3.Inparticular,Fig.2plotstheend-to-endresponsetime(normalized)ontheleft-handaxis,andthefront-endCPUutilizationontheright-handaxis,asafunctionoftheconcurrencylevel.Ascanbeseen,theresponsetimecurvebeginstoridealongalinearasymptote(shownbythedottedline)afteronlysevenconcurrentusers.Thatis,responseFig.1.Templateserverarchitecture.Fig.2.Resultsofinitialstresstestsoftemplateserver,plottedvs.concurrentusers.V.Apteetal./ComputerCommunications26(2003)888898890timeincreasesproportionallywiththenumberofusers,indicatingsaturationinaclosedsystem6.Additionally,CPUutilizationlevelsoffafter11usersat6570%(indicatinganon-CPUsystembottleneck).Equivalently,Fig.3plotsend-to-endresponsetimeasafunctionofthroughput(requests/s).Ascanbeseen,themaximumsystemthroughputpeaksatabout2requests/sec,andthendegradesunderoverloadtoabout11/2requests/s.Inotherwords,therewasactuallyadropincapacityof25%(aconcurrencypenalty),likelyduetocontextswitching,object/threadmanagement,garbagecollection,etc.Asizinganalysisbasedontheexpectedcustomergrowthandusagebehavior,togetherwiththeseinitialcapacityresults,suggestedthattheresultinghardwarecostswouldbeprohibitivelylarge.Itwasalsoclearthatthescalabilityofthisapplicationwaspoor.Theresourceconsumptionresultsdemonstratedthattheapplicationcouldnotmakefu