開(kāi)放平臺(tái)

ApsaraCloudPlatform第一頁(yè)，共二十六頁(yè)。AboutAliyunChina’slargestcloudserviceprovider100softhousandsofcustomersBillionsofaccesseseveryday第二頁(yè)，共二十六頁(yè)。Alibaba-operatedIDCPartner-operatedIDCApsaraCloudPlatformACE3rd-partyPlatformServicesMap,Mail,Search3rd-partyApplicationServicesCustomersISVandSIDevelopersProvidingFoundationServicesoftheCloudEco-system第三頁(yè)，共二十六頁(yè)。PaybyusageElasticitySafety(like“tapwater”)TheNatureofCloudComputingScale大規(guī)模Economy低成本PublicUtility服務(wù)運(yùn)營(yíng)Internet-scalecomputing2.5EBgeneratedperday,doublingevery40monthsBillionsoftxnsonTaobaoeveryday,mustbeprocessedin6hoursEconomymeansmorethanlowpricesLeadingtobehaviorchanges(like“telephone”)Keyisscheduling(like“powergrid”)第四頁(yè)，共二十六頁(yè)。TwoDesignPrinciplesLarge-scalegeneralcomputingplatformasthebaseOnesystemsupportingbothofflineandonlineservicesMulti-tenancy,resourcesharing,loadshiftingWeb-basedAPIasthedeliverymechanismOnlineactivation,pay-by-usageLocation-transparency第五頁(yè)，共二十六頁(yè)。LinuxClusterIDCResourceManagement(伏羲)Security(鐘馗)RPC(夸父)Naming/Coordination(女?huà)z)ClusterDeployment(大禹)ClusterMonitor(神農(nóng))Distributed(盤(pán)古)JobScheduling(伏羲)ACEOSSOTSODPSECS/SLBRDSMap,Mail,Search,etcCloudMartOtherCloudServicesOSPS第六頁(yè)，共二十六頁(yè)。CloudComputingServicesElasticComputing彈性計(jì)算ECS:virtualizedinstancesofserversthatcanbecreatedandtailoredtomeetapplicationrequirementsSLB:softwareloadbalancingtechnologythatcanelasticallyexpandservicecapacityondemandACE:ConvenientandefficientexecutionenvironmentforWebservices,supportingJava,PHP,Node.jsStorageandDatabases海量存儲(chǔ)和數(shù)據(jù)庫(kù)Large-scaleComputing大規(guī)模計(jì)算第七頁(yè)，共二十六頁(yè)。CloudComputingServicesElasticComputing彈性計(jì)算StorageandDatabases海量存儲(chǔ)和數(shù)據(jù)庫(kù)OSS:large-scaleobjectstorageserviceforunstructureddatasuchasphotos,music,orvideoOTS:largescalestorageserviceforstructuredorsemi-structureddatastorageandreal-timequeryRDS:managedinstancesforrelationaldatabaseswithautomaticbackupandfailoverLarge-scaleDataComputing大規(guī)模計(jì)算第八頁(yè)，共二十六頁(yè)。AComparisonofStorageandDatabaseServicesOSSOTSRDSDataModelUnstructuredSemi-structuredFully-structuredTargetDataVolume~10PB~100TB/table~TB/dbTxnSupportNoneLimitedsupportFullSupportProgrammingInterfaceRESTfulAPIRESTfulAPISQL第九頁(yè)，共二十六頁(yè)。CloudComputingServicesElasticComputing彈性計(jì)算StorageandDatabase海量存儲(chǔ)和數(shù)據(jù)庫(kù)Large-scaleComputing大規(guī)模計(jì)算ODPS:large-scaledatabatchprocessingandcomputation,supportingSQLandMapReducestyleprogramminglanguagesOSPS:streamdataprocessingservice,supportingSQL-likequerylanguageandautomaticfailurerecovery第十頁(yè)，共二十六頁(yè)。ApsaraTechnicalHighlightsAcommonplatformsupportingbothofflineandonlineservicesSearch:24Bpagesprocessed,13BonlineindexMail:100Mmailsreceived,10Mmailssent,10mslatencyCapability-basedsecuritymanagementframework,enforcingthePrincipleofLeastPrivilegeDistributeddeployment,monitoringanddiagnosticsZeroSPOF(single-point-of-failure):availability>99.9%Alldatahas3replicas:datareliability>99.99999999%第十一頁(yè)，共二十六頁(yè)。5K2013/08/15:First-ever5000-nodeApsaracluster(ODPS)wentintoproduction100KCPUcores,100PBrawstorageProcessingpetabytesperday2013/09/24:OpenedaccesstoODPS

for4universities&researchinstitutionsSorting100TBin30minutesCurrentknownrecord:72minutes(Yahoo!,2013/07/03)第十二頁(yè)，共二十六頁(yè)。Pangu:Large-scaleDistributedMaster-SlaveArchitectureMasterformetadatamgmt,Slave(ChunkServer)forIOmgmtPaxos-basedmulti-masterarchitecture,failurerecoverytime<1minuteEnd-to-endinlinechecksumScalesto1billionfilesCSCSCSCSCSMMMPaxos第十三頁(yè)，共二十六頁(yè)。SeparatedIOPipelineandStorageMgmtAdaptiveIOPipelineReplicationmaster:chunkservervsclientReplicationpolicy:chainingvsstar-replicationChunkingpolicy:fixed,variable,orRAIDDurabilityguarantee:txnloggingvssequentialwriteCommonStorageManagementPhysicalIOmanagementPriorityandQoSBackgroundre-replicationChunkplacement第十四頁(yè)，共二十六頁(yè)。StagedEvent-drivenPhysicalIOMgmtChunkServerwouldrearrangeIOrequeststosupportpriority,QoS,andreduceIOseekoverhead第十五頁(yè)，共二十六頁(yè)。DistributedRe-replication1TB1TB1TBTypical:Mirroring(10hours)1TB1TB1TBPangu:Distributedre-replication(20min,50-nodes)1TBIntelligentschedulingBalancedstorageBandwidththrottlingMinimizingdataloss第十六頁(yè)，共二十六頁(yè)。RAIDBuiltintothecoresysteminsteadofanadd-onlayer(asinHDFSRAID)Bettermanagementofdataintegrity,recovery,andchunkplacementSynchronousredundancyblockgenerationLow-latencyfailurerecoverySmall第十七頁(yè)，共二十六頁(yè)。FuxiMasterFuxiMaster...App

MasterAPP

WorkerApp

MasterAPP

Worker...Client.........TuboTuboJobcontrolResourcerequestsNodecontrolJobsubmissionAPP

WorkerAPP

WorkerAPP

WorkerTuboTuboFuxiResourceSchedulingMulti-dimensionresourcesElasticquotaCGroup-basedisolationFuxiMasterHAAppMasterfailoverIncrementalscheduling第十八頁(yè)，共二十六頁(yè)。FuxiJobProgrammingModelJob:ADAGVertex:TaskEachtaskmayhavemultipleinstancesbasedoninputdatachunksEdge:dataflow,eachtaskmayhavemultipleinput/outputflowsAdataflowconnectingtwotasksrepresentsdatashufflingDAinput1input2output2BCDAABBCDoutput1MapReduceoutputinputMapMapReduceMapReduceisadegeneratedcase第十九頁(yè)，共二十六頁(yè)。Example:FindBest-SellersSELECTprod_id,Sum(count)ASquantityFROMordersGROUPBYprod_id

ORDERBYquantityDESC;order_idprod_idunit_pricecount0001042151003000203343101000003012345010000404215805…………prod_idquantity0251831790075632845109641229430421520043……orders第二十頁(yè)，共二十六頁(yè)。FuxiMapReduceComparisonofJob-executionPlanInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityInputMMR1R2R2prod_idcountprod_idquantityprod_idquantityM2prod_idquantity第二十一頁(yè)，共二十六頁(yè)。FuxiMapReduceComparisonofJobExecutionR1MR1MM2R2R2第二十二頁(yè)，共二十六頁(yè)。ABriefHistory02/04/2009FirstlineofApsaracode08/27/2010Apsarabecamethecommonplatformforsearch,mail,storage,VMandlarge-scaledataprocessing.07/28/2011OfficialWebsiteofAliyunwentonline.ECSbecamethefirstAliyunserviceopentopublic.08/15/20135000-nodeApsaraclusterwentintoproduction.10/24/20133rdAliyundevconferenceheldinHangzho