才云科技CTO/鄧德源2016-4-222016-4-22AndAndwhenIwasyounginthegoodolddays…kSend()ApplicationReboot()vDiskkvkSend()ApplicationReboot()vDiskkvApplicationRead()Read()Reboot()Send()HaHardwareBaremetalRead()Read()…vvHardwareHyperviHypervisorHaHardwareVirtualMachineContainer:Runtime+ImageRead()Read()Reboot()Send()RoutingRoutingProcessCapabilities/SeccompChildNetNSChildNetNSChildMntNSChildMntNSleUsingcontainersforadecadeRunning2billioncontainersaweekSolvesapplicationmigrationnightmareSavesbilliondollarsayearDocker:5xyearlygrowthrate1SREhandles~10,000machineswith99.999%reliabilityNOteamdedicatedforcontainerstudyClusteringistheNOteamdedicatedforcontainerstudyHUNDREDSofengineersbuiltTHREEclustermanagersystemsHUNDREDSofteamsbuildingecosystemsClusteringistherealvalueinbuildingseriousproductioncontainersystemsnHeredeclarative>imperativesimple>complexlabels>hierarchylegacycompatibleextensibleandpluggableapplicationcentricThingsThingsSwarm,MesosandKubernetesdoincommon(moreor?Servicediscovery?Schedulingandorchestration?Scaling?Cross-hostnetworkingandcommunicationHoweverHowever,?SwarmisDockernativeBUThardtoextend?whatifIwanttouserkt?vendorlockin!?Mesosisnotdesignedforcontainers?forresourcesharingMoreconcretely,Kubernetesempowersmore:LessenLessenlearnedfromBorg?Jobsareusuallygrouped?e.g.logoffloadingVolumesHowtogroupresources?AppcontainersConfigAppcontainers?Allowteamstodevelopdistinctpartofapplication?improverobustness?e.g.logcanbeoffloadedevenifcontainercrashes?Atomicscheduling?C1:1core,C2:2cores?M1:2cores,M2:8cores?C1->M1?bookingenv:bookingenv:bookingenv:bookingenv:Howtomanagemassiveresourcesinaflexibleway??LabelsanditsqueryAPI?Selectorsappapp:bookingteam:travelversion:v1env:prodpprodversion:v2team:travelapp:appapp:mongoteam:storageenv:prodappapp:uatteam:travelversion:v2bookingebookingeapp:mongoebookingebookingeapp:mongoestorageprodversion:v1team:travelapp:app:bookingteam:travelversion:v2mode:uatprodversion:v2team:travelapp:HowtoperformA/BDeployment??Labels?Replicationcontrollers?ServicesReplicationReplicationControllerLabelselector:“version:v1”Replicas:N>0ServiceService:“booking”Labelselector:“app:booking”ReplicationReplicationControllerRCselector:“version:v2”Replicas:0>Npprodteam:HowHowtodoservicediscoveryforexternalservices??UseservicesandendpointstogetherProdProdEndpoint:Name:“Oracle”IP:10.254.1.1QAQAEndpoint:Name:“Oracle”IP:192.168.1.1 Howtodealwithconfigurationsvaryingindifferentenvironments?APPinPod:ref:ENV[ORACLE_PASSWD]Singleimage,decoupledfromvaryingconfigurationsConfigMapinPod:ORACLE_PASSWD:“7h6#f)”__JETTYCONFIGPATH__ConfigMapinUAT:_ORACLEPASSWD:“123456”___JETTYCONFIGPATH__apiVersion:apiVersion:v1kind:ReplicationControllermetadata:name:caicloud-cluster-managerspec:…volumes:-name:aliyun-api-keyssecret:secretName:aliyun-api-keys-name:caicloudapp-ssl-certsecret:secretName:caicloudapp-ssl-certHowtoNOTletdockerpersistmycredentials??SecretsapiVersion:v1kind:Secretmetadata:name:aliyun-api-keysdata:api-client1:…api-client2:…aliyun-api-keys:…Howtoperformfine-grainedresourcecontrolandaccesscontrol??NamespacesandserviceaccountsHowtohandleservicesorapplicationsthatarestateful??L7loadbalancer?Ingresscontroller?nodeaffinity?PetSet(coming)?Pod?LifecyleinterfacesHowtoautomaticallycreate,delete,andallocatestoragees?PersistentvolumesandclaimsOurPracticeonUsingKubernetestoBuildClusterManagementSystemClustervsContainerastheoperationunits?Additionalhigher-levelmanagementE.g.,cloneentireclustersandensureholisticconsistency(e.g.,config)?CLaaS:ESclusterwithdata,client,andmasternodes+offloaddataprocessing?CaaS:dockerrunelasticsearch:1.7.4Hostclusterexposurevscontainerblackbox?CLaaS:requiresdedicatedclusters;additionalhostinformationandaccess?CaaS:hostsareabstractedaway;obscuresdebugging,tooling,andcustomizationScenario:?Tomcat,RedisCluster,ElasticSearch,MongoDB?Wanttosetupdevelopment,testing,andproductionenvironmentsandbe1)fastand2)consistentCaaS:image-levelconsistency?HowtohandledifferentIPs,references,configfiles,dependencies?TheMOSTtroublesomepartunsolved!CLaaS:system-levelconsistency?Tomcat1,tomcat2andtomcat3haveconsistentimages?IPsuseconsistentnames(evenforexternalservices)?Configusesconsistentreferences,dependenciesarerespectedbackendfrontend Elastic Searchbackendfrontend Elastic SearchJetty VPSVPSHowCloudTreadsTheirCustomersToday?Usershavetomaplogicalsolutionstothelistofresources/containers?ContainersDONOThelp!mid-tierMongoageageVPVPSCodisutionManagerSolutionsasstClassCitizenCodisKeybenefits?Includesbothworkloadsandinfrastructureandthetopology?Additionalhigherlevelmeta-dataandmanagementinterfacessbackendMongo storagestoragemid-tier Elastic Search storagestoragefrontendRateLimitingSSLTerminationSolutionManagerPublicAPIPagingPublicAPIClusterManagerPublicAPIRateLimitingSSLTerminationSolutionManagerPublicAPIPagingPublicAPIClusterManagerPublicAPIAuthenticationAuthenticationAuthenticationTransformationLoggingAuthorizationPagingPrivateAPISolutionManagerPrivateAPI……UserUserbrowserProblems:ginx-Duplicatedfunctionalities-Systemtendstobemonolithic-LanguageandEnvironmentDependent-ComplexfrontendlogicduetovariedAPIsClusterClusterManagerPrivateAPI……PagingPagingThirdPartyAPICaicloudadminbrowser Cargo(registry)SolutionM(compositcloudnaerCubernetesClustersMCaicloudadminbrowser Cargo(registry)SolutionM(compositcloudnaerCubernetesClustersMonitoringLoggingerCubernetesClustersMonitoringLoggingMonitoringLoggingerCubernetesClusters……AdminConsoleClusterManager(federation,isolation,logging,HA,lifecycle)APIGatewayanagereapps,tive)AccountingPaging……pull/pushimageerbrowserCubernetes/CaicloudAPIcallserCubernetesClustersMMonitoringLogging-Restartcreatingcluster-Easy,butbaduserexperience-Restartcreatingcluster-Easy,butbaduserexperience-Pickupwhereitleft-Errorprone-Gracefultermination-1ministoosmallandtoolarge-Anewdockercontainer-SpawnanewdockercontainerClusterManagerServerClusterManagerServerClusterManagerValidatorClusterManagerExecutorClusterManagerExecutorClusterManagerServerRollingupdateisgreat,when:-youwanttotestmultipleversionsofcodeorconfiguration-youwanttoupdateapplicationwithoutserviceinterruptionClusterClusterManagerServeretcdetcdcluster