三、并行計算模型 七、附錄1：Hadoop安裝.................................................................................................... 9.運試程 Hadoop就是一個實現(xiàn)了云計算系統(tǒng)的開源系統(tǒng)，包括并行計算模型目也很豐富，包括ZooKeeper，Pig，Chukwa，Hive，Hbase，Mahout，flume等。hadoop擅長日志分析，就用Hive來進行日志分析，2009年時就有非編程人員的30%的人使用HiveQL進行數(shù)據(jù)分析；淘寶搜索中的自定義篩選也使用的Hive；利用Pig還可以做高級的數(shù)據(jù)處理，包括、LinkedIn上用于發(fā)現(xiàn)您可能認識 40%的Hadoop作業(yè)是用pig運行的，包括郵件的識別和過濾，還有用戶特征建模。mahout設(shè)想一下這樣的應(yīng)用場景.100Msql文件.我現(xiàn)在想在不導相同關(guān)鍵字的記錄那么有幾種方式,一種是直接用linux令grep還有一種就是通過編文件,100M的數(shù)據(jù)庫備份.上1G,1T甚至1B上面2畢竟單臺服務(wù)器的性能總有其上限那么對于這種超大數(shù)據(jù)文件怎么得到我們想要的？有種方法就是分布式計算,分布式計算的就在于利用分布式算法把運行在單臺機器上的程序擴展到多臺機器上并行運行.從而使數(shù)據(jù)處理能力成倍增加.但是這種分布式計算一Haddop就是為了解決這個問題誕生的。Haddop可以很輕易的把很多l(xiāng)inux的廉價pc組成分布式結(jié)點,然后編程人員也不需要知道分布式算法之類,只需要根據(jù)mapreduce的規(guī)則定義好接口方法,Haddop。它會自動把相關(guān)的計算分布到各個結(jié)點上去,然后得出結(jié)果。例如上述的例子：Hadoop1PBHDFS中mapreduce,key,每行的內(nèi)容定義為value,然后進行正則匹配,reduce聚合起來返回.Hadoop就會把這個程序分布到N個結(jié)點去并行的操作.內(nèi).11相加得出計算結(jié)果,1億.但是計算機不知道.那么單臺計算機處理的方式做一個一億次的循環(huán)每次結(jié)果+11萬臺計算機,每個計算機只需要計算1萬個11萬臺計算機得到的結(jié)果再相加式計算，Hadoop中有兩個最的模塊，hdfs文件系統(tǒng)以及mapreduce并行計算模型。NameNode相當于一個，它管理集群內(nèi)的DataNode，當客戶發(fā)送請求過來后，NameNode會根據(jù)情況指定到哪些DataNode上，而其本身自己并不真實的數(shù)據(jù)。那NameNode怎么知道集群內(nèi)DataNode的信息呢？DataNode發(fā)送心跳信息給 >DataNode >Block2:若為DataNode節(jié)點，那block時，規(guī)則為：副本1，同的節(jié)點上；若不為DataNode節(jié)點，那block時，規(guī)則為：副本1，隨機選擇一個節(jié)點d.向DataNode發(fā)送block1；發(fā)送過程是以流式寫入。1>64Mblock164kpackage劃分NameNodehos2 MapReduce是hadoop的組件之一，hadoop要分布式包括兩部分，一是分布式文件系統(tǒng)hdfs,一部是分布式計算框，就是mapreduce,，也就是說，可以通過mapreducehadoop我：你等一下。讓我來編一個完整的情節(jié)，這樣你肯定可以在15MapReduce的概念應(yīng)用到食譜上。Map和Reduce其實是兩種Map（映射）:MapMap一個洋蔥，Map一一地拿給Map，你也會得到各種碎塊。所以，當你在切像洋蔥這樣的蔬菜時，你執(zhí)行就MapMap操作適用于每一種蔬菜，它會相應(yīng)地生產(chǎn)出一種或多種碎塊，在Map操作中可能會出現(xiàn)有個洋蔥壞掉了的情況，你只要把壞洋蔥丟了就行了。所以，如果出現(xiàn)壞洋蔥了，Map操作就會過濾掉壞洋蔥而不會生產(chǎn)研磨機通常將map操作的蔬菜碎在了一起。MapReduce的一部分，MapReduce我：沒問題。椒醬食譜大受歡迎，于是你想要開始品牌的辣椒醬。假設(shè)你每天需要生產(chǎn)10000妻子：當然不能了，我會雇傭的工人來切蔬菜。我還需要的研磨機，這樣我Map操作。每一個人都將不（每個人工作的地方）上就有了洋蔥MapReduce遺漏的階段—攪拌階段。MapReduce將所有輸出的蔬keykeys都會攪10年計算機出現(xiàn)最多的幾個單詞，看看大家都在研究些什么，那收集好后，該怎么辦呢？方法一：我可以寫一個小程序，把所有按順序遍歷一遍，統(tǒng)計每一個遇到的單詞的方法二：寫一個多線程程序，并發(fā)遍歷。法一多了，須自己同步共享數(shù)據(jù)，比如要防止兩個線程重復統(tǒng)計文件。N臺機器上去，然后把集分成N份，一臺機N個運行結(jié)果進行整合（當然我們也可MapReducecopy程序，如何整合結(jié)架會將map函數(shù)產(chǎn)生的中間鍵值對里鍵相同的值傳遞給一個reduce函數(shù)。//key://value: foreachwordwinvalue:EmitIntermediate(w,"1");reduce(Stringkey,I //key:a//values:alistofcountsintresult=0;foreachvinresult+=ParseInt(v);計詞頻的例子里，map函數(shù)接受的鍵是文件名，值是文件的內(nèi)容，map逐個遍歷單詞，每遇到一個單詞w，就產(chǎn)生一個中間鍵值對<w,"1">，這表示單詞w咱又找到了一個；對的個數(shù)，然后將這些“1”w的出現(xiàn)次數(shù)。最后這些單詞的出現(xiàn)次數(shù)會被寫到用戶定義的位置，在底層的分布式系統(tǒng)HDFS中。MapReduceMapReduce函數(shù)。圖中執(zhí)行的順序都用workermaster，workersplit一一對應(yīng)；Map作業(yè)從輸入數(shù)據(jù)中抽取出鍵值對，每一個鍵值對都作為參數(shù)傳遞給map函數(shù)，map函數(shù)產(chǎn)生的中間鍵值對被緩存在內(nèi)存中。Reducemaster，master負責將信息轉(zhuǎn)發(fā)給Reduceworker。master通知分配了Reduceworker它負責的分區(qū)在什么位置（肯定不止一個Reduce作業(yè)（誰讓分區(qū)少呢，reduceworker遍歷排序后的中間鍵值對，對于每個唯一的鍵，都將鍵與關(guān)聯(lián)的值傳遞給reduce函數(shù)，reduce函數(shù)產(chǎn)生的輸出會添加到這個分區(qū)的輸出文件中。MapReduce作業(yè)都完成了，masteruserprogram，MapReduce函數(shù)調(diào)用返回userprogram的代碼。所有執(zhí)行完畢后，aRuce輸出放在了R個分區(qū)的輸出文件中（分別對應(yīng)一個Rdce作業(yè)。用戶通常并不需要合并這R個文件，而是將其作為輸入交給另一個eF）間數(shù)據(jù)是放在本地文件系統(tǒng)的，最終輸出數(shù)據(jù)是寫入底層分布式文件系統(tǒng)（FS）的。而apRucempdceapmapRuceece函數(shù)，eueMapReducejava來實現(xiàn)，C++來實現(xiàn)。Map的作用是過椒醬。也就是我們使用hadoop，比方來進行日志處理之后，得到我們想要的關(guān)心的數(shù)據(jù)Mapreduce 將的輸出作為輸入傳給reducer稱為shuffle。學習shuffle是如何工作的有助于我們理解mapreduce工作機制。shufflehadoop不斷被優(yōu)化和改進的代碼庫的一部分。從許多里shuffle的工作原理：mapmap函數(shù)開始產(chǎn)生輸出時，并不是簡單地將它輸出到磁盤。這個過程更復雜，利用緩沖的方式寫到內(nèi)存，并出于效率的考慮進行預排序。shuffle原理圖就看出來?？梢酝ㄟ^io.sort.mb屬性調(diào)整。一旦緩沖內(nèi)容達到閥值（io.sort.spill.percent,0.80，或沖區(qū)，但如果在此期間緩沖區(qū)被填滿，map會阻塞直到寫磁盤過程完成。在寫磁盤之前，線程首先根據(jù)數(shù)據(jù)最終要傳送到reducer把數(shù)據(jù)劃分成相應(yīng)的分區(qū)，在每個分區(qū)中，線程按鍵進行內(nèi)排序，如果有一個combiner，它會在排序后的輸出上運行。槽。默認值是40，在運行大型作業(yè)的大型集群上，此值可以根據(jù)需要調(diào)整。mapmaptasktracker的本地磁盤，現(xiàn)在，tasktracker需reduce任務(wù)就開始其輸出。這就是reduce任務(wù)的階段。reduce任務(wù)有少量map5maptasktrackertasktracker通jobtracker。這些通知都是通過心跳機制傳輸?shù)?。因此，對于指定作業(yè)，jobtracker知道得map輸出的位置，直到它獲得所有輸出位置。reducertasktracker并沒有在第一個reducer檢索到map輸出時如果map輸出相當小，則會被到reducetasktracker的內(nèi)存（緩沖區(qū)大小由(mapred.inmem.merge.threshold控制)，則合并后溢出寫到磁盤中。的合并節(jié)省一些時間。注意，為了合并，壓縮的map輸出都必須在內(nèi)存中被解壓縮。完所有map輸出被期間，reduce任務(wù)進入排序階段(sortphase更恰當?shù)恼fmap端進行的)map輸出，維持其順序排50個map1010默認值設(shè)置，成一個文件，因此最后有5個中間文件。reducereduce函數(shù)，從而省略了一次磁盤往5個文件合并成一個已排序的文件作為最后一趟。最后的合并既可來自reducereduce函數(shù)。此階hdfs和mapreduce的工作原理，已經(jīng)可以初步的使用Hadoop進行并行計算，下面我們通過一個例子來具體的講解如何使用Hadoop。 .htmlPythonHadoopMapReduce程序。為了提高運行效率各個Mapper應(yīng)該分別統(tǒng)計文章某一部分中單詞的詞頻，然后匯總到計，而Reducer階段完成單詞頻率排序。實驗的系統(tǒng)環(huán)境為Ubuntu14.04、Hadoop2.5.2、python2.7.6。有很多種途徑可以實Python與Hadoop的對接，在此主要采用Python+Streaming+MapReduce這種途徑Reduce任務(wù)的輸入數(shù)據(jù)。Reduce任務(wù)的主要目標就是把前面若干個Map的輸出匯總到一起并輸出。Mapper和會創(chuàng)建MapReduce作業(yè)，發(fā)送給各個tasktracker，同時整個作業(yè)的執(zhí)行過程。MapperMapper任務(wù)運行時，它把輸入切分成行并把每一行提供給可執(zhí)行文件進程的標準輸入。同時，Mapper收集可執(zhí)行文件進據(jù)key或valuereduce的數(shù)量來決定當前的這對輸出數(shù)據(jù)最終應(yīng)該交由哪個reducetask處理。每個Map任務(wù)都有一個用來寫入輸出數(shù)據(jù)的循環(huán)內(nèi)存緩沖區(qū)。當緩沖區(qū)中的數(shù)Partition排序，然后每個partition中再按Key排序。輸出包括一個索引文件和數(shù)據(jù)文件。Combiner，將在排序輸出的基礎(chǔ)上運行。CombinerMiniReducer，MapMapReduceMap的輸出更緊湊，更少的數(shù)據(jù)會被寫入磁盤和傳送到Reducer。最后再通過Merge將會由同一reducer處理的數(shù)據(jù)放一起，等待reducer來提取數(shù)據(jù)進行處理。ReducerMapTaskTracker，告知狀態(tài)MapTaskTrackers的映射關(guān)系。Reducer中有一個線程會間歇的向JobTracker詢問Map輸出的地址，直到把所有的數(shù)據(jù)都取到。在Reducer取走了Map輸出之后，TaskTrackersReducer可能會失敗。它們會在整個作業(yè)完成reducerreduce函數(shù)，處理排好序的輸出的每個key，最后的結(jié)果寫入HDFS。Map/ReduceReducerMapper然后在Reducer中直接接受屏幕的輸入*codingutf-8*-importforlineinsys.stdin:words=[a.strip(',.;:()?!\"\'`forainwords]forwordinifwordnotin

forkey,valueinwordcount.items():print'%s\t%s'reduce程序：#-*-coding:utf-8-*-import#對mapperforlineinforlinprintlprint'Thetotalnumberofwordsis:%d'print'Thewrongwords:'forliner:printworknodes里面，則需要用-fileframework把可執(zhí)行文件作為作業(yè)的一部分，一起打包提交。但是要處理的文件必須要放到hadoop的hdfs上。#其中上傳$bin/hadoopfs-put/home/carrie/Desktop/1/TREASURE1.txt/#執(zhí)行MapReduce PARTITHEOLDSQUIRETRELAWNEY,*DrLivesey,andtherestofthesegentlemenhavingaskedmetowritedownthewholeparticularsaboutTreasureisland,fromthebeginningtotheend,keenothingbackbutthebearingsoftheisland,andthatonlybecausethereisstilltreasurenotyetlifted,Itakeupmypenintheyearofgrace17-,andgobacktothetimewhenmyfatherkeptthe`AdmiralBenbow'inn,andthebrownoldseaman,withthesabrecut,firsttookuphislodgingunderourroof.Irememberhimasifitwereyesterday,ashecameploddingtotheinndoor,hissea-chestfollowingbehindhiminahandbarrow;atall,strong,heavy,nut-brownman;histarrypigtailfallingovertheshouldersofhissoiledbluecoat;hishandsraggedandscarred,withblack,brokennails;andthesabrecutacrossonecheek,adirty,lividwhite.*Irememberhimlookingroundthecoveandwhistlingtohimselfashedidso,andthenbreakingoutinthatoldsea-songthathesangsooftenafterwards:-`Fifteenmenonthedeadman'schest-Yo-ho-ho,andabottleofrum!'*inthehigh,oldtotteringvoicethatseemedtohavebeentunedandbrokenatthecapstanbars.Thenherappedonthedoorwithabitofsticklikeahandethathecarried,andwhenmyfatherappeared,calledroughlyforaglassofrum.This,whenitwasbroughttohim,hedrankslowly,likeaconnoisseur,lingeringonthetaste,andstilllookingabouthimatthecliffsandupatoursignboard.`Thisisahandycove,'sayshe,atlength;`andapleasantsittyatedgrog-shop.Much mate?'Myfathertoldhimno,verylittle ,themorewasthepity.'`Well,then,'saidhe,`thisistheberthforme.Hereyoumatey,'hecriedtothemanwhotrundledthebarrow;`bringupalongsideandhelpupmychest.I'llstayhereabit,'he.`I'mainman;rumandbaconandeggsiswhatIwant,andthatheaduptherefortowatchshipsoff.Whatyoumoughtcallme?Youmoughtcallmecaptain.Oh,Iseewhatyou'reat-there;'andhethrewdownthreeorfourgoldpiecesonthethreshold.`YoucanlmewhenI'veworkedthroughthat,'sayshe,lookingasfierceasacommander.And,indeed,badashisclotheswere,andcoarselyashespoke,hehadnoneoftheappearanceofamanwhosailedbeforethemast;butseemedlikeamateorskipperaccustomedtobeobeyedortostrike.Themanwhocamewiththebarrowtoldusthemailhadsethimdownthismorningbeforeatthe`RoyalGeorge;'thathehadinquiredwhatinnstherewerealongthecoast,andhearingourswellspokenof,Isuppose,anddescribedaslonely,hadchosenitfromtheothersforhisceofresidence.Andthatwasallwecouldlearnofourguest.Hewasaverysilentmanbycustom.Alldayhehungroundthecove,oruponthecliffs,withabrassescope;alleveninghesatinacorneroftheparlournextthefire,anddrankrunandwaterverystrong.Mostlyhewouldnotspeakwhenspokento;onlylookupsuddenandfierce,andblowthroughhisnoselikeafog-horn;andweandthepeoplewhocamaboutourhousesoonlearnedtolethimbe.Everyday,whenhecamebackfromhisstroll,hewouldaskifanyseafaringmenhadgonebyalongtheroad.Atfirstwethoughtitwasthewantof ofhisownkindthatmadehimaskthisquestion;butatlastwebegantoseehewasdesiroustoavoidthem.Whenaseamanputupatthe`AdmiralBenbow'(asnowandthensomedid,makingbythecoastroadforBristol)hewouldlookinathimthroughthecurtaineddoorbeforeheenteredtheparlour;andhewasalwayssuretobeassilentasamousewhenanysuchwaspresent.Forme,atleast,therewasnosecretaboutthematter;forIwas,inaway,asharerinhisalarms.Hehadtakenmeasideoneday,andpromisedmeasilverfourpennyonthefirstofeverymonthifIwouldonlykeepmy`weather-eyeopenforaseafaringmanwithoneleg,'andlethimknowthemomentheappeared.Oftenenough,whenthefirstofthemonthcameround,andIappliedtohimformywage,hewouldonlyblowthroughhisnoseatme,andstaremedown;butbeforetheweekwasouthewassuretothinkbetterofit,bringmemyfourpennypiece,andrepeathisorderstolookoutfor`theseafaringmanwithoneleg.'Howthat agehauntedmydreams,Ineedscarcely lyou.Onstormynights,whenthewindshookthefourcornersofthehouse,andthesurfroaredalongthecoveandupthecliffs,Iwouldseehiminathousandforms,andwithathousanddiabolicalexpressions.Nowthelegwouldbecutoffattheknee,nowatthehip;nowhewasamonstrouskindofacreaturewhohadneverhadbuttheoneleg,andthatinthemiddleofhisbody.Toseehimleapandrunandpursuemeoverhedgeandditchwastheworstofnightmares.AndaltogetherIpaidprettydearformymonthlyfourpennypiece,intheshapeoftheseabominablefancies. ButthoughIwassoterrifiedbytheideaoftheseafaringmanwithoneleg,Iwasfarlessafraidofthecaptainhimselfthananybodyelsewhoknewhim.Therewerenightswhenhetookadealmorerumandwaterthanhisheadwouldcarry;andthenhewouldsometimessitandsinghiswicked,old,wildsea-songs,mindingnobody;butsometimeshewouldcallforglassesround,andallthetremblingtolistentohisstoriesorbearachorustohissinging.OftenIhaveheardthehouseshakingwith`Yo-ho-ho,andabottleofrum;'alltheneighboursjoininginfordearlife,withthefearofdeathuponthem,andeachsinginglouderthantheother,toavoidremark.Forinthesefitshewasthemostoverridingcompanioneverknown;hewouldslaphishandonthetableforsilenceallround;hewouldflyupinapassionofangerataquestion,orsometimesbecausenonewasput,andsohejudgedthe wasnotfollowinghisstory.Norwouldheallowanyonetoleavetheinntillhehaddrunkhimselfsleepyandreeledofftobed. Hisstorieswerewhatfrightenedpeopleworstofall.Dreadfulstoriestheywere;abouthanging,andwalkingthenk,andstormsatsea,andtheDryTortugas,andwilddeedsandcesontheSpanishMain.ByhisownaccounthemusthavelivedhislifeamongsomeofthewickedestmenthatGodeveralloweduponthesea;andthelanguageinwhichhetoldthesestoriesshockedourincountrypeoplealmostasmuchasthecrimesthathedescribed.Myfatherwasalwayssayingtheinnwouldberuined,forpeoplewouldsoonceasecomingtheretobetyrannisedoverandputdown,andsentshiveringtotheirbeds;butIreallybelievehispresencedidusgood.Peoplewerefrightenedatthetime,butonlookingbacktheyratherlikedit;itwasafineexcitementinaquietcountrylife;andtherewasevenapartyoftheyoungermewhopretendedtoadmirehim,callinghima`truesea-dog,'anda`realoldsalt,'andsuchlikenames,andsayingtherewasthesortofmanthatmadeEnglandterribleatsea.Inoneway,indeed,hebadefairtoruinus;forhekeptonstayingweekafterweek,andatlastmonthaftermonthsothatallthemoneyhadbeenlongexhausted,andstillmyfatherneverpluckedupthehearttoinsistonhavingmoreIfeverhementionedit,thecaptainblewthroughhisnossoloudly,thatyoumightsayheroared,andstaredmypoorfatheroutoftheroom.Ihaveseenhimwringinghishandaftersucharebuff,andIamsuretheannoyanceandtheterrorhelivedinmusthavegreatlyhastenedhisearlyandunhappydeath.Allthetimehelivedwithusthecaptainmadenochangewhateverinhisdressbuttobuysomestockingsfromhawker.Oneofthecocksofhishathavingfallendown,heletithangfromthatdayforth,thoughitwasagreatannoyancewhenitblew.Iremembertheappearanceofhiscoat,whichhepatchedhimselfupstairsinhisroom,antwhich,beforetheend,wasnothingbutpatches.Heneverwroteorreceivedaletter,andheneverspokewithanybuttheneighbours,andwiththese,forthemostpart,onlywhendrunkonrum.Thegreatsea-chestnoneofushadeverseenopen.Hewasonlyoncecrossed,andthatwastowardstheend,whenmypoorfatherwasfargoneinadeclinethattookhimoff.DrLiveseycamelateoneafternoontoseethepatient,tookabitofdinnerfrommymother,andwentintotheparlourtosmokeapipeuntilhishorseshouldcomedownfromthehamlet,forwehadnostablingattheold`Benbow.'Ifollowedhimin,andIrememberobservingthecontrasttheneat,brightdoctor,withhispowderaswhiteassnow,andhisbright,blackeyesandpleasantmanners,madewiththecoltishcountryfolk,andaboveall,withthatfilthy,heavy,blearedscarecrowofapirateofours,sitting,fargoneinrum,withhisarmsonthetable.Suddenlyhe-thecaptain,thatis-begantopipeuphiseternalsong:--`Fifteenmenonthedeadman'schestYo-ho-ho,andabottleofDrinkandthedevilhaddonefortherest-Yo-ho-ho,andabottleofrum!'AtfirstIhadsupposed`thedeadman'schest'tobethatidenticalbigboxofhisupstairsinthefrontroom,andthethoughthadbeenmingledinmynightmareswiththatoftheone-leggedseafaringman.Butbythistimewehadalllongceasedtopayanyparticularnoticetothesong;itwasnew,thatnight,tonobodybutDrLivesey,andonhimIobserveditdidnotproduceanagreeableeffect,forhelookedupforamomentquiteangrilybeforehewentonwithhistalktooldTaylor,thegardener,onanewcurefortherheumatics.Intheme ,thecaptaingraduallybrightenedupathisownmusic,andatlastflappedhishanduponthetablebeforehiminawayweallknewtomean-silence.Thevoicesstoppedatonce,allbutDrLivesey's;hewentonasbefore,speakingclearandkind,anddrawingbrisklyathispipebetweeneverywordortwo.Thecaptainglaredathimforawhile,flappedhishandagain,glaredstillharder,andatlastbrokeoutwithavillainous,lowoath:`Silence,there,betweendecks!'`Wereyouaddressingme,sir?'saysthedoctor;andwhentheruffianhadtoldhim,withanotheroath,thatthiswasso`Ihaveonlyonethingtosaytoyou,sir,'repliesthedoctorthatifyoukeepondrinkingrum,theworldwillsoonbequitofaverydirtyscoundrel!'Theoldfellow'sfurywasawful.Hesprangtohisfeet,drewandopenedasailor'sclasp-knife,and,balancingitopenonthepalmofhishand,threatenedtopinthedoctortothewall.Thedoctorneversomuchasmoved.Hespoketohim,asbefore,overhisshoulder,andinthesametoneofvoice;ratherhigh,sothatalltheroommighthear,butperfectlycalmantsteady:--`IfyoudonotputthatknifethisinstantinyourpocketIpromise,uponmyhonour,youshallhangatthenextassizes.'Thenfollowedabattleoflooksbetweenthem;butthiscaptainsoonknuckledunder,putuphisweapon,andresumedhisseat,grumblinglikeabeatendog.`Andnow,sir,'thedoctor,`sinceInowknowthere'ssuchafellowinmydistrict,youmaycountI'llhaveaneyeuponyoudayandnight.I'mnotadoctoronly;I'mamagistrate;andifI