PASA 為什么需要 areliabledistributedmemory-centricstorageenablingmemory-speeddatasharing.Deployedat>50companies(Dec. ,Yahoo,Pivotal,Redhat,BaiduandsoSparkandMapReduceapplicationscanrunwithoutanycodeTachyonTachyon-46-30TachyonTachyon-15Tachyon

Tachyon-3-1Dec

Apr

Feb

July7TachyonTalksgivenbyHaoyuanin 如何使用//Loadinputfromfile.count();file.take(10);//StoreRDDOFF_HEAPinfile.count();file.take(10);

ThisfeatureinSparkiscontributedbyme!//Saveoutputtofile.flatMap(line=>line.split("")).map(s=>(s,1)).reduceByKey((a,b)=>a+如何使用Memory-localitykeyto ctiveresponse ??? 讀類型|數(shù)據(jù)位于|--- → → WorkerMaster&Worker Tachyon-PerfAGeneralPerformanceTestFrameworkfor concrete

Fileseveralfileutility+getTaskConfigurationPerfFileSystemLocal+getTaskConfigurationPerfFileSystemLocalTaskType+get+get+get+get+...(fsPerfFileSystemHdfs-mH+...(fs -m+...(fs--m--mTask-mThreads XXXTask XXXTaskBen

read setFromToolsDfsPerfCollectorDfsPerfCleanerXXXThreadToolsDfsPerfCollectorDfsPerfCleanerXXXThread-XXXTaskContext-+read+setFrom

setupThread()meme-mStart-mFinish-m

initialFromTaskCPerfFileSystem++PerfFileSystem+++++abstract+.(fsPerfFileSystemTfs-m+...(fsXXXTotalReportXXXTotalReport-+initial+static+static+static+staticmetadata()+...(otherOperators SequentiallyRead(Write)theSkipReadfilesinrepeatedpattern“skip-Readfilesinrepeatedpattern“randomlyskip-Repeatperformingtouch,mkdir,exists,rename,deletemetadata ng“writeabunch->readShuffle,readfromthewholeworkspace(mayberemoteNo-Shuffle,readfileswrittenbyitself(dataMassivelyandconcurrentlyAlsoShuffleorNo-Eachben arkcontainsasetofconcreteclasseswhichareextendedfromthebasicToaddben ark,extendthe withnewimplementationandmodifythe基于Tachyon-Perf1masternode+16slave2xIn XeonX566032GBmemory

2xIn XeonX562024GBmemoryExt3LocalFileMetadataMetadataPerformance0Tachyon(multi- Tachyon(multi-多線程/進(jìn)程測試模式下Tachyon元數(shù)據(jù)操作的性能（單客戶端節(jié)點(diǎn)，并發(fā)度Read/WriteRead/WriteThroughput

RandomlyReadRandomlyReadRead/WriteThroughput

(a)多線程模式(b)多進(jìn)程模式統(tǒng)計(jì)的是整個(gè)集群的讀寫速率（單客戶端節(jié)點(diǎn)，并發(fā)度AverageAverageReadThroughput

（單客戶端節(jié)點(diǎn)，并發(fā)度

0AverageAverageWriteThroughput

AverageAverageReadThroughput0AverageAverageWriteThroughput

(d)Multi-

8642050

8642050

(d)Multi-ComplexTransactionThroughputinShuffleModeComplexComplexTransactionThroughputinShuffleMode

50ComplexTransactionComplexTransactionThroughputinNon-ShuffleMode0

50ComplexTransactionComplexTransactionThroughputinNon-ShuffleMode0

(d)Multi-

Tachyon(In-Memory)ReadTachyon(In-Memory) Tachyon(In-Memory)ReadTachyon(In-Memory)WriteTachyon(Through)ReadTachyon(Through)Write8 Tachyon(In-Memory)ReadTachyon(In-Memory)WriteTachyon(Through)ReadTachyon(Through)Write

420024681012141618202224262830#of

420024

81012141618202224262830#ofprocesses（單客戶端節(jié)點(diǎn)，并發(fā)度

Tachyon(In-Memory) Tachyon(In-Memory)ReadTachyon(In-Memory)WriteTachyon(Through)ReadTachyon(Through)Write

Tachyon(In-Memory) Tachyon(In-Memory)ReadTachyon(In-Memory)WriteTachyon(Through)