阿里巴巴離線大數(shù)據(jù)處理平臺(tái)3社交網(wǎng)絡(luò)、用戶行為、購(gòu)買關(guān)系企業(yè)內(nèi)外IT系統(tǒng)的整合數(shù)據(jù)規(guī)模爆炸式增長(zhǎng)結(jié)構(gòu)化與非結(jié)構(gòu)化數(shù)據(jù)并存、噪聲增多組合離線、在線、流式等多項(xiàng)計(jì)算能力提供解決方案在大數(shù)據(jù)上應(yīng)用傳統(tǒng)統(tǒng)計(jì)學(xué)和數(shù)據(jù)挖掘算法用戶隱私保護(hù)、商業(yè)數(shù)據(jù)泄漏、復(fù)雜的訪問控制需求企業(yè)發(fā)展的達(dá)摩克利斯之劍4阿里巴巴的大數(shù)據(jù)產(chǎn)品探索67阿里巴巴離線大數(shù)據(jù)處理平臺(tái)ODPS

(Open

Data

Processing

Service)支持海量結(jié)構(gòu)化數(shù)據(jù)的離線存儲(chǔ)和計(jì)算以RESTful

API的方式提供服務(wù)支持MapReduce、BSP編程框架支持常用的矩陣運(yùn)算和數(shù)據(jù)挖掘算法支持多用戶管理和基于ACL和policy的權(quán)限控制基于ODPS可以打造完整的數(shù)據(jù)倉庫解決方案多集群架構(gòu)，運(yùn)維和管理系統(tǒng)如何了解業(yè)務(wù)運(yùn)行狀態(tài)？元數(shù)據(jù)管理，指標(biāo)系統(tǒng)服務(wù)化，統(tǒng)一計(jì)算入口，可度量和控制的計(jì)算資源10避免IO消耗節(jié)約調(diào)度成本Failover資源占用數(shù)據(jù)規(guī)模11統(tǒng)一的API、授權(quán)、數(shù)據(jù)存儲(chǔ)、metaMaster-slave，worker負(fù)責(zé)子圖通過迭代在節(jié)點(diǎn)間通訊和修改圖拓?fù)鋽?shù)據(jù)傾斜時(shí)的內(nèi)存控制與其他類型作業(yè)共存錯(cuò)誤恢復(fù)推薦、社交網(wǎng)絡(luò)、物流、…利用圖編程實(shí)現(xiàn)SSSP算法(1)GraphLoader<LongWritable,

LongWritable,

LongWritable,

LongWritable>

{public

void

load(Record

record,

MutationContext<...>

context)

{SSSPVertex

vertex

=

new

SSSPVertex();vertex.setId((LongWritable)

record.get(0));String[]

edges

=

record.get(1).toString().split(",");for

(int

i

=

0;

i

<

edges.length;

i++)

{String[]

ss

=

edges[i].split(":");vertex.addEdge(new

LongWritable(Long.parseLong(ss[0])),new

LongWritable(Long.parseLong(ss[1])));}context.addVertexRequest(vertex);}利用圖編程實(shí)現(xiàn)SSSP算法(2)Vertex<LongWritable,

LongWritable,

LongWritable,

LongWritable>

{@Overridepublic

void

compute(ComputeContext<…>

context,

Iterable

messages)

{long

minDist

=

(getId()

==

1)

?

0

:

Integer.MAX_VALUE;for

(LongWritable

msg

:

messages)

{if

(msg

<

minDist)

{if

(minDist

<

this.getValue())

{this.setValue(minDist);for

(Edge

e

:

this.getEdges())

{context.sendMessage(e.getDestVertexId(),

minDist

+

getValue());}}利用圖編程實(shí)現(xiàn)SSSP算法(3)支持算法：SVD分解邏輯回歸隨機(jī)森林…連接R與ODPS集成ODPS的算法業(yè)務(wù)增長(zhǎng)的速度永遠(yuǎn)快于技術(shù)完善的速度，如何支撐日益膨脹的存儲(chǔ)和計(jì)算需求？17解決規(guī)模瓶頸統(tǒng)一meta準(zhǔn)實(shí)時(shí)數(shù)據(jù)同步減少熱數(shù)據(jù)跨機(jī)房業(yè)務(wù)遷移多控制機(jī)群灰度發(fā)布業(yè)務(wù)分離1819用戶空間授權(quán)訪問20數(shù)據(jù)安全是每個(gè)平臺(tái)產(chǎn)品需要解決的核心問題ODPS安全架構(gòu)INSERTOVERWRITEprjB.t1

ASSELECTa.shop_name,

b.sale_totalFROMprjA.shopaLEFTOUTERJOINprjB.sale_detailbONa.shop_name=b.shop_name;…OTS(MetaStore)2.檢查用戶權(quán)限,檢查數(shù)據(jù)保護(hù)策略.“pangu://odps/prjA/shop/”:

R,“pangu://odps/prjB/sale_detail/”:

R,“pangu://odps/prjB/t1/”:

W}用戶提交的MapReduce程序可能會(huì)：惡意耗盡集群資源(CPU,Memory,Network,Disk)；直接訪問Pangu文件，竊取或篡改其他用戶數(shù)據(jù)；竊取Linux節(jié)點(diǎn)上的飛天系統(tǒng)進(jìn)程的敏感數(shù)據(jù)（如Tubo的capability）；...…基于沙箱的縱深防御體系OTS(MetaStore)ProcessSandboxJavaSandbox["Resource":

"acs:odps:*:projects/sampleprj/tables/test_*","Action":

["odps:Update",

"odps:Select"]"Condition"

:

{]25如何解決系統(tǒng)與多個(gè)數(shù)據(jù)源之間的數(shù)據(jù)交換問題？