利用python程序通過thrift讀取hbase中數(shù)據(jù)一環(huán)境介紹：master機(jī)器：slave機(jī)器兩臺(tái)：mysql數(shù)據(jù)庫：二軟版本介紹：hadoop版本：hadoop-0.20.2sqoop版本：sqoop-1.2.0-CDH3B4java版本：1.6,0_05mysql版本：5.1.25-rc三將mysql中數(shù)據(jù)導(dǎo)入到Hbase中首先查看myql數(shù)據(jù)庫中students表內(nèi)容:mysql>deletefromstudents;QueryOK,5rowsaffected(0.00m(ysql>++m(ysql>++Ind|++1112三i45I++5rowsseiect*fromstudents;nauiE+abcdefaaaaefsazkiassI+1-Q-86O3e.一22266g-a-將mysql的數(shù)據(jù)導(dǎo)入hbase的命令格式為：sqoopimport--connectjdbc:mysql://mysqlserver_IP/databaseName--username--passwordpassword--tabledatatable--hbase-create-table--hbase-tablehbase_tablename--column-familycol_fam_name--hbase-row-keykey_col_name說明：databaseName和datatable是mysql的數(shù)據(jù)庫和表名，hbase_tablename是要導(dǎo)成hbase的表名，key_col_name可以指定datatable中哪一列作為hbase新表的rowkey，col_fam_name是除rowkey之外的所有列的列族名例如：可通過如下命令將Mysql中的students表導(dǎo)入到Hbase中：sqoopimport--connectjdbc:mysql://5/sqoop--usernamesqoop--passwordsqoop--tablestudents--hbase-create-table--hbase-tablestudents--column-familystuinfo--hbase-row-keyid查看hbase中數(shù)據(jù)：[hadoop@node26conf]$hbaseshellHBaseShell;enter'help<RETURN>'forlistofsupportedcommands.Type"exit<RETURN>"toleavetheHBaseShellVersion0.90.5,r1212209,FriDec905:40:36UTC2011hbase(main):001:0>scan'students'ROWCOLUMN+CELL1column=stuinfo:age,timestamp=1358927364631,value=291column=stuinfo:name,timestamp=1358927364631,value=abc2column=stuinfo:age,timestamp=1358927364566,value=282column=stuinfo:name,timestamp=1358927364566,value=def3column=stuinfo:age,timestamp=1358927368741,value=26column=stuinfo:name,timestamp=1358927368741,value=aaaa4column=stuinfo:age,timestamp=1358927364563,value=604column=stuinfo:name,timestamp=1358927364563,value=efsaz5column=stuinfo:age,timestamp=1358927364563,value=635column=stuinfo:name,timestamp=1358927364563,value=kiass5row(s)in1.4110seconds四thrift結(jié)合python實(shí)驗(yàn)thrift安裝鏈接：/download/python版本：2.7.3步驟為：1)安裝python2.7.3說明：python2.7.3與thrift結(jié)合沒問題，python2.5版本好像不行！生成的Hbase.py文件中的語法rhel5自帶的python2.4不支持tarfvxjPython-2.7.3.tar.bz2./configure&&make&&makeinstallpython2.7.3路徑為：/usr/local/bin/python執(zhí)行python命令，看到的版本是否是2.7.3[hadoop@node26~]$/usr/local/bin/pythonPython2.7.3(default,Jan242013,15:24:19)[GCC4.1.220080704(RedHat4.1.2-46)]onlinux2Type"help","copyright","credits"or"license"formoreinformation.>>>2)安裝thrifttarfvxzthrift-0.9.0.tar.gzcdthrift-0.9.0./configure&&make&&makeinstall在configure過程中缺什么開發(fā)包就安裝什么開發(fā)包thrift0.9.0TOC\o"1-5"\h\zBuildingC++Library:noBuildingC(GLib)Library:yesBuildingJavaLibrary:noBuildingC#Library:noBuildingPythonLibrary:yesBuildingRubyLibrary:noBuildingHaskellLibrary:noBuildingPerlLibrary:noBuildingPHPLibrary:noBuildingErlangLibrary:no

BuildingGoLibrary:noBuildingDLibrary:noPythonLibrary:UsingPython:/usr/local/bin/python因?yàn)槭墙opython用，所以這些no就忽略吧makemakeinstall3）通過thrift生成Hbase.py[root@hadoop1thrift-0.9.0]#thrift-versionThriftversion0.9.0截圖：acIocaI.m4ZHANGES8mp

confiler/contig.mnconf-to.1ogconfstatusconfig.subThriftversion0.9.0|contigure.accontrib/CONTRIBUTORSdepcompl-vetstondoc/instal1-shlib/1ibtoolIt吼ain.snMakefi1eMakef1le.amMakefile,inNEWSNOTICEREADMEStHRip-hltutor-ylwrap[root@node26thrift-0.9.0]#[root@hadoop1thrift-0.9.0]#thrift--genpy/home/kyo/hbase/src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift[root@hadoop1thrift-0.9.0]#treegen-py/gen-py/|--__init__.py'--hbase|--Hbase-remote|--Hbase.py|--__init__.py|--constants.py'--ttypes.py1directory,6files截圖：copyHbase相關(guān)文件到python2.7.3包目錄下面:

cp-rgen-py/hbase//usr/local/lib/python2.7/site-packages/看thrift的python模塊是否存在，如不存在就做個(gè)軟連接：ls/usr/local/lib/python2.7/site-packages/hbaseREADMEthriftthrift-0.9.0-py2.7.egg-infoln-s/usr/lib/python2.7/site-packages/thrift*/usr/local/lib/python2.7/site-packages/在node26節(jié)點(diǎn)上啟動(dòng)thrift。前提是hadoop和hbase已經(jīng)正常啟動(dòng)了!命令為：hbasethrift-p9090start截圖：[root@node26thrift-O.9.0]#su-hadoopEhadoop@node26$hbasethrift-p9090start13/01/2413/01/2413/01/2413/01/2413/01/2413/01/2416:31:5616:31:5616:31:5616:31:5616:31:5616:31:56INFOINFOINFOINFOINFOINFOzookeeper.ZooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:clientclientcl1entclientclientclientenvironment:zookeeper.version=3.3.2-1031432,environment:=node26environment:iava.version=l.6.0_05environment:lava.vendor=sunMicrosystemsIncava.hon)e=/usr/java/jdkl.6.0_05/;environment:ervironment:java.class.path=/usr/local/hbase:ib/tools.jar:/usr/1ocal/hbase-0.90.5:/usr/1ocal/hbase-0.90.5/hbase-0.90.5.jar:/usr/local/hbase-i□cal/hbase-0.90.5/1ib/activafion-1.1.^ar:/usr/1ocal/hbase-0.90.5/11b/asm-3.1.jar:/usr/1ocal/hba：cal/hbase-0.90.5/11b/coRimons-cl1-1.2.jar:/usr/local/hbase-0.90.5/11b/commons-codec-1.4.jarz/tisr,.0.jar:/usr/loca?/hbase-O.90.5/11b/commons-httpclient-B.1.jar:/usr/1ocal/hbase-O.90.5/11b/commoi1.1.1.jar:/usr/1ocal/hbase-O.90.5/11b/commons-net-1.4.1.jar:/usr/local/I0.5/Iib/commons-1ogging-：

/Iocal/hbase-0.90.5/71B/uava-r06.jar:/usr/1ocal/hbase-O.90.5/lib/hadoop-0.20.2-core.jar:/usr/lisi-1.5.5.jar:/usr/1ocal/hbase-O.90.5/1ib/jackson-jaxrs-1.5.5.jar:/usr/local/hbase-O.90.5/11b/ja.7/hbase-0.90.5/1ib/jackson-xc-1.5.5.jar:/usr/local/hbase-O.90.5/11b/jasper-compiler-5.5.23.^jar\□nf]me-5.5.23.jar:/usr/1ocal/hbase-O.90.5/11b/jaxb-api-2.1.jar:/usr/1ocal/hbase-O.90.5/lib/jaxb90.5/11b/jersey-core-1.4.jarr/usr/1ocal/hbase-O.90.5/11b/jersey-json-1.4.jarr/usr/1ocal/hbase-0/Iocal/hbase-O.90.5/1ib/jetfison-1.1.jar:/usr/1ocal/hbase-O.90.5/Hb/jerty-6.1.26.jar:/usr/1oca.jar:/usr/local/hbase-O.90.5/}ib/jruby-complete-1.6.0.jar:/usr/loca}/nbase-Q.9Q.5/1_ib/jsp-2.l-6b/jsp-api-2.1-6.1.14.jar:/usr/Iocal./hbase-O.90.5/7ib/jsr311-api-1.1.1.jar:/usr/1ocal/hbase-O.90base-0.90.5/1ib/protoouf-java-2.3.0.jar:/usr/local/hbase-O.90.5/7ib/servlet-api-2.5-6.1.14.jar\i-1.5.8.jar:/usr/1ocal/hbase-O.90.5/11b/slf4j-loq4jl2-l.5.8.jar:/usr/1ocal/hbase-O.90.5/1ib/sta:0.5/11b/thrift-O.2.0.jar:/usr/1ocal./hbase-O.90.S/lib/xmlenc-0.52.jar:/usr/local/hbase-O.90.5/111se-0.90.5/conf/13/01/2416:31:56infozookceper.zooKeeper:clientenvironment:java.1ibrary.path=/usr/java/jdkla/jdkl.6.0_05/jre/lib/i386:/usr/java/jdkl.6.0_05/jre/../Iib/i386:/usr/java/packages/11d/i386:/I"gtI』“一~>---clientenvironmentava.i□.tmpdir=/tmpenvironment:pi1er=<NA>environment:=Linux13/01/2416:31:5613/01/2413/01/2413/01/2413/01/2413/01/2413/01/2413/01/2413/01/2416:31:5616:31:5616:31:5616:31:5616:31:5616:31:5616:31:5616:31:56INFOINFOINFOINFOINFOINFOINFOINFOINFOzookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.ZooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:zookeeper.zooKeeper:cl1entclientclientclientclientcl1errtcl1entenvironment:os.arch=i386environment:os.version^2.6.18-164.el5environment:=hadoopenvironmerrt:user.home=/home/inacloop81,:2181sessionTimeout=180000watcher=hconnecfianenvironment:user.dir=/home/hadoopinitiatingclientconnect!on,connectstr1ng=node28.?13/01/2416:31:56INFO13/01/2416:31:56INFOzookeeper.clientcnxnropeningsocketconnecfiontoservernode28.puppet.?zookeeper.Clientcnxn:Socketconnect!onestablishedtonode28.puppet,com,onon13/01/2416:31:56INFOMrhi/rr—UH比1a-?仁」rrncnczookeeper.clientcnxn:sessionestabliishmentcompleteonserverrtode28.pti|HTt’rj-ix-J，HWXrnjrr.nrwHKT口fhfhpython文件如下:#!/usr/local/bin/python#coding=utf-8importsys#Hbase.thrift生成的py文件放在這里sys.path.append('/usr/local/lib/python2.7/site-packages/hbase')fromthriftimportThriftfromthrift.transportimportTSocketfromthrift.transportimportTTtocolimportTBinaryProtocolfromhbaseimportHbase#如ColumnDescriptor等在hbase.ttypes中定義fromhbase.ttypesimport*#Makesocket#此處可以修改地址和端口transport=TSock