任務(wù)一Hadoop系統(tǒng)日志結(jié)構(gòu)及分析Hadoop系統(tǒng)日志?查看Hadoop系統(tǒng)日志。默認(rèn)存儲位置:Hadoop系統(tǒng)目錄下的logs子目錄下。?.log文件:存儲運(yùn)行進(jìn)程中的所有消息,到達(dá)一定長度后會另生成一個,的文件,數(shù)字越小內(nèi)容越新。.out文件：存儲進(jìn)程的啟動消息,存儲機(jī)制與.log文件相同。:~$cd/usr/local/hadoop/logs::~$cd/usr/local/hadoop/logs:/usr/local/hadoop八ogs$Ishadoohadochadoop-hadoop-namenode-Master.loghadoop-hadoop-namenode-Master.outhadoop-hadoop-namenode-Master.out.1hadoop-hadoop-namenode-Master.out.2hadoop-hadoop-namenode-Master.out.3hadoop-hadoop-namenode-Master.out.4hadoop-hadoop-secondarynamenode-Master?loghadoop-hadoop-secondarynamenode-Master.outhadoop-hadoop-secondarynamenode-Master.out.1hadoop-hadoop-secondarynamenode-Master.out.2hadoop-hadoop-secondarynamenode-Master.out.3hadoop-hadoop-secondarynamenode-Master?out.4hadoop^Haster:/usr/local./hadoop/logs$Hadoop文件內(nèi)容。?關(guān)注包含"ERROR"關(guān)鍵字的行。:/usr/local/hadoop/logs$20XX-08-O717:59:43,823.node.FSEdttLog:Startinglogsegmentat9820XX-08-0717:59:43,972INFOnode.TransferFsImage:Transfertook0.02sat58.82KB/node.TransferFsImage:Downloadedfilefsimage.ckptO000000000000000097size1832bytes...:Coingtoretain2imageswithtxtd>=95.:PurgingoldtmageFSImageFtle(ftle=/usr/local/hadoop/tnp/dfs/name/current/fstmage0OOO0OOO00000000093>.hadoop.:.hadoop..:SHUTDOWNMSG:SHUTDOWNMSC:ShuttingdownNameNodeatMaster/0************************************************************yhadoop@Maste「:/us「/local/hadoop/logs$■更改HadoopEJ志默認(rèn)存儲路徑。?將Hadoop的日志存儲路徑更改到/var/logo?在/var/log下創(chuàng)立hadoop子目錄,并修改權(quán)限。:$cd/var/loghp syslogwtmp:/v三 5iscupsaptdtst-upgradeinstallervboxadd?setup?logjournalfalllogvboxadd?gdm3lastlogtallylogvboxadd-btmpgpu-manager.:y:廠/I$1sudomkdirhadoop<i-cupsLhadoopspeech-dispatcherunattended-upgradesaptdist-upgrade下syslogwtmpinstallerfaillogjournal\vboxadd-setup?kern.log\syslog?vboxadd-?2gdm3kern.log.vboxadd-btmpgpu-manager.'loglastlog、allylogvboxadd-:;r/1 |:/var/log$sudochown-Rhadoop/var/log/hadoop[sudo]hadoop的密碼:d :/var/log$

修改Hadoop的配置文件:~$cd/usr/local/hadoop/etc/hadoopster:/us「八ocal/hadoop/etc/hadoop$Iscapacity-scheduler?xmlconfiguration.xslcontainer-executor.cfgcore-site.xmlhadooD-envdhadoop-env.shr:/usr/local/hadoop/etc/hadoop$kms-Ipertieskms-site.xmlpertiesmapred-envdmapred-env.shmapped-queues?xml?templatemapred-stte.xnlslavesssl-cltent.xml.examplessl?exportHADOOP-SECURE-DN.USER=${HADOOPSECURE-DNUSER}#exDortHADOOPLOGDIRMIHADOOPLOGDI#Wherelogfilesarestored.$HADOOPHOME/1ogsbydefault.一}/SUSER#exDortHADOOPLOGDIRMIHADOOPLOGDIexportHADOOPLOGDIR=/var/log/hadoop#Wherelogfilesarestoredinthesecuredataenvironment,exportHADOOP-SECUREDN-LOGDIR=${HADOOPLOGDIR}/S{HADOOPHDFSUSER}###HDFSMoverspecificparameters###SpecifytheJVMoptionstobeusedwhenstartingtheHDFSMover.TheseoptionswillbeappendedtotheoptionsspecifiedasHADOOPOPTS--插入--任務(wù)二NameNode弟節(jié)點(diǎn)展障的風(fēng)險頊防噩渣。任務(wù)二硬件志和HadooP日志,hadoop@bigdata2:~$df-h文件系統(tǒng)容量已用可用udev0tmpfs395M393M/dev/sdal63G52Gtmpfs26Mtmpfstmpfs0/dev/loop00/dev/loopl141M141M0/dev/loop243M43M0/dev/loop3141M141M0/dev/loop40/dev/loop535M35M0/dev/loop689M89M0/dev/loop7150M150M0/dev/loop80/dev/loop955M55M0/dev/loopl015M15M0/dev/loopll91M91M0/dev/loopl215M15M0/dev/loopl30/dev/loopl413M13M0/dev/loopl50tmpfs395M28K395Mtmpfs395M36K395M/dev/sr074M74M0hadoopQbigdata2:~$

已用％掛裁點(diǎn)0%/dev1%/run15%/2%/dev/shm1%/run/lock0%/sys/fs/cgroup100%/snap/gnome-system-monitor/57100%/snap/gnome-3-26-1604/90100%/snap/gtk-common-themes/1313100%/snap/gnome-3-26-1604/74100%/snap/gnome-system-monitor/100100%/snap/gtk-common-themes/818100%/snap/core/7270100%/snap/gnome-3-28-1804/67100%/snap/gnome-logs/61100%/snap/corel8/1074100%/snap/gnome-characters/296100%/snap/core/6350100%/snap/gnome-logs/45100%/snap/gnome-calculater/260100%/snap/gnome-characters/139100%/snap/gnome-calculator/4061%/run/user/1211%/run/user/1001100%磁盤故障分析命令。hadoop@bigdata2:~$sudofdisk?l/dev/loopl5Disk/dev/loopl5:140.7MiB,147496960字節(jié),288686個扇區(qū)單元:扇區(qū)/1*512=512字節(jié)扇區(qū)大小(邏輯/物理):512字節(jié)/512字節(jié)I/O大?。ㄗ钚?最正確）:512字節(jié)/512字節(jié)hadoop@bigdata2:~$磁盤故障分析命令。?執(zhí)行"parted新目錄"。任務(wù)四Hadoop的隱私及平安認(rèn)證問題?Hadoop并沒有向用戶提供平安認(rèn)證的效勞,由此常常會引發(fā)一系列平安問題。?用戶可直接訪問HDFS或MapReduce集群,因此,不僅任何用戶都可獲取平臺上的數(shù)據(jù),還容易引發(fā)用戶強(qiáng)制對整個集群環(huán)境執(zhí)行命令,造成整個集群受影響。?惡意攻擊程序可能會偽裝成為Hadoop平臺上的某項(xiàng)效勞,對效勞器甚至是用戶數(shù)據(jù)造成惡意攻擊。授權(quán)問題。?Hadoop的數(shù)據(jù)節(jié)點(diǎn)Datanode未提供有效授權(quán)機(jī)制,來限制訪問數(shù)據(jù)節(jié)點(diǎn)的用戶和效勞,所以未授權(quán)用戶也可通過數(shù)據(jù)ID獲取數(shù)據(jù),這大大增加了Datanode上存放數(shù)據(jù)的風(fēng)險。Hadoop平安驗(yàn)證問題解決方案 Kerberoso?針對上述種種問題,Hadoop提供了Kerberos平安機(jī)制。Kerberos時一種分布式身份驗(yàn)證效勞,其搭建在Hadoop集群之上,當(dāng)客戶效勞器訪問Hadoop節(jié)點(diǎn)數(shù)據(jù)時,會直接訪問Kerberos,而不是直接訪問Hadoop節(jié)點(diǎn),由此保證了Hadoop節(jié)點(diǎn)的數(shù)據(jù)不會因被任意訪問而收到攻擊。任務(wù)五Hadoop的未來i.挑戰(zhàn)大數(shù)據(jù)時代,各種大數(shù)據(jù)平臺面臨的首要調(diào)整便是數(shù)據(jù)量的不斷增長,能否應(yīng)對不斷擴(kuò)大的數(shù)據(jù)體量,也是Hadoop系統(tǒng)未來的挑戰(zhàn)之一。此外,隨著各種大數(shù)據(jù)公司的不斷升級,能否適應(yīng)難度逐漸加深的數(shù)據(jù)處理應(yīng)用場景,并提供相應(yīng)解決方案,是Hadoop系統(tǒng)面臨的又一挑戰(zhàn)。通過前面的學(xué)習(xí)了解到,Hadoop自身也有各種限制,例如實(shí)時數(shù)據(jù)交互、數(shù)據(jù)處理延遲、平臺本身沒有提供有效平安機(jī)制保