1、江西省農(nóng)信社GPFS安裝手冊V1.0目錄1.江西省農(nóng)信社GPFS規(guī)劃21.1江西省農(nóng)信社GPFS集群規(guī)劃21.2集群Quorum方式31.3NSD磁盤組成31.4文件系統(tǒng)基本信息31.5GPFS系統(tǒng)環(huán)境示意圖52.GPFS安裝前準備52.1安裝介質(zhì)及補丁準備52.2Openssl及Openssh62.3Linux節(jié)點安裝ksh62.4Linux節(jié)點修改網(wǎng)絡參數(shù)63.GPFS安裝配置73.1配置ssh信任關(guān)系73.2AIX安裝GPFS軟件103.3Linux安裝GPFS軟件113.4GPFS集群配置124.GPFS日常維護194.1GPFS啟動194.2GPFS停止194.3查看GPFS使用狀況

2、201. 江西省農(nóng)信社GPFS規(guī)劃1.1 江西省農(nóng)信社GPFS集群規(guī)劃IP地址節(jié)點名操作系統(tǒng)集群角色8AIX 7100-02-02-1316QAIX 7100-02-02-1316QRedhat 6.4 for PRedhat 6.4 for Powerclient10.0.147

3、.51Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient6gpfs56.abs.j

4、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat

5、6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powercl

6、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient2

7、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient7gpfs77.abs.jxnx

8、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4

9、 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclien

10、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient3gpf

11、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient1.2 集群Quorum方式集群Quorum方式采用Node Quorum + Tiebreaker disks,兩個AIX節(jié)點加上一塊專用的SAN磁盤,目前只有一個AIX

12、節(jié)點,第二個AIX節(jié)點和SAN磁盤需要后續(xù)添加。1.3 NSD磁盤組成PV_NAMESIZENSD_NAMEFS_NAMENSD SERVERS LISThdisk2500Gbjxrccb_hdisk3500Gbjxrccb_1.4 文件系統(tǒng)基本信息1.5 GPFS系統(tǒng)環(huán)境示意圖GPFS Server: 2個AIX分區(qū)連接V7000SAN Storage: SVCGPFS Client: 48個Power Linux分區(qū)GPFS管理網(wǎng)絡: 千兆交換機2. GPFS安裝前準備2.1 安

13、裝介質(zhì)及補丁準備安裝包:GPFS v補?。篏PFS-12.2 Openssl及Openssh確認AIX上openssl及openssh是否已安裝,Linux默認已經(jīng)安裝;lslpp -l |grep open2.3 Linux節(jié)點安裝ksh#rpm -ivh ksh-20100621-19.el6.ppc64.rpm2.4 Linux節(jié)點修改網(wǎng)絡參數(shù)修改/etc/sysctl.conf文件,需修改的網(wǎng)絡參數(shù)如下:# increase Linux TCP buffer limitsnet.core.rmem_max = 8388608net.core.wmem_max

14、 = 8388608# increase default and maximum Linux TCP buffer sizesnet.ipv4.tcp_rmem = 4096 262144 8388608net.ipv4.tcp_wmem = 4096 262144 8388608# increase max backlog to avoid dropped dev_max_backlog = 2500修改完畢后執(zhí)行sysctl -p /etc/sysctl.conf讓配置生效3. GPFS安裝配置3.1 配置ssh信任關(guān)系n 在節(jié)點gpfs98.abs.

15、上生成并配置ssh密鑰對#ssh-keygen -t rsa# cd .ssh# lsid_rsa id_rsa.pub# cat id_rsa.pub authorized_keys# chmod 600 authorized_keys# echo StrictHostKeyChecking no config# chmod 600 confign 修改各節(jié)點的/etc/hosts文件,加入對集群中各節(jié)點的別名和ip地址9 0 1 gp

16、2 3 4 5 6 7 8 9 0 gpfs60.abs.jxn

17、1 2 3 4 5 6 7 8 9 10.0.1

18、47.70 1 2 3 4 5 6 7 8 9 gpfs79

19、.0 1 2 3 4 5 6 7 8 gpfs88.abs.jxnxs.c

20、om9 0 1 2 3 4 5 6 7

21、8 n 驗證本機無需密碼的ssh登錄#ssh hostnamen 將.ssh目錄復制到其他各節(jié)點#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/驗證各節(jié)點間無密碼的ssh登錄3.2 AIX安裝GPFS軟件1） gpfs3.5.0安裝包放在/home/gpfsaix/iso目錄下2

22、）切換/home/gpfsaix/iso目錄下,通過smitty installp安裝GPFS軟件3) 安裝補丁gpfs3.5版本，需要升級到1以上，補丁下載地址：/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=AIX&function=all下載后上傳到服務器后解壓：gzip -d -c filename.tar.gz

23、|tar -xvf -使用smitty update_all安裝補丁4）安裝后檢查#lslpp -l|grep gpfs如果之前已經(jīng)創(chuàng)建了cluster，在所有node都down的狀態(tài)下，加載GPFS版本到最新版本：mmchconfig release=LATEST;如果是新裝的GPFS,則可以忽略此步。3.3 Linux安裝GPFS軟件1) 上傳安裝介質(zhì)安裝文件位于/home/gpfsrpm/，補丁文件位于/home/gpfsrpm/updte；2) 安裝GPFS軟件在安裝介質(zhì)所在目錄下執(zhí)行：#rpm -ivh gpfs*.rpm3) 升級GPFS軟件版本在補丁文件所在目錄下執(zhí)行：#rpm

24、-Uvh gpfs*.rpm4) 檢查安裝結(jié)果#rpm -qa |grep gpfs5) Building the GPFS portability layer#cd /usr/lpp/mmfs/src#make Autoconfig#make World#make InstallImages3.4 GPFS集群配置n 修改系統(tǒng)環(huán)境變量AIX平臺下修改/etc/environment在PATH后增加/usr/lpp/mmfs/bin#vi /etc/environmentLinux平臺下修改/root/.bashprofile,在PATH后增加/usr/lpp/mmfs/binn 創(chuàng)建集群按照

25、規(guī)劃，本次集群中包含1個AIX節(jié)點和48個Linux client；首先在1個AIX節(jié)點平臺創(chuàng)建GPFS集群創(chuàng)建集群定義文件#cd /home#mkdir gpfs#cd gpfs#vi jxrcc_cluster8:quorum-manager a) 創(chuàng)建GPFS集群#mmcrcluster N jxrccb_cluster p 8 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_cluster A# mmcrcluster -N jxrccb_cluster -p 8 -r /usr/bin/s

26、sh -R /usr/bin/scp -C jxrccb_cluster -AWarning: Permanently added (RSA) to the list of known hosts.Tue Nov 19 20:01:38 CST 2013: 6027-1664 mmcrcluster: Processing node mmcrcluster: Command successfully completedmmcrcluster: 6027-1254 Warning: Not all nodes ha

27、ve proper GPFS license designations. Use the mmchlicense command to designate licenses as needed.b) 配置許可# mmchlicense server -accept -N 8# mmchlicense server -accept -N 8The following nodes will be designated as possessing GPFS server licenses: mmchlicense: Co

28、mmand successfully completedc) 添加Linux節(jié)點通過如下命令依次添加48個Linux節(jié)點：# mmaddnode N # mmchlicense server -accept -N # mmaddnode N # mmchlicense server -accept -N 查看集群狀態(tài)#mmlsclusterd) 修改集群參數(shù)#mmchconfig prefetchThreads=1000 N 1

29、8# mmchconfig pagepool=4G N 8# mmchconfig failureDetectionTime=15# mmchconfig MaxMBps=4096e) 啟動集群#mmstartup -a啟動后，查看集群狀態(tài)#mmgetstate -a確保所有節(jié)點的狀態(tài)為Activen 創(chuàng)建NSD磁盤a) 修改fscsi參數(shù)在所有直接訪問SAN盤的節(jié)點上執(zhí)行#lsdev |grep fscsi|awk print chdev -l,$1,-a dyntrk=yes -a fc_err_recov=fast_fail|shb) 識別外置存儲磁

30、盤在所有直接訪問SAN盤的節(jié)點上執(zhí)行#cfgmgrc) 修改hdisk參數(shù)在所有直接訪問SAN盤的節(jié)點上執(zhí)行#lsdev -Cc disk|grep MPIO |awk print chdev -l,$1,-a algorithm=round_robin -a queue_depth=8 -a reserve_policy=no_reserve|shd) 創(chuàng)建NSD描述文件#cd /gpfs#vi nsd.confhdisk2: 8:jxrccb_nsd1:hdisk3: 8:jxrccb_nsd2:備注文件格式：DiskName:ServerList:

31、DiskUsage:FailureGroup:DesiredName:StoragePool在本集群中，將AIX節(jié)點定義為NSD Server，如果NSD較多可以將每塊盤的首選NSD Server設置為不同，以均衡NSD Sever的負載；e) 生成NSD盤#mmcrnsd -F nsd.conff) 查看NSD盤#mmlsnsd# mmlsnsd File system Disk name NSD servers - (free disk) jxrccb_nsd1 (free disk) jxrccb_nsd2 gpfs98.abs.jxnxs.co

32、m n 定義Tiebreaker disks集群處于Down狀態(tài)時才能定義；#mmshutdown -a# mmchconfig tiebreakerDisks= 定義后可用如下命令查看；# mmlsconfign 創(chuàng)建GPFS文件系統(tǒng)#mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfs# mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfsGPFS: 6027-531 The following disks of gpfs will be formatted on node ABSGPFS1: jxrccb_nsd1: size 536870912 KB jxrccb_nsd2: size 536870912 KBGPFS: 6027-540 Formatting file system .GPFS: 6027-535 Disks up to size 4.4 TB can be added to storage pool system.Creating Inode FileCreating Allocation MapsCreating Log FilesClearing In