大數(shù)據(jù)采集與預(yù)處理技術(shù)*

*項(xiàng)目一表情圖像數(shù)據(jù)采集和預(yù)處理序號(hào)軟件配置要求1python3運(yùn)行計(jì)算機(jī)內(nèi)存8G以上2pytorch最新版本3pycharm最新版本一、項(xiàng)目目標(biāo)：1、完成表情圖像數(shù)據(jù)采集環(huán)境安裝配置，掌握爬蟲必備知識(shí)；2、完成表情圖像數(shù)據(jù)爬蟲采集程序設(shè)計(jì)和數(shù)據(jù)采集；3、完成表情圖像數(shù)據(jù)預(yù)處理。二、環(huán)境要求：任務(wù)二表情圖像數(shù)據(jù)采集程序一、任務(wù)目標(biāo)1、完成表情圖像數(shù)據(jù)爬蟲程序編寫；2、完成表情圖像數(shù)據(jù)的采集和保存；二、知識(shí)儲(chǔ)備1、爬蟲網(wǎng)站結(jié)構(gòu)分析；2、爬蟲程序編寫方法；1、爬蟲程序編寫1）導(dǎo)入需要的庫(kù)導(dǎo)入requests庫(kù)：importrequests導(dǎo)入etree庫(kù)：fromlxmlimportetree2）構(gòu)造請(qǐng)求頭信息headers={"user-agent":"Mozilla/5.0(WindowsNT10.0;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/96.0.4664.110Safari/537.36"}3）構(gòu)造循環(huán)爬取字母A-Z的用戶照片。（請(qǐng)問(wèn)循環(huán)的含義和作用？）forclass_numberinrange(65,90):#url是需要爬取以class_number開(kāi)頭的圖片地址url='/photos/people-'+chr(class_number)+'.html’#發(fā)起請(qǐng)求response=requests.get(url,headers)#獲取返回?cái)?shù)據(jù)html_data=etree.HTML(response.text)#找到要爬取的人名及其urlcelebs_url_list=html_data.xpath('//div[@class="model_card"]/a/@href')name_list=html_data.xpath('//div[@class="model_card"]/a/div/span/text()')print(celebs_url_list)print(name_list)4）針對(duì)每個(gè)人進(jìn)行爬蟲循環(huán)foriinrange(0,len(celebs_url_list)):#獲取每個(gè)人的相應(yīng)頁(yè)面內(nèi)容person_url=''+celebs_url_list[i]person_res=requests.get(person_url,headers=headers).textperson_data=etree.HTML(person_res)#print(person_data)#查找每個(gè)人的照片頁(yè)數(shù)，通過(guò)item類查找page_number_info=person_data.xpath('//li[@class="item"][last()]/a/text()')#初始化頁(yè)數(shù)是1page_num=1#如果有更多頁(yè)，指向最大頁(yè)iflen(page_number_info)>0:page_num=int(page_number_info[0])print('當(dāng)前人圖像頁(yè)數(shù)：{}'.format(page_num))4）針對(duì)每個(gè)人進(jìn)行爬蟲循環(huán)(接上頁(yè)循環(huán)，注意每個(gè)用戶的首頁(yè)照片地址和后邊各頁(yè)地址的區(qū)別) #detail_url保存要爬取的照片列表頁(yè)地址 detail_url='' forpageinrange(1,page_num+1): ifpage==1: #第一頁(yè)使用個(gè)人頁(yè)第一頁(yè)

detail_url=person_url elifpage>1: #如果有下一頁(yè)，構(gòu)造下一頁(yè)的url

detail_url=person_url+'page{}/'.format(page)5）對(duì)每張圖片進(jìn)行爬取保存#爬取每張圖片進(jìn)行保存forimage_infoinimg_url_list: image_complete_url="/"+image_info image_get_result=requests.get(image_complete_url,headers=headers).content

f=open("images/"+name_list[i]+"-"+str(page)+"-"+str(img_url_list.index(image_info))+".jpg","wb") f.write(image_get_result)2、完整代碼#導(dǎo)入需要的庫(kù)importos.pathimportrequestsfromlxmlimportetreeheaders={"user-agent":"Mozilla/5.0(WindowsNT10.0;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/96.0.4664.110Safari/537.36"}#循環(huán)爬取字母A-B的用戶照片forclass_numberinrange(65,66):#url是需要爬取圖片的網(wǎng)頁(yè)

url='/photos/people-'+chr(class_number)+'.html'response=requests.get(url,headers)html_data=etree.HTML(response.text)celebs_url_list=html_data.xpath('//div[@class="model_card"]/a/@href')name_list=html_data.xpath('//div[@class="model_card"]/a/div/span/text()')#print(celebs_url_list)#print(name_list)foriinrange(0,len(celebs_url_list)):#獲取相應(yīng)頁(yè)面內(nèi)容

person_url=''+celebs_url_list[i]#print(person_url)person_res=requests.get(person_url,headers=headers).textperson_data=etree.HTML(person_res)#print(person_data)

#查找頁(yè)數(shù)，通過(guò)item類查找

page_number_info=person_data.xpath('//li[@class="item"][last()]/a/text()')#初始化頁(yè)數(shù)是1page_num=1#如果有更多頁(yè)，指向最大頁(yè)

iflen(page_number_info)>0:page_num=int(page_number_info[0])print('當(dāng)前人圖像頁(yè)數(shù)：{}'.format(page_num))detail_url=''forpageinrange(1,page_num+1):ifpage==1:print("first")detail_url=person_urlelifpage>1:print("second")#如果有下一頁(yè)，構(gòu)造下一頁(yè)的urldetail_url=person_url+'page{}/'.format(page)#發(fā)起請(qǐng)求

print(detail_url)res=requests.get(detail_url,headers=headers).textdata=etree.HTML(res)#圖片的urlimg_url_list=data.xpath('//div[@class="gallery-pics-listd-flexflex-wrap"][1]/div/div/a/img/@src')print(len(img_url_list))ifnotimg_url_list:print('break')break#對(duì)每張圖片進(jìn)行爬取保存

forimage_infoinimg_url_list:image_complete_url="/"+image_infoimage_get_result=requests.get(image_complete_url,headers=headers).contentf=open("images/"+name_list[i]+"-"+str(page)+"-"+str(img_url_list.index(image_info))+".jpg","wb")f.write(image_get_result)3、執(zhí)行結(jié)果部分爬取圖片列表爬取圖片結(jié)果（部分）4、拓展要求1