第二章爬蟲爬蟲的基本流程HTTP協(xié)議總結(jié)、安裝PyCharm、應(yīng)用舉例目錄爬蟲的基本流程模擬瀏覽器發(fā)送請(qǐng)求(獲取網(wǎng)頁代碼)→提取有用的數(shù)據(jù)→存放于數(shù)據(jù)庫或文件中HTTP協(xié)議Request：用戶將自己的信息通過瀏覽器發(fā)送給服務(wù)器。Response：服務(wù)器接收請(qǐng)求，分析用戶發(fā)來的請(qǐng)求信息，然后返回?cái)?shù)據(jù)（返回的數(shù)據(jù)中可能包含其他鏈接，如：圖片，Javascript，CSS等）。總結(jié)爬蟲流程爬取→解析→存儲(chǔ)。爬蟲所需工具請(qǐng)求庫：Requests,Selenium（可以驅(qū)動(dòng)瀏覽器解析渲染CSS和Javascript，但有性能劣勢(shì)，有用沒用的網(wǎng)頁都會(huì)加載）解析庫：正則表達(dá)式，BeautifulSoup，Pyquery存儲(chǔ)庫：文件，MySQL，Mongodb，Redis安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)安裝AnPaycCohnadram(安社裝區(qū)版)如果想建立一個(gè)爬蟲文件，首先檢查是否安裝了requests，選擇File→Settings→Project

Interpreter選項(xiàng)，然后單擊右上角的+，搜requests，選中后點(diǎn)左下角的Install

package，進(jìn)行安裝。應(yīng)用舉例【例2-1】爬取豆瓣讀書Top

250第一頁的內(nèi)容，并存入douban.csv文件中。應(yīng)用舉例import

requestsimport

refrom

bs4

import

BeautifulSoupdef

gethtml(url):try:r

=

requests.get(url,timeout

=

30)r.raise_for_status()r.encoding

=

r.apparent_encodingreturn

r.textexcept:return

"It

is

failed

to

get

html!“def

getcontent(url):headers

=

{"User-Agent":"Mozilla/5.0

(Windows

NT

10.0;

WOW64)應(yīng)用舉例data

=

requests.get(url,headers=headers)

soup

=

BeautifulSoup(data.text,

"lxml")div

=

soup.find("div",id="content")

tables

=

div.find_all("table")price

=

[]date

=

[]nationality

=

[]nation

=

[]bookname=[]link

=

[]score

=

[]comment

=

[]num

=

[]應(yīng)用舉例for

table

in

tables:bookname.append(table.find_all("a")[1]["title"])

link.append(table.find_all("a")[1]["href"])score.append(table.find("span",class_="rating_nums").string)

comment.append(table.find_all("span")[-1].string)people_info

=

table.find_all("span")[-2].textpeople_num=re.findall(r"\d+",

people_info)num.append(people_num[0])navistr

=

(table.find("p").string)infos

=

str(navistr.split("/"))infostr

=

str(navistr)應(yīng)用舉例if

re.findall(r"\[",

s[0]):w

=

re.findall(r"\s\D+",

s[0])author.append(w[0])else:author.append(s[0])price_info

=

re.findall(r"\d+\.\d+",

infos)price.append((price_info[0]))date.append(s[-2])nationality_info

=

re.findall(r"[[](\D)[]]",

infos)nationality.append(nationality_info)for

i

in

nationality:if

len(i)

==

1:nation.append(i[0])else:應(yīng)用舉例dataframe=pd.DataFrame({"書名":bookname,"作者":author,"國籍

nation,"評(píng)分":score,"評(píng)分人數(shù)":num,"出版時(shí)間":date,"價(jià)格":pr接":link,})#將DataFrame存儲(chǔ)為csv,index表示是否顯示行名，default=True

dataframe.to_csv("douban.csv",index=False,encoding=