語料庫基礎(chǔ)Python自然語言處理第三章CONTENT目錄

01語料庫基礎(chǔ)03獲取語料庫02自然語言處理工具包NLTK04綜合案例：走進紅樓夢課前回顧NLTK簡介NLTK安裝NLTK使用StanfordNLP獲取語料庫03國內(nèi)外著名語料庫網(wǎng)絡(luò)數(shù)據(jù)獲取NLTK獲取語料庫英文語料庫BNC——英國國家語料庫（BritishNationalCorpus）：/?target=http%3A//www.natcorp.ox.ac.uk/BOE——柯林斯英語語料庫（theBankofEnglish）：/lanBASE——英國學(xué)術(shù)口語語料庫（BritishAcademicSpokenEnglishCorpus）：http://www2.warwick.ac.uk/fac/Lextutor：http://www.lextutor.ca/MyMemory：TAUS：/index.php/中文語料庫(1)北外語料庫語言學(xué)：/

(2)現(xiàn)代漢語平衡語料庫：.tw/SinicaCorpus/(3)古漢語語料庫：.tw/ftms-bin/ftmsw(4)近代漢語標(biāo)記語料庫：.tw/Early_Mandarin/(5)樹圖數(shù)據(jù)庫：.tw/(6)搜文解字：.tw/(7)漢籍電子文獻：.tw/~tdbproj/handy1/中文語料庫(8)中國傳媒大學(xué)文本語料庫檢索系統(tǒng)：/RawPub/(9)哈工大信息檢索研究室對外共享語料庫資源：/demo/ltp/Sharing_Plan.htm(10)香港教育學(xué)院語言資訊科學(xué)中心及其語料庫實驗室：/index.php?lang=sc(11)中文語言資源聯(lián)盟：/(12)搜狗實驗室新聞|互聯(lián)網(wǎng)數(shù)據(jù)：/labs/(13)數(shù)據(jù)堂：/(14)文國尋寶記：.tw/wen/獲取語料庫03國內(nèi)外著名語料庫網(wǎng)絡(luò)數(shù)據(jù)獲取NLTK獲取語料庫網(wǎng)絡(luò)數(shù)據(jù)獲取除了通過訪問網(wǎng)絡(luò)獲取，還可以通過編寫程序的方式訪問網(wǎng)絡(luò)和硬盤文本的方式獲取，例如：通過編寫程序，在線獲取傷寒雜病論的語料庫，實現(xiàn)代碼如下：from__future__importdivisionimportnltk,re,pprintfromurllib.requestimporturlopenurl=r'/files/24272/24272-0.txt'raw=urlopen(url).read()raw=raw.decode('utf-8')print(len(raw))print(raw[1500:2000])網(wǎng)絡(luò)數(shù)據(jù)獲取運行結(jié)果如圖：網(wǎng)絡(luò)數(shù)據(jù)獲取再例如，通過編寫程序，在線獲取處理HTML文本（紅樓夢），實現(xiàn)代碼如下所示：importre,nltkfromurllib.requestimporturlopenurl='/cache/epub/24264/pg24264-images.html'html=urlopen(url).read()html=html.decode('utf-8')print(html[6000:6500])網(wǎng)絡(luò)數(shù)據(jù)獲取運行結(jié)果如圖：獲取語料庫03國內(nèi)外著名語料庫網(wǎng)絡(luò)數(shù)據(jù)獲取NLTK獲取語料庫網(wǎng)絡(luò)和聊天文本步驟一：獲取網(wǎng)絡(luò)聊天文本，代碼如下所示：fromnltk.corpusimportwebtextforfileidinwebtext.fileids(): print(fileid,webtext.raw(fileid))網(wǎng)絡(luò)和聊天文本步驟二：查看網(wǎng)絡(luò)聊天文本信息，代碼如下所示：forfileidinwebtext.fileids(): print(fileid,len(webtext.words(fileid)),len(webtext.raw(fileid)),len(webtext.sents(fileid)),webtext.encoding(fileid))運行結(jié)果：

firefox.txt1024575646011142ISO-8859-2grail.txt16967650031881ISO-8859-2overheard.txt21841383011817936ISO-8859-2pirates.txt22679953681469ISO-8859-2singles.txt486721302316ISO-8859-2wine.txt313501497722984ISO-8859-2網(wǎng)絡(luò)和聊天文本步驟三：獲取即時消息聊天會話語料庫，代碼如下所示：運行結(jié)果：

fromnltk.corpusimportnps_chatchatroom=nps_chat.posts('10-19-20s_706posts.xml')chatroom[123]['i','do',"n't",'want','hot','pics','of','a','female',',','I','can','look','in','a','mirror','.']布朗語料庫步驟一：查看語料庫信息，實現(xiàn)代碼如下所示：運行結(jié)果：

fromnltk.corpusimportbrownprint(brown.categories())['adventure','belles_lettres','editorial','fiction','government','hobbies','humor','learned','lore','mystery','news','religion','reviews','romance','science_fiction']布朗語料庫步驟二：比較文體中情態(tài)動詞的用法，實現(xiàn)代碼如下所示：運行結(jié)果：

can:94could:87may:93might:38must:53will:389importnltkfromnltk.corpusimportbrownnew_texts=brown.words(categories='news')fdist=nltk.FreqDist([w.lower()forwinnew_texts])modals=['can','could','may','might','must','will']forminmodals:print(m+':',fdist[m])布朗語料庫步驟三：NLTK條件概率分布函數(shù)，實現(xiàn)代碼如下所示：cfd=nltk.ConditionalFreqDist((genre,word)forgenreinbrown.categories()forwordinbrown.words(categories=genre))genres=['news','religion','hobbies','science_fiction','romance','humor']modals=['can','could','may','might','must','will']cfd.tabulate(condition=genres,samples=modals)運行結(jié)果如右圖：就職演說語料庫步驟一：查看語料信息，實現(xiàn)代碼如下所示：運行結(jié)果如圖：fromnltk.corpusimportinaugurallen(inaugural.fileids())inaugural.fileids()58['1789-Washington.txt','1793-Washington.txt','1797-Adams.txt','1801-Jefferson.txt','1805-Jefferson.txt','1809-Madison.txt','1813-Madison.txt','1817-Monroe.txt','1821-Monroe.txt','1825-Adams.txt','1829-Jackson.txt','1833-Jackson.txt','1837-VanBuren.txt','1841-Harrison.txt','1845-Polk.txt','1849-Taylor.txt','1853-Pierce.txt','1857-Buchanan.txt','1861-Lincoln.txt','1865-Lincoln.txt','1869-Grant.txt','1873-Grant.txt','1877-Hayes.txt','1881-Garfield.txt','1885-Cleveland.txt','1889-Harrison.txt','1893-Cleveland.txt','1897-McKinley.txt','1901-McKinley.txt','1905-Roosevelt.txt','1909-Taft.txt','1913-Wilson.txt','1917-Wilson.txt','1921-Harding.txt','1925-Coolidge.txt','1929-Hoover.txt','1933-Roosevelt.txt','1937-Roosevelt.txt','1941-Roosevelt.txt','1945-Roosevelt.txt','1949-Truman.txt','1953-Eisenhower.txt','1957-Eisenhower.txt','1961-Kennedy.txt','1965-Johnson.txt','1969-Nixon.txt','1973-Nixon.txt','1977-Carter.txt','1981-Reagan.txt','1985-Reagan.txt','1989-Bush.txt','1993-Clinton.txt','1997-Clinton.txt','2001-Bush.txt','2005-Bush.txt','2009-Obama.txt','2013-Obama.txt','2017-Trump.txt']就職演說語料庫步驟二：查看演說語料的年份，實現(xiàn)代碼如下所示：運行結(jié)果：

print([fileid[:4]forfileidininaugural.fileids()])['1789','1793','1797','1801','1805','1809','1813','1817','1821','1825','1829','1833','1837','1841','1845','1849','1853','1857','1861','1865','1869','1873','1877','1881','1885','1889','1893','1897','1901','1905','1909','1913','1917','1921','1925','1929','1933','1937','1941','1945','1949','1953','1957','1961','1965','1969','1973','1977','1981','1985','1989','1993','1997','2001','2005','2009','2013','2017']就職演說語料庫步驟三：條件概率分布，實現(xiàn)