当前位置：首页>Php>正文

你好，爬取《你好，李煥英》影評，并生成詞云圖

Php12-25

你好，爬取《你好，李煥英》影評，并生成詞云圖

自學了python爬蟲，最近在實踐，就拿《你好，李煥英》的豆瓣影評來試試手吧！

思路：

首先是爬取豆瓣影評的短評，保存下來
豆瓣影評每頁顯示20條評論，我爬取了前面50頁的評論，先瀏覽找到翻頁規律，批量生成網頁url鏈接，然后解析每個頁面，用的BeautifulSoup，提取評論文字，保存為txt文件
然后將評論生成詞云
代碼里注釋的很清晰了，就不贅述了

上代碼

你好。爬取評論的代碼

#引用time庫、random庫、requests庫、BeautifulSoup4
import timeimport randomimport requestsfrom bs4 import BeautifulSoupdef get_info(url):dict = {}UA = ["Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36","Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/537.75.14","Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)",'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11','Opera/9.25 (Windows NT 5.1; U; en)','Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)','Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Kubuntu)','Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/20070731 Ubuntu/dapper-security Firefox/1.5.0.12','Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/1.2.9',"Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.7 (KHTML, like Gecko) Ubuntu/11.04 Chromium/16.0.912.77 Chrome/16.0.912.77 Safari/535.7","Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 "]user_agent = random.choice(UA)#請求頁面r = requests.get(url,headers={'User-Agent':user_agent})#創建BeautifulSoup對象soup = BeautifulSoup(r.text,'lxml')#找到所有的class屬性為comment-item的標簽comment_items = soup.select('.comment-item')#遍歷所有符合要求的標簽for comment_item in comment_items:#找到包含短評內容的標簽shorts = comment_item.select('.short')#找到包含時間的標簽times = comment_item.select('.comment-time')#遍歷找到的標簽for short,time in zip(shorts,times):#提取文字short_text = short.get_text()#提取時間time_num = time['title']#把文字保存下來#使用前先創建一個txt文件with open('你好李煥英.txt','a+',encoding='utf-8') as f:f.write(short_text + '\n')print(short_text,time_num)dict = {'short':short_text,'time':time_num}return dictif __name__ == '__main__':#生成大量的url鏈接urls = ['https://movie.douban.com/subject/34841067/comments?start={}&limit=20&status=P&sort=new_score'.format(str(i)) for i in range(0,1000,20)]for url in urls:get_info(url)time.sleep(1)

生成詞云的代碼

import jieba
from wordcloud import wordcloud,WordCloud,ImageColorGenerator 
from matplotlib import colors
from imageio import imread#打開模板圖片，賦值給mask
mask = imread("火.png")#打開文本文件，只讀模式，utf8編碼
f = open("你好李煥英.txt","r",encoding="utf-8")
t = f.read()
#提取模板顏色
image_colors = ImageColorGenerator(mask)
#關閉文件
f.close()#文本分詞，賦值給列表
ls = jieba.lcut(t)
#以空格分隔列表里的每個元素
txt = " ".join(ls)#定義詞云字體，形狀，界面高、寬，背景色，最大文字數量，最大字號，字號遞增為1，最大字號為6
w = wordcloud.WordCloud(font_path = "msyh.ttc",mask=mask,\width = 1000,height = 700,background_color = "white",\max_words = 1000,min_font_size=3,font_step=3,\max_font_size=50,)w.generate(txt)#修改詞云字體顏色為提取的模板顏色
w_color = w.recolor(color_func=image_colors)
#導出為png文件
w.to_file("你好，李煥英詞云.png")