为什么需要生成词云

对于文本数据有个直观的了解，为后续的工作提供一定的数据可视化分析依据。

词云的生成步骤：

• 导入工具库
• 读取数据
• 清洗数据
• 统计词频保留前K个词作为词云生成库
• 绘制词云图

导入工具库

import warnings
warnings.filterwarnings('ignore')
import jieba
import numpy
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import matplotlib
matplotlib.rcParams['figure.figsize'] = (10,5)
from wordcloud import WordCloud

读取数据

df = pd.read_csv("./entertainment_news.csv", encoding='utf-8')

清洗数据

使用结巴分词

为了在处理数据时不会因为某条数据有问题，导致整个任务停止，故使用try except continue
df = df.dropna()
content=df["content"].values.tolist()
segment=[]
for line in content:
    try:
        segs=jieba.lcut(line)
        for seg in segs:
            if len(seg)>1 and seg!='\r\n':
                segment.append(seg)
    except:
        print(line)
        continue      

统计词频保留前K个词作为词云生成库

# 去除停用词
words_df = pd.DataFrame({'segment':segment})
stopwords = pd.read_csv("origin_data/stopwords.txt",index_col=False,quoting=3,sep='\t',names=['stopword'],encoding='utf-8')
words_df = words_df[~words_df.segment.isin(stopwords.stopword)]

words_stat = words_df.groupby(by=['segment'])['segment'].agg({'计数':numpy.size})
words_stat = words_stat.reset_index().sort_values(by=['计数'],ascending=False)
words_stat.head()

绘制词云图

matplotlib.rcParams['figure.figsize'] = (12.0, 12.0)
wordcloud=WordCloud(font_path='origin_data/simhei.ttf',background_color='black',max_font_size=80)
word_frequence = {x[0]:x[1] for x in words_stat.head(1000).values}
wordcloud=wordcloud.fit_words(word_frequence)
plt.imshow(wordcloud)