scrapy爬虫不调用process_item函数
·
首先要确定setting中ITEM_PIPELINES设置已添加:
# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'spider.spider.pipelines.SpiderPipeline': 300,
}
pipline生成文件中有如下提示:
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
然后检查parse()函数中创建item后是否返回,即是否有yield item
即以下代码的spiderItem
for item in employItems:
spiderItem = SpiderItem()
spiderItem['base'] = item.css('p::text').extract_first()[0:2]
spiderItem['company'] = item.css('.company-text>.name>a::text').extract_first()
spiderItem['position'] = item.css('.job-title::text').extract_first()
spiderItem['minPayment'] = item.css('.red::text').extract_first()[0:2]
spiderItem['maxPayment'] = item.css('.red::text').extract_first()[4:6]
spiderItem['eduBg'] = item.css('p::text').extract()[-2:]
spiderItem['updateTime'] = item.css('.info-publis>p::text').extract_first()[3:]
yield spiderItem
更多推荐
所有评论(0)