scrapy簡單學(xué)習(xí)6—爬取百度貼吧圖片（仿寫向）

trigkit4 發(fā)布于2019-07-24 18:10 / 3058人閱讀

摘要：主要是對上一篇文章的簡單仿寫，大家以后想批量下載什么圖片照格式仿寫就好。

主要是對上一篇文章的簡單仿寫，大家以后想批量下載什么圖片照格式仿寫就好。由于本人是tfboys的粉絲，所以平常沒事愛逛貼吧欣賞我家三小只的美圖，所以這次就以貼吧[小王的討論樓]為例，批量爬取該樓的圖片[1]

itme.py編寫

import scrapy


class WangyuantuItem(scrapy.Item):
    image_urls=scrapy.Field()#就編寫個圖片路徑就好

spider的編寫

import scrapy
import requests
import os
from wangyuantu.items import WangyuantuItem

class XiaowangSpider(scrapy.Spider):
    name = "xiaowang"
    allowed_domains = ["tieba.baidu.com/p/3888309273"]
    start_urls = [
        "http://tieba.baidu.com/p/3888309273?pn=%d" % i for i in range(21,45)
        ]
    
    

    def parse(self, response):
        item = WangyuantuItem()
        item["image_urls"]=response.xpath("http://img[@class="BDE_Image"]/@src").extract()
        
        yield item

pipelines編寫：這個部分都是可以套用的

import requests
from wangyuantu import settings
import os

#圖片下載類
class ImageDownloadPipeline(object):
    def process_item(self, item, spider):
        if "image_urls" in item:#如何‘圖片地址’在項目中
            images = []#定義圖片空集
            
            dir_path = "%s/%s" % (settings.IMAGES_STORE, spider.name)

            if not os.path.exists(dir_path):
                os.makedirs(dir_path)
            for image_url in item["image_urls"]:
                us = image_url.split("/")[3:]
                image_file_name = "_".join(us)
                file_path = "%s/%s" % (dir_path, image_file_name)
                images.append(file_path)
                if os.path.exists(file_path):
                    continue

                with open(file_path, "wb") as handle:
                    response = requests.get(image_url, stream=True)
                    for block in response.iter_content(1024):
                        if not block:
                            break

settings編寫

BOT_NAME = "wangyuantu"

SPIDER_MODULES = ["wangyuantu.spiders"]
NEWSPIDER_MODULE = "wangyuantu.spiders"

ITEM_PIPELINES = {"wangyuantu.pipelines.ImageDownloadPipeline": 1}
#圖片儲存
IMAGES_STORE = "C:UsersLenovoPictures"

結(jié)果

寄語：wili源源小可愛，希望你快快樂樂的長大