問題描述

用scrapy-splash爬取網站 為啥iframe下的內容沒有被返回？

import scrapyfrom scrapy import Selectorclass Music163Spider(scrapy.Spider): name = 'music163' allowed_domains = ['music.163.com/#/discover/toplist'] start_urls = [’http://music.163.com/#/discover/toplist/’] def parse(self, response):print('parse:',response.text) def start_requests(self):for url in self.start_urls: yield scrapy.Request(url, self.parse, meta={’splash’: { ’endpoint’: ’render.html’, ’args’: {’wait’: 0.5, }} })

問題解答

你需要在parse中獲取iframe的url再次請求內容。