我的任务是使用 Scrapy 从网站上提取 pdf 文件。我对 Python 并不陌生，但 Scrapy 对我来说却很陌生。我一直在尝试控制台和一些基本的蜘蛛。我找到并修改了这段代码：
import urlparse
import scrapy
from scrapy.http import Request
class pwc_tax(scrapy.Spider):
name = "pwc_tax"
allowed_domains = ["www.pwc.com"]
start_urls = ["http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html"]
def parse(self, response):
base_url = "http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html"
for a in response.xpath('//a[@href]/@href'):
link = a.extract()
if link.endswith('.pdf'):
link = urlparse.urljoin(base_url, link)
yield Request(link, callback=self.save_pdf)
def save_pdf(self, response):
path = response.url.split('/')[-1]
with open(path, 'wb') as f:
f.write(response.body)
我在命令行中运行此代码
scrapy crawl mySpider
我什么也没得到。我没有创建 scrapy 项目，因为我想抓取并下载文件，没有元数据。我将不胜感激任何帮助。
蜘蛛的逻辑似乎不正确。
我快速浏览了您的网站，似乎有几种类型的页面：
-
http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html初始页面
- 特定文章的网页，例如http://www.pwc.com/us/en/tax-services/publications/insights/australia-introduces-new-foreign-resident-cgt-withholding-regime.html http://www.pwc.com/us/en/tax-services/publications/insights/australia-introduces-new-foreign-resident-cgt-withholding-regime.html可以从第 1 页导航
- 实际的 PDF 位置，例如http://www.pwc.com/us/en/state-local-tax/newsletters/salt-insights/assets/pwc-wotc-precertification-period-extended-to-june-29.pdf http://www.pwc.com/us/en/state-local-tax/newsletters/salt-insights/assets/pwc-wotc-precertification-period-extended-to-june-29.pdf可以从第 2 页进行导航
因此，正确的逻辑如下：首先获取#1 页面，然后获取#2 页面，然后我们可以下载#3 页面。
然而，您的蜘蛛会尝试直接从#1 页面提取指向#3 页面的链接。
EDITED:
我已经更新了您的代码，这是实际有效的代码：
import urlparse
import scrapy
from scrapy.http import Request
class pwc_tax(scrapy.Spider):
name = "pwc_tax"
allowed_domains = ["www.pwc.com"]
start_urls = ["http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html"]
def parse(self, response):
for href in response.css('div#all_results h3 a::attr(href)').extract():
yield Request(
url=response.urljoin(href),
callback=self.parse_article
)
def parse_article(self, response):
for href in response.css('div.download_wrapper a[href$=".pdf"]::attr(href)').extract():
yield Request(
url=response.urljoin(href),
callback=self.save_pdf
)
def save_pdf(self, response):
path = response.url.split('/')[-1]
self.logger.info('Saving PDF %s', path)
with open(path, 'wb') as f:
f.write(response.body)
本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)
