为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？

2024-03-29

我正在抓取一个网站来检查各种产品的库存状态。不幸的是，这需要实际单击产品页面上的“添加到购物车”并检查下一页的消息以确定是否有库存（即它需要解析两个响应）。

我跟着优秀的文档 http://doc.scrapy.org/en/latest/topics/request-response.html#passing-additional-data-to-callback-functions对于这种情况，并编写了我的解析函数来返回Request带有对我的辅助解析函数的回调的对象。然而，这个函数很少被调用。大多数产品只会在日志中看到“退货请求之前”，但一小部分产品确实会正确调用它。

知道这里出了什么问题吗？我已经没有主意了。

foo/spiders/atlantic_firearms_spider.py:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.http import FormRequest
from foo.items import AtlanticFirearmsItem

import datetime
import re

class AtlanticFirearmsSpider(CrawlSpider):
    name = "atlantic_firearms"
    allowed_domains = ["atlanticfirearms.com"]
    start_urls = [
        "http://www.atlanticfirearms.com"
    ]

    rules = (
        Rule(SgmlLinkExtractor(allow=['detail.html']), callback='parse_product'),
        Rule(SgmlLinkExtractor(allow=[], deny=['/bro', '/news', '/howtobuy', '/component/search', 'askquestion'])),
    )

    def parse_product(self, response):
      hxs = HtmlXPathSelector(response)
      product = AtlanticFirearmsItem()
      add_to_cart = any([hxs.select("descendant-or-self::input[@name = 'addtocart']"),
                         hxs.select("descendant-or-self::input[@value = 'Add to Cart']"),
                         hxs.select("//a[text() = 'Add to Cart']")])
      product['url'] = response.url
      product['as_of_time'] = datetime.datetime.now()

      if add_to_cart:
          # attempt to add to cart to verify availability
          request = FormRequest.from_response(response, formname="addtocartForm", callback=self.parse_add_to_cart)
          request.meta['product'] = product
          print "Before return request"
          return request
      else:
          product['in_stock'] = False
          return product

    def parse_add_to_cart(self, response):
        print "Inside parse_add_to_cart"
        product = response.meta['product']
        hxs = HtmlXPathSelector(response)
        product['in_stock'] = not(hxs.select("//text()[contains(.,'We regret to inform you that this product')]"))
        return product

foo/items.py:

from scrapy.item import Item, Field

class AtlanticFirearmsItem(Item):
    in_stock = Field()
    url = Field()
    as_of_time = Field()

Edit：按要求添加日志文件：

2013-09-21 07:25:14-0500 [scrapy] INFO: Scrapy 0.18.2 started (bot: foo)
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Optional features available: ssl, http11
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Overridden settings: {'SPIDER_MODULES': ['foo.spiders'], 'BOT_NAME': 'foo'}
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRef
reshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Enabled item pipelines: 
2013-09-21 07:25:14-0500 [atlantic_firearms] INFO: Spider opened
2013-09-21 07:25:14-0500 [atlantic_firearms] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-09-21 07:25:14-0500 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-09-21 07:25:16-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com> (referer: None)
2013-09-21 07:25:16-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.cloudflare.com': <GET http://www.cloudflare.com/email-protection>
2013-09-21 07:25:16-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.constantcontact.com': <GET http://www.constantcontact.com/jmml/email-marketing.jsp>
2013-09-21 07:25:16-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.fdicreative.com': <GET http://www.fdicreative.com/>
2013-09-21 07:25:16-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.redjacketfirearms.com': <GET https://www.redjacketfirearms.com/>
2013-09-21 07:25:17-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/wolf-ammunition-45acp-500-round-case-detail.
html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:18-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/vector-arms-sp89-k-style-pistol-9mm-detail.h
tml?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:18-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:18-0500 [atlantic_firearms] DEBUG: Filtered duplicate request: <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/vector-arms-sp89-k-style-pisto
l-9mm-detail.html?Itemid=0> - no more duplicates will be shown (see DUPEFILTER_CLASS)
2013-09-21 07:25:18-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/wolf-223-ar15-rifle-ammo-500-round-case-deta
il.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:18-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/us-palm-air-save-plate-carrier-detail.html?I
temid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:19-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/545-x-39-russian-ak74-ammo-1080-round-case-d
etail.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:19-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/red-army-standard-7-62x39mm-360-round-range-
pack-detail.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:19-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/vector-arms-mp5-style-rifle-detail.html?Itemid=0> (
referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:19-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-accessories/wolf-ammunition-for-sale-ak47-detail.html?Item
id=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:20-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/dsa-zm4-flat-top-ar15-carbine-dszm4cv1r-detail.html
?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:20-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-accessories/m92-ak47-yugoslavian-7-62x39mm-bolt-hold-open-
metal-mags-pack-of-two-detail.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:21-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/vector-arms-v94-9mm-mp5-style-pistol-full-size-deta
il.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:21-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/zastava-ak-47-m70b1-pap-7-62x39mm-rifles-w-2-hi-cap
-mags-detail.html?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:21-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/ptr-91-gi-rifle-939-atlanticfirearms-com-detail.htm
l?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:21-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/pap-m92-7-62x39-pistol-detail.html?Itemid=0> (refer
er: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/content/article/86-static-pages/159-resources.html> (referer: http://www.atlan
ticfirearms.com)
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.atsconsultingcorp.com': <GET http://www.atsconsultingcorp.com/>                                  [52/1905]
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.bullseyemarket.com': <GET http://www.bullseyemarket.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.corilam.com': <GET http://www.corilam.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'chancebrownrealestate.com': <GET http://chancebrownrealestate.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.delsolservices.com': <GET http://www.delsolservices.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.elkhornoutfitters.com': <GET http://www.elkhornoutfitters.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.frontierlogistics.com': <GET http://www.frontierlogistics.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.gpstrackingkey.com': <GET http://www.gpstrackingkey.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.hanshawkennedy.com': <GET http://www.hanshawkennedy.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'worldenv.com': <GET http://worldenv.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'purgexonline.com': <GET http://purgexonline.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'bumpfirestocks.com': <GET http://bumpfirestocks.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.texrestaurantequipment.com': <GET http://www.texrestaurantequipment.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.houston-refinance.com': <GET http://www.houston-refinance.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'johnson-bryan.com': <GET http://johnson-bryan.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'kanesforms.com': <GET http://kanesforms.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.markfoxrealestate.com': <GET http://www.markfoxrealestate.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.mphoa.org': <GET http://www.mphoa.org/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.outfitterwebsites.com': <GET http://www.outfitterwebsites.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'outdoortrailsnetwork.com': <GET http://outdoortrailsnetwork.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.psychologicalriskservices.com': <GET http://www.psychologicalriskservices.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.rcshouston.com': <GET http://www.rcshouston.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.rollingcreekcarwash.com': <GET http://www.rollingcreekcarwash.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'slammc.com': <GET http://slammc.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.texassaltwaterfishingguide.com': <GET http://www.texassaltwaterfishingguide.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.waynepigment.com': <GET http://www.waynepigment.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'bancroftfeldman.com': <GET http://bancroftfeldman.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'elilanddesign.com': <GET http://elilanddesign.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'dpharms.com': <GET http://dpharms.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'contractlandstaff.com': <GET http://contractlandstaff.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'knightsplumbing.com': <GET http://knightsplumbing.com/>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/component/virtuemart/shipping-rifles/ati-omni-5-56-poly-competition-m4-carbine-detail.ht
ml?Itemid=0> (referer: http://www.atlanticfirearms.com)
Before return request
2013-09-21 07:25:22-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/featured-not-published/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-accessories/index.php>
2013-09-21 07:25:23-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/dallas-gun-shop.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-accessories/index.php>
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:24-0500 [atlantic_firearms] DEBUG: Crawled (404) <GET http://www.atlanticfirearms.com/component/content/?Itemid=803&id=148> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/houston-texas-gun-shop.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/california-gun-shop.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Redirecting (303) to <GET http://www.atlanticfirearms.com/browse-our-products.html> from <POST http://www.atlanticfirearms.com/component/vi
rtuemart/shipping-rifles/index.php>
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/browse-our-products.html> (referer: http://www.atlanticfirearms.com/component/virtuemart
/featured-not-published/vector-arms-sp89-k-style-pistol-9mm-detail.html?Itemid=0)
Inside parse_add_to_cart
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Scraped from <200 http://www.atlanticfirearms.com/browse-our-products.html>
        {'as_of_time': datetime.datetime(2013, 9, 21, 7, 25, 18, 365559),
         'in_stock': True,
         'url': 'http://www.atlanticfirearms.com/component/virtuemart/featured-not-published/vector-arms-sp89-k-style-pistol-9mm-detail.html?Itemid=0'}
2013-09-21 07:25:25-0500 [atlantic_firearms] DEBUG: Crawled (404) <GET http://www.atlanticfirearms.com/www.atlanticfirearms.com> (referer: http://www.atlanticfirearms.com/dallas-gun-shop.html
)
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/login-or-register/editaddress.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/privacy-policy.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/subscribe.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Crawled (200) <GET http://www.atlanticfirearms.com/links.html> (referer: http://www.atlanticfirearms.com)
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.gunbroker.com': <GET http://www.gunbroker.com/user/dealernetwork.asp>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.auctionarms.com': <GET http://www.auctionarms.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.gunsamerica.com': <GET http://www.gunsamerica.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.ar15.com': <GET http://www.ar15.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.olyarms.com': <GET http://www.olyarms.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.cheaperthandirt.com': <GET http://www.cheaperthandirt.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.ammoman.com': <GET http://www.ammoman.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.ak47.net': <GET http://www.ak47.net/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.atf.treas.gov': <GET http://www.atf.treas.gov/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'caag.state.ca.us': <GET http://caag.state.ca.us/firearms/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.nra.org': <GET http://www.nra.org/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.masterpiecearms.com': <GET http://www.masterpiecearms.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'atlantic1.readyhosting.com': <GET http://atlantic1.readyhosting.com/programming/listview.asp?CatId=2>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.vulcanarmament.com': <GET http://www.vulcanarmament.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.bushmaster.com': <GET http://www.bushmaster.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.rockriverarms.com': <GET http://www.rockriverarms.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'dpmsinc.com': <GET http://dpmsinc.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.colt.com': <GET http://www.colt.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.armalite.com': <GET http://www.armalite.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.redstick-firearms.com': <GET http://www.redstick-firearms.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.vectorarms.com': <GET http://www.vectorarms.com/indexframe.html>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.arsenalinc.com': <GET http://www.arsenalinc.com/about.htm>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.ak47.com': <GET http://www.ak47.com/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.jldenter.com': <GET http://www.jldenter.com/store/>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.springfield-armory.com': <GET http://www.springfield-armory.com/index.shtml>
2013-09-21 07:25:26-0500 [atlantic_firearms] DEBUG: Filtered offsite request to 'www.dsarms.com': <GET http://www.dsarms.com/>
^C2013-09-21 07:25:26-0500 [scrapy] INFO: Received SIGINT, shutting down gracefully. Send again to force 
^C2013-09-21 07:25:26-0500 [scrapy] INFO: Received SIGINT twice, forcing unclean shutdown

发布我之前的评论作为答案。

由于您所有的 POST 请求（来自FormRequest.from_response()) 被重定向到http://www.atlanticfirearms.com/browse-our-products.html，你应该设置dont_filter=True:

    if add_to_cart:
        # attempt to add to cart to verify availability
        request = FormRequest.from_response(response, formname="addtocartForm",
                      callback=self.parse_add_to_cart, dont_filter=True)

See 根据请求提供 Scrapy 文档 http://doc.scrapy.org/en/latest/topics/request-response.html#request-objects:

不过滤(boolean) – 指示调度程序不应过滤此请求。当您想要多次执行相同的请求以忽略重复项过滤器时，可以使用此选项。

此外，您可能想要设置CONCURRENT_REQUESTS = 1在购物车中逐一添加商品（我想知道服务器如何处理并行购物车添加。）

本文内容由网友自发贡献，版权归原作者所有，本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容，请联系:hwhale#tublm.com(使用前将#替换为@)

python

Scrapy

为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？的相关文章

如何在Python中选择要写入(.csv)的列

import csv f csv reader open lmt csv r open input file for reading Date Open Hihh mLow Close Volume zip f s plit it into
使用 OpenCV 和/或 Numpy 对两个图像进行 Alpha 混合 [重复]

这个问题在这里已经有答案了我想将一个填充纯色的半透明矩形添加到已加载的半透明 PNG 中这是我正在使用的输入图像示例该图像加载了标准cv2 IMREAD UNCHANGED标志以便完美保留 alpha 通道该输入图像存储在imag
Python 子进程（ffmpeg）仅在我按 Ctrl-C 程序时启动？

我正在尝试使用 Cygwin 和 Python 2 7 并行运行一些 ffmpeg 命令这大概是我所拥有的 import subprocess processes set commands ffmpeg i input mp4 outpu
pandas Wide_to_long 后缀参数

我对在 pandas 中使用 Wide to long 时的参数有疑问有一个参数叫suffix我不明白在文档中它说后缀 str 默认 d 捕获所需后缀的正则表达式 d 捕获数字后缀没有数字的后缀可以用否定字符类 D 指定您还可以进
Python GTK + webkit - 在 gtk.main() 之后插入 JavaScript

我在终端中尝试了这个一切正常但是如果我在脚本内运行这个我无法在 gtk main 之后插入 JavaScript import gtk import webkit w gtk Window b webkit WebView w add
正则表达式，选择最接近的匹配

假设以下单词序列 BLA text text text text text text BLA text text text text LOOK text text text BLA text text BLA 我想做的是将 BLA 中的文本
组和平均 NumPy 矩阵

假设我有一个任意的 numpy 矩阵如下所示 arr 6 0 12 0 1 0 7 0 9 0 1 0 8 0 7 0 1 0 4 0 3 0 2 0 6 0 1 0 2 0 2 0 5 0 2 0 9 0 4 0 3 0 2 0 1 0
类型错误：float() 参数必须是字符串或数字，而不是“列表”python

我的 Python 有问题这是我的代码 def calcola a input b float a 0 split c float a 0 split d float a 0 split e float a 0 split j float
如何在Python代码中查找列号

简短问题当按上述方式调用函数时我可以找到行号here https stackoverflow com questions 3056048 filename and line number of python script 同样如何找到
张量流和线程

下面是来自 Tensorflow 网站的简单 mnist 教程即单层 softmax 我尝试通过多线程训练步骤对其进行扩展 from tensorflow examples tutorials mnist import input dat
scikit-learn 和tensorflow 有什么区别？可以一起使用它们吗？

对于这个问题我无法得到满意的答案据我了解 TensorFlow是一个数值计算库经常用于深度学习应用而Scikit learn是一个通用机器学习框架但它们之间的确切区别是什么 TensorFlow 的目的和功能是什么我可以一起使用它
Tensorflow 不分配完整的 GPU 内存

Tensorflow 默认分配所有 GPU 内存但我的新设置实际上只有 9588 MiB 11264 MiB 我预计大约 11 000MiB 就像我的旧设置一样张量流信息在这里 from tensorflow python client
按多个键分组并对字典列表的值进行汇总/平均值

在Python中按多个键进行分组并对字典列表进行汇总平均值的最Pythonic方法是什么假设我有一个字典列表如下所示 input dept 001 sku foo transId uniqueId1 qty 100 dept 001
Pandas groupby apply 执行缓慢

我正在开发一个涉及大量数据的程序我正在使用 python pandas 模块来查找数据中的错误这通常工作得非常快然而我当前编写的这段代码似乎比应有的速度慢得多我正在寻找一种方法来加快速度为了让你们正确测试它我上传了一段相当大的
如何使用 Celery 多工作人员启用自动缩放？

命令celery worker A proj autoscale 10 1 loglevel info启动具有自动缩放功能的工作人员当创建多个工人时 me mypc projects x celery multi start mywork
使用 WSGI 在 Windows XAMPP 中设置 Python 路径

我正在 Webfaction 上设置实时服务器的开发版本在本地计算机上的虚拟 Apache 服务器环境运行没有任何错误中运行 Django 应用程序 XP 使用 Python 2 6 运行 XAMPP Lite 我可以提交更改通过 G
将列表中的 None 替换为最左边的非 none 值

Given a None 1 2 3 None 4 None None I d like a None 1 2 3 3 4 4 4 目前我已经用以下方法强制它 def replaceNoneWithLeftmost val last Non
用于多个窗口的 Tkinter 示例代码，为什么按钮无法正确加载？

我正在编写一个程序应该按一下按钮即可打开一个窗口按另一个按钮关闭新打开的窗口我使用类以便稍后可以将代码插入到更大的程序中但是我无法正确加载按钮 import tkinter as tk class Demo1 tk Frame
python dicttoxml 多次使用相同的键

我正在尝试做如下所示的 xml
在父类中访问子类变量

我有一个父类和一个继承的子类我想知道如何访问我的父类中的子类变量我尝试了这个但失败了 class Parent object def init self print x class Child Parent x 1 x Child Er

随机推荐

无需表单即可在屏幕上绘图

是否可以在不使用表单应用程序的情况下在屏幕上创建一个大的白色矩形如果可能的话它应该覆盖整个屏幕我知道我必须使用System Drawing并尝试了几个步骤但没有一个真正在我的屏幕上打印任何内容方法一调用Windows API 你
ANDROID从url获取内容

我在使用 Android 时遇到问题我必须能够从生成内容的 URL 中读取内容然后将它们放入数组中分割读取的字符串我使用了不同的方法但无法读取该字符串我能怎么做这是我现在使用的函数继续出现字符串 nn va private
具有可比较列表与 TreeSet

选项 1 创建一个实现 Comparable 的列表并在每次添加值时使用 collections sort List l 对它进行排序选项 2 创建一个 TreeSet 它始终保持排序哪一个会更快我问这个是因为 List 为我提供了
git am 错误：“补丁不适用”

我正在尝试使用 git 将多个提交从一个项目移动到第二个类似的项目所以我创建了一个补丁包含 5 个提交 git format patch 4af51 stdout gt changes patch 然后将补丁移动到第二个项目的文件夹并想
如何在 C# 中使用 SendMessageTimeout 获取窗口标题

我需要枚举所有打开的窗口并获取它们的标题但问题是某些窗口属于同一进程但属于不同的线程该线程被阻塞等待互斥体因此我不能对属于我自己的进程的窗口使用 GetWindowText 因为这将导致 SendMessage 调用这将阻止我的
向类型类实例添加类约束

我正在尝试实现康托配对函数作为通用 Pair 类型类如下所示 module Pair Pair CantorPair where Pair interface class Pair p where pi a gt a gt p a k
如何使用 C++ 从 RAM 运行可执行文件？

如何使用 C 从 RAM 运行可执行文件可执行文件位于 RAM 中并且我知道地址如何从我的程序中调用该程序这类事情通常来自世界的黑暗角落结合像metasploit这样的工具用内存创建进程会很棒因此一些人尝试重新实现Create
向应用程序小部件发送更新广播

我有一个带有配置活动的应用程序小部件我想在单击活动中的确定按钮时触发小部件的更新我写了这段代码 Intent initialUpdateIntent new Intent AppWidgetManager ACTION APPWID
为什么 firefox/chrome 显示的页面与 IE8 不同？

当我看着我看到最新版本 with Firefox and Chrome 但是一个旧版本 with IE8 另外通过屏幕抓取PHP Curl给我一个旧版本我试过了CTRL 刷新在 IE8 中但我无法让它向我显示最新版本无论heade
警告：格式字符串不是字符串文字[重复]

这个问题在这里已经有答案了我从以下行收到格式字符串不是字符串文字警告 NSString formattedString NSString alloc initWithFormat format arguments valist 我在以
从 Delphi 7 中的 C++ DLL 接收字符串数组

我正在用 C 创建一个 DLL 它将在 Delphi 7 项目中使用这个问题与this one https codereview stackexchange com questions 43347 produce an array of
尝试从列表中删除对象时迭代期间的并发修改

我试图在多个列表中循环最后比较它们的名称如果它们不匹配则将其从列表中删除但我收到此错误迭代期间并发修改虽然我已经复制了原始列表只是为了避免这个错误但仍然得到它我尝试的是 globals filteredPollsList p
在 R 中获取生存估计

我试图获得不同人在特定时间的生存估计我的代码如下 s Surv outcome 1 outcome 2 survplot survfit s person list 1 plot survplot mark time FALSE summ
什么是serialVersionUID？为什么要使用它？

当出现以下情况时 Eclipse 会发出警告serialVersionUID不见了可序列化类 Foo 未声明静态final long 类型的serialVersionUID 字段 What is serialVersionUID为什么它很
如何优化Excel VBA点击url

VBA 运行时出现运行时错误 70 有时代码运行顺利但有时则不然想知道是否有更可靠的代码可以继续它总是停在If link innerHTML Balance Sheet Then end if Public Sub Get Dim i
如何以编程方式将 QMainWindow 大小调整为其最小大小

当我有一个带有网格布局的 QMainWindow 时当用鼠标调整它的大小时它不会低于其中所有控件正确显示所需的最小尺寸在我的应用程序中我有时会以编程方式隐藏控件但随后窗口保持相同的大小其余控件看起来分散开来它们之间的空间太大
将图像文件打开到 WriteableBitmap

问题就在这里我想从本地驱动器打开一个文件然后将其放入 WritableBitmap 中以便我可以编辑它但问题是我无法从 Uri 或类似的东西创建 WritableBitmap 我也知道如何将文件打开到 BitmapImage 中
设置EditText的HINT多行

我知道我可以更改 EditeText 文本的行数但我也可以更改 EditText 提示的行数吗我在网上找不到解决方案谢谢您的帮助 My Code Override public void onCreateOptionsMenu Men
在控制台窗口中以横向树格式漂亮打印输出

我有一本使用 Python 创建的字典 d a Adam Book 4 b Bill TV 6 Jill Sports 1 Bill Computer 5 c Bill Sports 3 d Quin Computer 3 Adam Com
为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？

我正在抓取一个网站来检查各种产品的库存状态不幸的是这需要实际单击产品页面上的添加到购物车并检查下一页的消息以确定是否有库存即它需要解析两个响应我跟着优秀的文档 http doc scrapy org en latest topi

为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？

为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？ 的相关文章

随机推荐

热门标签

为什么我的 scrapy 蜘蛛没有遵循我的项目解析函数中的请求回调？的相关文章