看起来名称解析最终是由socket.create_connection
.
-> urllib2.urlopen
-> httplib.HTTPConnection
-> socket.create_connection
不过,一旦设置了“Host:”标头,您就可以解析主机并将 IP 地址传递给 opener。
我建议你子类化httplib.HTTPConnection
,并包裹connect
修改方法self.host
在将其传递给之前socket.create_connection
.
然后子类化HTTPHandler
(and HTTPSHandler
)来替换http_open
方法与通过你的HTTPConnection
而不是 httplib 自己的do_open
.
像这样:
import urllib2
import httplib
import socket
def MyResolver(host):
if host == 'news.bbc.co.uk':
return '66.102.9.104' # Google IP
else:
return host
class MyHTTPConnection(httplib.HTTPConnection):
def connect(self):
self.sock = socket.create_connection((MyResolver(self.host),self.port),self.timeout)
class MyHTTPSConnection(httplib.HTTPSConnection):
def connect(self):
sock = socket.create_connection((MyResolver(self.host), self.port), self.timeout)
self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
class MyHTTPHandler(urllib2.HTTPHandler):
def http_open(self,req):
return self.do_open(MyHTTPConnection,req)
class MyHTTPSHandler(urllib2.HTTPSHandler):
def https_open(self,req):
return self.do_open(MyHTTPSConnection,req)
opener = urllib2.build_opener(MyHTTPHandler,MyHTTPSHandler)
urllib2.install_opener(opener)
f = urllib2.urlopen('http://news.bbc.co.uk')
data = f.read()
from lxml import etree
doc = etree.HTML(data)
>>> print doc.xpath('//title/text()')
['Google']
显然,如果您使用 HTTPS,则会出现证书问题,并且您需要填写 MyResolver...