当前对版权保护日益严格,因此在此说明爬取的图像仅做研究和个人使用,禁止用作商用目的。
该爬虫方法可应用到其它允许爬虫的网址
查看网页源代码寻找高清图像链接
以关键词明星为例,搜索后的网址为 https://www.vcg.com/creative/search?phrase=明星
在chrome下ctrl+u或者右键查看源代码,搜索.jpp会找到很多图片的网址,经查验,最高清的图像链接格式为:“url800”:“图像网址”,即在源代码的最后,见下图.
程序示例
Step 1 导入必要Python包
from urllib import request
import os
import re
Step 2 定义保存路径函数
def get_path(classname,subclassname,filename):
cwd = os.getcwd()
dir_path = cwd+'/vcg_test/' + classname +'/'+ subclassname
if os.path.exists(dir_path):
pass
else:
os.makedirs(dir_path)
file_path = dir_path +'/'+ filename
return file_path
Step 3 按照关键词爬取图片
假设我们希望爬取“明星”和“动漫人物”两种关键词下的图片,我们希望爬取的图片中仅有一个人方便使用,则代码如下:
classnames = ['super_star','cartoon']
keypoints = ['%E6%98%8E%E6%98%9F','%E5%8A%A8%E6%BC%AB%E4%BA%BA%E7%89%A9']
gender_file_path = ['male','female']
all_page = 35
for class_index, phrase in enumerate(classnames):
sum_all_num = 0
if class_index >= 0 :
for gender_index,gender in enumerate(gender_file_path):
if gender_index >= 0 :
for page in range(1,all_page+1):
num_in_page = 1
url = 'https://www.vcg.com/creative/search?phrase=' + keypoints[class_index] +'&creativePeopleNum=2&creativeGender=' +str(gender_index+1)+'&page='+str(page)
header = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 Safari/537.36',
}
req = request.Request(url=url,headers=header)
openhtml = request.urlopen(req).read().decode('utf8')
com = re.compile('"url800":.*?/creative/.*?.jpg"')
urladds = com.findall(openhtml)
for urladd in urladds:
try:
add = 'http:'+urladd.strip('"url800":')
filename = classnames[class_index]+'_'+gender_file_path[gender_index]+'_vcg_raw_page'+str(page)+'_'+str(num_in_page)+'.jpg'
path = get_path(classnames[class_index],gender_file_path[gender_index],filename)
print('当前下载...',filename)
dom = request.urlopen(add).read()
with open(path,'wb') as f:
f.write(dom)
sum_all_num += 1
num_in_page += 1
except:
print('当前该任务总共总共下载:',sum_all_num)
if sum_all_num % 50 == 0:
print('当前该任务总共下载:',sum_all_num)
Step 4 查看爬取图像的质量
单张图片约100kb左右,尺寸约800*533.
由于未获取图片版权,因此就不展示爬取的图片了.
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)