抓取 Finviz 页面以获取表中的特定值

首先我要说的是，我不支持抓取服务条款不允许的网站，这纯粹是为了从各个网站假设收集财务数据的学术研究。

如果有人想看这个链接：

...存储在 URLs.csv 文件中，想要抓取第 2-5 列（即 Ticker、Perf Week、Perf Month、Perf Quarter）并希望将其导出到 CSV 文件，代码可能是什么样子喜欢？

尝试使用我过去的问题中另一个用户的答案，到目前为止我得到的内容如下：

from bs4 import BeautifulSoup
import requests
import csv, random, time


# Open 'URLs.csv' to read list of URLs in the list
with open('URLs.csv', newline='') as f_urls, open('Results.csv', 'w', newline='') as f_output:
csv_urls = csv.reader(f_urls)
csv_output = csv.writer(f_output, delimiter=',') 

headers = requests.utils.default_headers() 
headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'

csv_output.writerow(['Ticker', 'Perf Week', 'Perf Month', 'Perf Quarter'])


# Start to read the first URL in the .csv and loop for each URL/row in the .csv
for line in csv_urls:


# Start at first url and look for items
    page = requests.get(line[0])
    soup = BeautifulSoup(page.text, 'html.parser')

    symbol = soup.findAll('a', {'class':'screener-link-primary'})

    perfdata = soup.findAll('a', {'class':'screener-link'})


    lines = list(zip(perfdata, symbol))

    # pair up every two teams
    for perfdata1, symbol1 in zip(lines[1::2], lines[::2]):

        # extract string items
        a1, a2, a3, _ = (x.text for x in symbol1 + perfdata1)

        # reorder and write row
        row = a1, a2, a3
        print(row)
        csv_output.writerow(row)

...我得到以下输出：

('1', 'A', '7.52%')
('-0.94%', 'AABA', '5.56%')
('10.92%', 'AAL', '-0.58%')
('4.33%', 'AAOI', '2.32%')
('2.96%', 'AAP', '1.80')
('2.83M', 'AAT', '0.43')
('70.38', 'AAXN', '0.69%')
...

因此它会跳过一些行并且不按正确的顺序返回数据。我希望在我的最终输出中看到：

('A', '7.52%', -0.94%, 5.56%)
('AA', '0.74%', 0.42%, -20.83%)
('AABA', '7.08%', '0.50%', '7.65%')
('AAC', '31.18%', '-10.95%', '-65.14%')
...

我知道代码的最后部分不正确，但正在寻求一些指导。谢谢！

问题是你只提取列Ticker和随机单元（.screener-link)，改为提取行。

for line in csv_urls:
    # Start at first url and look for items
    page = requests.get(line[0])
    soup = BeautifulSoup(page.text, 'html.parser')
    rows = soup.select('table[bgcolor="#d3d3d3"] tr')
    for row in rows[1:]:
        # extract string items
        a1, a2, a3, a4 = (x.text for x in row.find_all('td')[1:5])
        row = a1, a2, a3, a4
        print(row)
        # write row
        csv_output.writerow(row)

output

('A', '7.52%', '-0.94%', '5.56%')
('AA', '0.74%', '0.42%', '-20.83%')
('AABA', '7.08%', '0.50%', '7.65%')
('AAC', '31.18%', '-10.95%', '-65.14%')
('AAL', '-0.75%', '-6.74%', '0.60%')
('AAN', '5.68%', '6.51%', '-6.55%')
('AAOI', '5.47%', '-17.10%', '-23.12%')
('AAON', '0.62%', '1.10%', '8.58%')
('AAP', '0.38%', '-3.85%', '-2.30%')
('AAPL', '2.72%', '-9.69%', '-29.61%')
('AAT', '3.26%', '-2.39%', '10.74%')
('AAWW', '15.87%', '1.55%', '-9.62%')
('AAXN', '7.48%', '11.85%', '-14.24%')
('AB', '1.32%', '6.67%', '-2.73%')
('ABBV', '-0.85%', '0.16%', '-5.12%')
('ABC', '3.15%', '-7.18%', '-15.72%')
('ABCB', '5.23%', '-3.31%', '-22.35%')
('ABEO', '1.71%', '-10.41%', '-28.81%')
('ABG', '1.71%', '8.95%', '12.70%')
('ABM', '7.09%', '26.92%', '5.90%')