在单词后获取文本——R Webscraping

几周前，这里有人帮助我极大地获得了名人数据库中所有链接的列表。我能够运行此代码并获得以下输出

library(purrr)
library(rvest)
url_base <- "https://www.nndb.com/lists/494/000063305/"    
## Gets A-Z links
all_surname_urls <- read_html(url_base) %>%
      html_nodes(".newslink") %>%
      html_attrs() %>%
      map(pluck(1, 1))

all_ppl_urls <- map(
      all_surname_urls, 
      function(x) read_html(x) %>%
        html_nodes("a") %>%
        html_attrs() %>%
        map(pluck(1, 1))
    ) %>% 
      unlist()

all_ppl_urls <- setdiff(
      all_ppl_urls[!duplicated(all_ppl_urls)], 
      c(all_surname_urls, "http://www.nndb.com/")
    )

all_ppl_urls[1] %>%
      read_html() %>%
      html_nodes("p") %>%
      html_text()

# [1] "AKA Lee William Aaker"
# [2] "Born: 25-Sep-1943Birthplace: Los Angeles, CA"
# [3] "Gender: MaleRace or Ethnicity: WhiteOccupation: Actor"
# [4] "Nationality: United StatesExecutive summary: The Adventures of Rin Tin Tin"
# ...

我的初衷是获得一个数据框，在其中我可以得到name的人，他们的gender, race, 职业 and 国籍到单个数据帧中。

如果您的数据位于 html 表中，而著名姓名数据库的情况并非如此，那么我在这里和其他网站上看到的很多问题都会很有帮助。我知道所有 40K 站点都需要涉及一个循环，但经过一个周末的搜索答案后，我似乎无法找出如何实现。有人可以帮忙吗？

编辑添加我尝试遵循这里的一些规则但是有点复杂

## I tried to run list <- all_ppl_urls%>% map(read_html) but that was taking a LONG time so I decided to just get the first ten links for the sake of showing my example:

example <- head(all_ppl_urls, 10)

list  <- example %>% map(read_html)

test <-list  %>% map_df(~{
   text_1 <- html_nodes(.x, 'p , b') %>% html_text

我收到这个错误： 错误： 另外：警告消息： 关闭未使用的连接 3 (http://www.nndb.com/people/965/000279128/ http://www.nndb.com/people/965/000279128/)

在这里，您可以通过查看每个 html 文件来获取数据。这只是一种获得一些好的结果的方法...但是...您必须注意到应该编辑那些 gsub 函数以获得更好的结果。发生这种情况是因为该 url 列表，或者说，该网页，在数据显示方式上并不均匀。这是你必须处理的事情。例如，这里只是两个屏幕截图，您可以在其中找到网络演示中的差异：

无论如何，您可以通过调整此代码来管理此操作：

library(purrr)
library(rvest)

[...] #here is your data

all_ppl_urls[100] %>%
    read_html() %>%
    html_nodes("p") %>%
    html_text()
# [3] "Gender: MaleReligion: Eastern OrthodoxRace or Ethnicity: Middle EasternSexual orientation: StraightOccupation: PoliticianParty Affiliation: Republican"  

#-----------------------------------------------------------------------------------------------
# NEW WAY
toString(read_html(all_ppl_urls[100])) #get example of how html looks...
#><b>AKA</b> Edmund Spencer Abraham</p>\n<p><b>Born:</b> <a href=\"/lists/681/000106363/\" class=\"proflink\">12-Jun</a>-<a href=\"/lists/951/000105636/\" class=\"proflink\">1952</a><br><b>Birthplace:</b> <a href=\"/geo/604/000080364/\" class=\"proflink\">East Lansing, MI</a><br></p>\n<p><b>Gender:</b> Male<br><b>

#1. remove NA urls (avoid problems later on)
urls <- all_ppl_urls[!is.na(all_ppl_urls)]
length(all_ppl_urls)
length(urls)

#function that creates a list with your data
GetLife <- function (htmlurl) {
    htmltext <- toString(read_html(htmlurl))
    name <- gsub('^.*AKA</b>\\s*|\\s*</p>\n.*$', '', htmltext)
    gender <- gsub('^.*Gender:</b>\\s*|\\s*<br>.*$', '', htmltext)
    race <- gsub('^.*Race or Ethnicity:</b>\\s*|\\s*<br>.*$', '', htmltext)
    occupation <- gsub('^.*Occupation:</b>\\s*|\\s*<br>.*$|\\s*</a>.*$|\\s*</p>.*$', '', htmltext)
    #as occupation seems to have to many hyperlinks we are making another step
    occupation <- gsub("<[^>]+>", "",occupation)
    nationality <- gsub('^.*Nationality:</b>\\s*|\\s*<br>.*$', '', htmltext)
    res <- c(ifelse(nchar(name)>100, NA, name), #function that cleans weird results >100 chars
             ifelse(nchar(gender)>100, NA, gender),
             ifelse(nchar(race)>100, NA, race),
             ifelse(nchar(occupation)>100, NA, occupation),
             ifelse(nchar(nationality)>100, NA, nationality),
             htmlurl)
    return(res)
}

emptydf <- data.frame(matrix(ncol=6, nrow=0)) #creaty empty data frame
colnames(emptydf) <- c("name","gender","race","occupation","nationality","url") #set names in empty data frame
urls <- urls[2020:2030] #sample some of the urls
for (i in 1:length(urls)){
    emptydf[i,] <- GetLife(urls[i])
}
emptydf

以下是分析这 10 个 url 的示例：

name gender     race occupation   nationality                                       url
1                        <NA>   Male    White   Business United States http://www.nndb.com/people/214/000128827/
2  Mark Alexander Ballas, Jr.   Male    White     Dancer United States http://www.nndb.com/people/162/000346121/
3       Thomas Cass Ballenger   Male    White Politician United States http://www.nndb.com/people/354/000032258/
4  Severiano Ballesteros Sota   Male Hispanic       Golf         Spain http://www.nndb.com/people/778/000116430/
5  Richard Achilles Ballinger   Male    White Government United States http://www.nndb.com/people/511/000168007/
6      Steven Anthony Ballmer   Male    White   Business United States http://www.nndb.com/people/644/000022578/
7        Edward Michael Balls   Male    White Politician       England http://www.nndb.com/people/846/000141423/
8                        <NA>   Male    White      Judge United States http://www.nndb.com/people/533/000168029/
9                        <NA>   Male    Asian   Engineer       England http://www.nndb.com/people/100/000123728/
10         Michael A. Balmuth   Male    White   Business United States http://www.nndb.com/people/635/000175110/
11        Aristotle N. Balogh   Male    White   Business United States http://www.nndb.com/people/311/000172792/