What to do if rvest isn't recognizing a node in R?

Question

I am trying to scrape referee game data using rvest. See the code below:

page_ref<-read_html("https://www.pro-football-reference.com/officials/HittMa0r.htm")

ref_tab <- page_ref %>%
    html_node("#games") %>%
    html_text()
    #html_table()

But rvest does not recognize any of the nodes for the "Games" table in the link. It can pull the data from the first table "Season Totals" just fine. So, am I missing something? In general, what does it mean if rvest doesn't recognize a node identified with SelectorGadget and is clearly identified in the developer tools?

Answer 1

It is because the first table is in the html you get from the server and the other tables are filled by JavaScript. Rvest can only get you the things that are there in the html response from server. If you want to get the data filled by JS, you need to use some other tool like Selenium or Puppeteer, for example.

• Selenium
• Puppeteer / gepetto