web scraping - Is it possible to extract data from an interactive JS graph using python?

Question

I want to know if it is possible to extract from an interactive JS graph like the one here: https://www.eurocontrol.int/Economics/DailyTrafficVariation-States.html

A problem here is that in order to get the data I need I have to first select a gray bar on the first table in order to generate the needed second table, like shown:

Daily Air Traffic first table generates second table with needed percent data

Also only want to extract the percentages that display for four countries.

I tried a few python packages but they were not that effective for interactive JS graphs. Most seem good for only static tables like those found on wikipedia. BeautifulSoup, Pandas, Requests, and Selenium I tried and inspected the webpage to see its xhr data and tried to find if there was a csv file attached. None captured interactive JS graphs in order to extract its data.

Is it possible? and could I download it to an excel?

Thanks!

question from:https://stackoverflow.com/questions/65908164/is-it-possible-to-extract-data-from-an-interactive-js-graph-using-python

Answer 1

You must use a tool that is able to render and execute javascript. That basically means a web browser. There are several ones available, some based on Firefox some based on Chrome.

Given the links you provided I think Puppeteer from Google (https://pptr.dev/) will allow you to do what you need.

But it seems to me you are underestimating the complexity of scraping a website. Be prepared to overcome many difficulties, the most important one being the site you are scraping not liking having it's data scraped and taking drastic countermeasures. This approach will probably work to scrape a few pages but hardly a lot of data.