javascript - Using Google Apps Script to scrape Dynamic Web Pages

Question

I would like to read some data from other websites for a project using Google Script. The pages in questions are Dyanmic; they contain content that is loaded after the initial page load, via JavaScript calls to the server. Usually, with somewhat static content, this works fine but I am new to JavaScript and to Google Apps Script and thus do not know how to get the content if it is loaded asynchronously via JavaScript (e.g. via AJAX).

An example can be found here showing the last tracks played at a radio station. However, these tracks are loaded using JavaScript and instead of the table containing the Strings I get

<td class="row2"><span id="track_2">&nbsp;</span></td>

When I use:

UrlFetchApp.fetch(url).getContentText();

If I save the HTML in my browser, though, the right data Strings are there:

<td class="row2" id="track_2">15:12 Will Smith - Men In Black</td>
                     ^^^^^^^  ^^^^^ ^^^^^^^^^^   ^^^^^^^^^^^^

Is there any way to do this with Google Apps Script?

Answer 1

Not generally, no. If you can reverse engineer what it's doing, you might be able to do the same JavaScript calls, but the odds are against it if it requires any server coordination. In theory one could run a JavaScript browser implementation inside of Google Apps Script (like env-js) which could do this, but in practice I think it would be very difficult if not impossible to make it work.