how do web crawlers handle javascript

Question

Today a lot of content on Internet is generated using JavaScript (specifically by background AJAX calls). I was wondering how web crawlers like Google handle them. Are they aware of JavaScript? Do they have a built-in JavaScript engine? Or do they simple ignore all JavaScript generated content in the page (I guess quite unlikely). Do people use specific techniques for getting their content indexed which would otherwise be available through background AJAX requests to a normal Internet user?

Answer 1

JavaScript is handled by both Bing and Google crawlers. Yahoo uses the Bing crawler data, so it should be handled as well. I didn't look into other search engines, so if you care about them, you should look them up.

Bing published guidance in March 2014 as to how to create JavaScript-based websites that work with their crawler (mostly related to pushState) that are good practices in general:

• Avoid creating broken links with pushState
• Avoid creating two different links that link to the same content with pushState
• Avoid cloaking. (Here's an article Bing published about their cloaking detection in 2007)
• Support browsers (and crawlers) that can't handle pushState.

Google later published guidance in May 2014 as to how to create JavaScript-based websites that work with their crawler, and their recommendations are also recommended:

• Don't block the JavaScript (and CSS) in the robots.txt file.
• Make sure you can handle the load of the crawlers.
• It's a good idea to support browsers and crawlers that can't handle (or users and organizations that won't allow) JavaScript
• Tricky JavaScript that relies on arcane or specific features of the language might not work with the crawlers.
• If your JavaScript removes content from the page, it might not get indexed. around.