python - How to browse a whole website using selenium?

Question

Is it possible to go through all the URIs of a given URL (website) using selenium ?

My aim is to launch firefox browser using selenium with a given URL of my choice (I know how to do it thanks to this website), and then let firefox browse all the pages that URL (website) has. I appreciate any hint/help on how to do it in Python.

Answer 1

You can use a recursive method in a class such as the one given below to do this.

public class RecursiveLinkTest {
    //list to save visited links
    static List<String> linkAlreadyVisited = new ArrayList<String>();
    WebDriver driver;

    public RecursiveLinkTest(WebDriver driver) {
        this.driver = driver;
    }

    public void linkTest() {
        // loop over all the a elements in the page
        for(WebElement link : driver.findElements(By.tagName("a")) {
            // Check if link is displayed and not previously visited
            if (link.isDisplayed() 
                        && !linkAlreadyVisited.contains(link.getText())) {
                // add link to list of links already visited
                linkAlreadyVisited.add(link.getText());
                System.out.println(link.getText());
                // click on the link. This opens a new page
                link.click();
                // call recursiveLinkTest on the new page
                new RecursiveLinkTest(driver).linkTest();
            }
        }
        driver.navigate().back();
    }

    public static void main(String[] args) throws InterruptedException {
        WebDriver driver = new FirefoxDriver();
        driver.get("http://newtours.demoaut.com/");
        // start recursive linkText
        new RecursiveLinkTest(driver).linkTest();
    }
}

Hope this helps you.