web scraping - How to generate the start_urls dynamically in crawling?

Question

I am crawling a site which may contain a lot of start_urls, like:

http://www.a.com/list_1_2_3.htm

I want to populate start_urls like [list_d+_d+_d+.htm], and extract items from URLs like [node_d+.htm] during crawling.

Can I use CrawlSpider to realize this function? And how can I generate the start_urls dynamically in crawling?

Answer 1

The best way to generate URLs dynamically is to override the start_requests method of the spider:

from scrapy.http.request import Request

def start_requests(self):
      with open('urls.txt', 'rb') as urls:
          for url in urls:
              yield Request(url, self.parse)