python - scrapy spider sends spider_close signal before it closes

Question

I have a spider that takes a file as a parameter, this file contains the xpaths.

The spider parses the file and gets the xpaths and starts crawling.

Everything is working fine

Now, I want that spider to run many times, So I did this:

script.py

def setup_crawler(file):
    spider = MySpider(attributesXMLFilePath=file)
    settings = get_project_settings()
    crawler = Crawler(settings)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()


for oneFile in myFiles:
    setup_crawler(oneFile')
log.start()
reactor.run()

and in MySpider I do this:

def __init__(self, attributesXMLFilePath):
    dispatcher.connect(self.spider_closed, signals.spider_closed)
def spider_closed(self, spider):
        log.msg('The number of pages in the spider {1} are {0}'.format(self.numbers, self.attributesXMLFilePath))
        log.msg('The number of details pages in the spider {1} are {0}'.format(self.numbers2, self.attributesXMLFilePath))
        log.msg('The spider {0} with xml {2} finished working on {1}'.format(self.name, datetime.now(), self.attributesXMLFilePath), level=log.INFO)

but in the log file, I see this:

2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file1.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file1.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file1.xml finished working on 2014-06-08 18:18:03.746000
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of pages in the spider file2.xml are 1
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The number of details pages in the spider file2.xml are 0
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000
2014-06-08 18:18:03+0300 [scrapy] INFO: The spider MySpider with xml file2.xml finished working on 2014-06-08 18:18:03.748000

As you see:

1. every line is duplicated twice, why?
2. there are many times that the spider_close function is executed

Note

I have that log data in my log file a lot and I just showed you a sample to explain my problem

Note2

Ofc I am not using `MySpider`, `file1.xml`, and `file2.xml` names, but I couldn't show you the real name for privacy issues.

Answer 1

Waitting for answers