web scraping - Why do a request work on requests but not on scrapy

Question

I'm trying to scrape a webpage that loads the results for page 2 and so on when I scroll. So I get the url to the api (img) that it runs and it should work just fine.

But it only works if I use the requests lib. When i run requests.get() with the same url used with scrapy I get response 200, but with scrapy it returns 500 status. I don't know why this doesn't work with scrapy, any explanations for that?

Here's what I'm trying to do

Obrigado.

import scrapy
import json
import re

class ScrapeVagas(scrapy.Spider):
    name = "vagas"
    base_url = "https://www.trabalhabrasil.com.br/api/v1.0/Job/List?idFuncao=0&idCidade=5345&pagina=%d&pesquisa=&ordenacao=1&idUsuario="
    start_urls = [base_url % 100]
    download_delay = 1

    def parse(self, response):
        vagas = json.loads(response.text)
        
        for vaga in range(0, len(vagas)):
            yield {
                "vaga": vagas[vaga]["df"],
                "salario": re.sub("[R$.]", "", vagas[vaga]["sl"]).strip()
            }

question from:https://stackoverflow.com/questions/65829682/why-do-a-request-work-on-requests-but-not-on-scrapy

Answer 1

Your are getting 500 Internal Server Error server error response code indicates that the server encountered an unexpected condition that prevented it from fulfilling the request. here need Request header to get the proper response. See the output in scrapy shell.

import scrapy
base_url = "https://www.trabalhabrasil.com.br/api/v1.0/Job/List?idFuncao=0&idCidade=5345&pagina=%d&pesquisa=&o
rdenacao=1&idUsuario="
start_urls = [base_url % 100]
start_urls
url = start_urls[0]
headers = {"USER-AGENT":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.3",
                 "referer": "https://www.trabalhabrasil.com.br/vagas-empregos-em-sao-paulo-sp",
                  "authority": "www.trabalhabrasil.com.br",
                 "path": "/api/v1.0/Job/List?idFuncao=100&idCidade=5345&pagina=65&pesquisa=&ordenacao=1&idUsuario=",
       
                "scheme": "https",
                 "accept": "*/*",
               "accept-language": "en-US,en;q=0.9,bn;q=0.8",
      
               "dnt": "1",
                 "sec-fetch-dest": "empty",
                "sec-fetch-mode": "cors",
                   "sec-fetch-site": "same-origin",
      
                }
     
r = scrapy.Request(url, headers=headers)
fetch(r)
2021-01-22 00:30:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.trabalhabrasil.com.br/api/v1.0/Job/List?idFuncao=0&idCidade=5345&pagina=100&pesquisa=&ordenacao=1&idUsuario=> (referer: https://www.trabalhabrasil.com.br/vagas-empregos-em-sao-paulo-sp)
    
    
In [19]: response.status
Out[19]: 200