python - "SSL certificate verify failed" using pip to install packages

Question

I am trying to install the Scrapy package (among others) for python using pip. I have tried doing the installation using python 3 and python 2, I have installed/upgraded the setuptools like so: $ pip3 install --upgrade setuptools, I have tried to use the --trusted-host option like so: $ pip3 install --trusted-host pypi.python.org Scrapy. But I always get the same error message when I run $ pip3 install Scrapy. The complete output is this:

Collecting Scrapy
  Using cached Scrapy-1.3.2-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from Scrapy)
  Using cached PyDispatcher-2.0.5.tar.gz
Collecting service-identity (from Scrapy)
  Using cached service_identity-16.0.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from Scrapy)
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl
Collecting w3lib>=1.15.0 (from Scrapy)
  Using cached w3lib-1.17.0-py2.py3-none-any.whl
Collecting parsel>=1.1 (from Scrapy)
  Using cached parsel-1.1.0-py2.py3-none-any.whl
Collecting queuelib (from Scrapy)
  Using cached queuelib-1.4.2-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from Scrapy)
Collecting Twisted>=13.1.0 (from Scrapy)
  Using cached Twisted-17.1.0.tar.bz2
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/incremental/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
    Couldn't find index page for 'incremental' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
    No local packages or working download links found for incremental>=16.10.1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/gy/5xt04_452z791v1qjs1yzxkh0000gn/T/pip-build-nkv4jozy/Twisted/setup.py", line 21, in <module>
        setuptools.setup(**_setup["getSetupArgs"]())
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 317, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 372, in fetch_build_eggs
        replace_conflicting=True,
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 851, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1123, in best_match
        return self.obtain(req, installer)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1135, in obtain
        return installer(requirement)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 440, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 668, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1')

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/gy/5xt04_452z791v1qjs1yzxkh0000gn/T/pip-build-nkv4jozy/Twisted/

I am on a mac OS version 10.12.1 and am using python 3.6. Does anybody know a solution to this problem?

Answer 1

As stated here https://bugs.python.org/issue28150 in previous versions of python Apple supplied the OpenSSL packages but does not anymore.

Running the command pip install certifi and then pip install Scrapy fixed it for me