python - Proper way to parse requirements file after pip upgrade to pip 10.x.x?

Question

So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.req.

Since it is common practice to use the parse_requirements function in your setup.py to install all the dependencies out of a requirements file I now wonder if this practice should change since it is now lying under _internal?

Or what is actually best practice without using parse_requirements?

Answer 1

First, I believe parsing requirements.txt from within setup.py is not a good idea. It should be the other way around, install_requires in setup.py or setup.cfg should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. But everyone has different needs, that lead to different workflows.

So with that said...

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python3

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

Again, this will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not handled.

---

Notes:

• See also this other answer: https://stackoverflow.com/a/59971469/11138259