python - Why can't I get `pip install lxml` to work within a virtualenv?

Question

Note: I'm using virtualenvwrapper.

Before activating the virtual environment:

$ pip install lxml
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib/python2.7/dist-packages  
Cleaning up...  

After activating the virtual environment:

(test-env)$ pip install lxml
force/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error: 
libxml/xmlversion.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

----------------------------------------
Command /home/chaz/dev/envs/test-with-system-python-force/bin/python2
.7 -c "import setuptools;__file__='/home/chaz/dev/envs/test-with-
system-python-force/build/lxml/setup.py';exec(compile(open(__file__).
read().replace('
', '
'), __file__, 'exec'))" install --record 
/tmp/pip-bJ6Q_B-record/install-record.txt --single-version-externally
-managed --install-headers /home/chaz/dev/envs/test-env/include/site/python2.7 failed with error code 1 in 
/home/chaz/dev/envs/test-env/build/lxml
Storing complete log in /home/chaz/.pip/pip.log

Answer 1

You probably already have lxml installed on your system, perhaps installed due to a system package. Thus, the first attempt (pip install lxml without an active virtualenv) doesn't fail, but it also doesn't install it; it really doesn't do anything.

In a virtualenv, by default, the system packages are ignored. Therefore, pip thinks that lxml is not installed. Therefore, it tries to install it into your virtual environment.

lxml contains C modules that need to be compiled in order to install properly. However, the compilation of those C modules rely on your having some "development libraries" already installed as well. These development libraries are C libraries, not Python, and as such pip won't be able to automatically fetch them from the internet and install them for you.

Therefore, you will need to install these development libraries on your own, most likely using your package manager. In a Debian system (like Ubuntu), this is...

apt-get install libxml2-dev libxslt-dev

This will install the libxml2 and libxslt development libraries to your local system. If you try again to install lxml, the C module compilation step should work because now these development libraries are on your system.

The error message you were receiving was due to the fact that these libraries were missing (the libxml/xmlversion.h: No such file or directory part of the error message).

See also: How to install lxml on Ubuntu