numpy - Running Scipy on Heroku

Question

I got Numpy and Matplotlib running on Heroku, and I'm trying to install Scipy as well. However, Scipy requires BLAS[1] to install, which is not presented on the Heroku platform. After contacting Heroku support, they suggested me to build BLAS as a static library to deploy, and setup the necessary environment variables.

So, I compiled libblas.a on a 64bit Linux box, and set the following variables as described in [2] :

$ heroku config
BLAS             => .heroku/vendor/lib/libfblas.a
LD_LIBRARY_PATH  => .heroku/vendor/lib
LIBRARY_PATH     => .heroku/vendor/lib
PATH             => bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED => true

After adding scipy==0.10.1 in my requirements.txt, the push still fails.

     File "scipy/integrate/setup.py", line 10, in configuration

       blas_opt = get_info('blas_opt',notfound_action=2)

     File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 311, in get_info

       return cl().get_info(notfound_action)

     File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 462, in get_info

       raise self.notfounderror(self.notfounderror.__doc__)

   numpy.distutils.system_info.BlasNotFoundError:

       Blas (http://www.netlib.org/blas/) libraries not found.

       Directories to search for the libraries can be specified in the

       numpy/distutils/site.cfg file (section [blas]) or by setting

       the BLAS environment variable.

It seem that pip is not aware of the BLAS environment variable, so I check the environment using heroku run python:

(venv)bash-3.2$ heroku run python
Running python attached to terminal... import up, run.1
Python 2.7.2 (default, Oct 31 2011, 16:22:04) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system('bash')
~ $ echo $BLAS
.heroku/vendor/lib/libfblas.a
~ $ ls .heroku/vendor/lib/libfblas.a
.heroku/vendor/lib/libfblas.a
~ $ 

And it seems fine. Now I have no idea how to solve this.

[1] http://www.netlib.org/blas/ [2] http://www.scipy.org/Installing_SciPy/Linux

Answer 1

I managed to get this working on the cedar stack by building numpy and scipy offline as bdists and then modifying the heroku python buildpack to unzip these onto the dyno's vendor/venv areas directly. You can also use the buildpack to set environment variables that persist.

Heroku haven't officially published buildpacks yet - search for 'heroku buildpacks' for more thirdparty/heroku ones and information.

My fork of the python build pack is here: https://wyn@github.com/wyn/heroku-buildpack-python.git

The changes are in the bin/compile where I source two new steps, a scipy/numpy step and an openopt step. The scripts for the two steps are in bin/steps/npscipy and bin/steps/openopt. I also added some variables to bin/release. Note that I am assuming installation through a setup.py file rather than the requirements.txt approach (c.f. https://devcenter.heroku.com/articles/python-pip#traditional_distributions).

I also download the blas/lapack/atlas/gfortran binaries that were used to build numpy/scipy onto the dyno as there are c extensions that need to link through to them. The reason for building everything offline and downloading is that pip-installing numpy/scipy requires you to have a fortran compiler + associated dev environment and this made my slugs too big.

It seems to work, the slug size is now 35mb and scaling seems fast too. All but one of the numpy tests pass and all of the scipy tests pass.

This is still work in progress for me but I thought I'd share.