python - jupyter notebook running kernel in different env

Question

I've gotten myself into some kind of horrible virtualenv mess. Help?!

I manage environments with conda. Until recently, I only had a python2 jupyter notebook kernel, but I decided to drag myself kicking and screaming into the 21st century and installed a python3 kernel; I forget how I did it.

My main (anaconda) python defaults to 2.7.

So here I am, merrily trying to use beautiful soup from inside my shiny new python3 kernel, and I don't seem to be able to do anything to get at whatever environment it's finding packages in. Viz (all from notebook):

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

Ok, fine, I'll install it using shell magic. Right? Right?

! pip install bs4

--> Collecting bs4
  Downloading bs4-0.0.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4 in /Users/[MY-USER]/anaconda/lib/python2.7/site-packages (from bs4)
[...]
Successfully built bs4
Installing collected packages: bs4
Successfully installed bs4-0.0.1

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

Oh no. Does it think I'm in a 2.7 env even though I'm running a python3 kernel? That won't do.

! conda info --envs
--> # conda environments:
#
flaskenv                 /Users/[MY-USER]/anaconda/envs/flaskenv
mesa                     /Users/[MY-USER]/anaconda/envs/mesa
py35                     /Users/[MY-USER]/anaconda/envs/py35
root                  *  /Users/[MY-USER]/anaconda

Ok, I can fix that. One of those is a 3.5 env.

! source activate py35
--> prepending /Users/[MY-USER]/anaconda/envs/py35/bin to PATH

! conda install beautifulsoup4
--> Fetching package metadata .......
Solving package specifications: ..........

# All requested packages already installed.
# packages in environment at /Users/[MY-USER]/anaconda:
#
beautifulsoup4            4.4.1                    py27_0  

concerning...

! pip install bs4
--> Requirement already satisfied (use --upgrade to upgrade): bs4 in /Users/[MY-USER]/anaconda/lib/python2.7/site-packages

more concerning...

from bs4 import BeautifulSoup 

-> ImportError: No module named 'bs4'

ARRGH!!! headdesk Am I going to have to kill the kernel in order to fix this (and re-run a bit of work)? Is killing the kernel even going to work? How do I get my jupyter kernel to know what environment it's supposed to be running under?

thanks!

Answer 1

This is a tricky part of ipython / Jupyter. The set of kernels available are independent of what your virtualenv is when you start jupyter Notebook. The trick is setting up the the ipykernel package in the environment you want to identify itself uniquely to jupyter. From docs on multiple ipykernels,

source activate ENVNAME
pip install ipykernel
python -m ipykernel install --user --name ENVNAME --display-name "Python (whatever you want to call it)"

If you only want to have a single Python 3 kernel, from the conda environment, just use python -m ipykernel install --user and it will reset the default python to the one in the virtualenv.

And yes, you will need to restart the kernel and re-run the prior steps.

See Also Using both Python 2.x and Python 3.x in IPython Notebook