python - Getting PIL/Pillow 4.2.1 to upload properly to AWS Lambda Py3.6

Question

Background

I have been struggling for the past few days to deploy a Lambda that uses Pillow, and I am deploying using Python 3.6. It may be noteworthy also that I am developing this on a Windows 10 environment.

First Attempts

I began by having pip install my packages strictly in my workspace by doing the following:

pip3 install pillow -t "D:Work and Projects......<projectdir>pillow"

I have other packages, and tried installing the packages in the same manor, one of them specifically was praw and I did so by:

pip3 install praw -t "D:Work and Projects......<projectdir>praw"

After zipping the contents of my project together, I uploaded my package up to Lambda and upon my first test I received the error:

Unable to import module 'motw_lambda': cannot import name '_imaging'

I then removed the Pillow package in an attempt to see where this issue was stemming from (Pillow or praw or one of the other packages). With Pillow removed, the execution succeeded. I then removed the pillow package in my package and tried:

pip3 install pillow -t "D:Work and Projects......<projectdir>PIL"

and

pip3 install pillow -t "D:Work and Projects......<projectdir>Pillow"

But got the same error with the package '_imaging'.

Further Attempts

I then followed the directions per this resource and this. I also tried using virualenv and even lambda-uploader.

Strange enough, I get the same error! I am all out of options here, and feel either I am doing something dumb, or that this is not possible on Lambda-Python3.6 currently (Although I can't image someone else hasn't used pillow in a py3.6-lambda yet...)

Any info, help, or generic resources would be appreciated!

Answer 1

Basically, you have to compile the libraries (eg, PIL) either using Docker or, even better, an EC2 instance.

1. Launch an Docker container like this: docker run --rm -it -v "%cd%:/code" lambci/lambda:build-python3.6 sh

2. Inside there, navigate to the /code dir (cd /code), create a virtualenv (virtualenv env), activate it (source env/bin/activate) and finally, install your library (pip install pillow).

3. Once you have installed your library, you can exit the container. The secret here is to move your package library to the root folder (where your main .py file is). For example, move the folder env/lib/python3.6/site-packages/PIL to the root.

Then, zip your PIL folder together with your .py file and you're set!

Full Example:

The following example compiles and compresses PIL and other common Python libraries to run in AWS Lambda.

Dockerfile:

FROM lambci/lambda:build-python3.6
WORKDIR /code
CMD ["sh", "entrypoint.sh"]

entrypoint.sh:

#!/bin/sh

set -ex

cd /code/

if [ ! -d "PIL" ]; then
    # Create virtual env, activate it and install PIL
    virtualenv env && source env/bin/activate && pip install pillow requests

    # Copy necessary files to the root folder
    rm -f build-output.zip
    #mkdir PIL
    cp -f -r env/lib/python3.6/site-packages/PIL .
    cp -f -r env/lib/python3.6/site-packages/requests .
    
    # below are the dependencies for the requests pkg
    cp -f -r env/lib/python3.6/site-packages/urllib3 .
    cp -f -r env/lib/python3.6/site-packages/certifi .
    cp -f -r env/lib/python3.6/site-packages/chardet .
    cp -f -r env/lib/python3.6/site-packages/idna .
    
    # Remove temp files
    rm -r env
fi

# ZIP it
zip -9 build-output *.py 
zip -9 -r build-output PIL
zip -9 -r build-output requests
zip -9 -r build-output urllib3
zip -9 -r build-output certifi
zip -9 -r build-output chardet
zip -9 -r build-output idna

To build (Windows):

docker build -t build_lambda .
docker run --rm -v "%cd%:/code" build_lambda