python - flask make_response with large files

Question

So I'm real green with file I/O and memory limits and the such, and I'm having a rough time getting my web application to successfully serve large file downloads to a web browser with flask's make_response. The following code works on smaller files (<~1GB), but gives me a MemoryError Exception when I get into larger files:

raw_bytes = ""
with open(file_path, 'rb') as r:
    for line in r:
        raw_bytes = raw_bytes + line
response = make_response(raw_bytes)
response.headers['Content-Type'] = "application/octet-stream"
response.headers['Content-Disposition'] = "inline; filename=" + file_name
return response

I'm assuming that sticking over 2 GB worth of binary data into a string is probably a big no-no, but I don't know an alternative to accomplishing these file download black magicks. If someone could get me on the right track with a chunky[?] or buffered approach for file downloads, or just point me toward some intermediate-level resources to facilitate a deeper understanding of this stuff, I would greatly appreciate it. Thanks!

Answer 1

See the docs on Streaming Content. Basically, you write a function that yields chunks of data, and pass that generator to the response, rather than the whole thing at once. Flask and your web server do the rest.

from flask import stream_with_context, Response

@app.route('/stream_data')
def stream_data():
    def generate():
        # create and return your data in small parts here
        for i in xrange(10000):
            yield str(i)

    return Response(stream_with_context(generate()))

If the file is static, you can instead take advantage of send_from_directory(). The docs advise you to use nginx or another server that supports X-SendFile, so that reading and sending the data is efficient.