python - Django Filewrapper memory error serving big files, how to stream

Question

I have code like this:

@login_required
def download_file(request):
    content_type = "application/octet-stream"
    download_name = os.path.join(DATA_ROOT, "video.avi")

    with open(download_name, "rb") as f:
        wrapper = FileWrapper(f, 8192)
        response = HttpResponse(wrapper, content_type=content_type)
    response['Content-Disposition'] = 'attachment; filename=blabla.avi'
    response['Content-Length'] = os.path.getsize(download_name)
    # response['Content-Length'] = _file.size
    return response

It seems that it works. However, If I download bigger file (~600MB for example) my memory consumption increase by this 600MB. After few such a downloads my server throws:

Internal Server Error: /download/ Traceback (most recent call last):
File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 35, in inner response = get_response(request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py", line 128, in _get_response response = self.process_exception_by_middleware(e, request) File "/home/matous/.local/lib/python3.5/site-packages/django/core/handlers/base.py", line 126, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/home/matous/.local/lib/python3.5/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view return view_func(request, *args, **kwargs) File "/media/matous/89104d3d-fa52-4b14-9c5d-9ec54ceebebb/home/matous/phd/emoapp/emoapp/mainapp/views.py", line 118, in download_file response = HttpResponse(wrapper, content_type=content_type) File "/home/matous/.local/lib/python3.5/site-packages/django/http/response.py", line 285, in init self.content = content File "/home/matous/.local/lib/python3.5/site-packages/django/http/response.py", line 308, in content content = b''.join(self.make_bytes(chunk) for chunk in value) MemoryError

What I am doing wrong? Is it possible to configure it somehow to stream it the piece by piece from hard-drive without this insane memory storage?

Note: I know that big files should not be served by Django, but I am looking for simple approach that allows to verify user access rights for any served file.

Answer 1

Try to use StreamingHttpResponse instead, that will help, it is exactly what you are looking for.

Is it possible to configure it somehow to stream it the piece by piece from hard-drive without this insane memory storage?

import os
from django.http import StreamingHttpResponse
from django.core.servers.basehttp import FileWrapper #django <=1.8
from wsgiref.util import FileWrapper #django >1.8

@login_required
def download_file(request):
   file_path = os.path.join(DATA_ROOT, "video.avi")
   filename = os.path.basename(file_path)
   chunk_size = 8192
   response = StreamingHttpResponse(
       FileWrapper(open(file_path, 'rb'), chunk_size),
       content_type="application/octet-stream"
   )
   response['Content-Length'] = os.path.getsize(file_path)    
   response['Content-Disposition'] = "attachment; filename=%s" % filename
   return response

This will stream your file in chunks without loading it in memory; alternatively, you can use FileResponse,

which is a subclass of StreamingHttpResponse optimized for binary files.