Power of Django StreamingHttpResponse

While learning Django, a python-based web framework, I decided to perform an example in Django where my goal was to upload a CSV file and display its data onto the frontend in Table format using StreamingHttpResponse. Let's first discuss what is StreamingHTTPResponse.

What is a StreamingHttpResponse?

Most Django applications use HttpResponse. In this, the body of the response is loaded in memory and sent to the HTTP client in a single piece. On the contrary, A StreamingHttpResponse is a response whose body is sent to the client in multiple pieces, or "chunks."

Here's a good example:

from django.http import StreamingHttpResponse

def hello():
    yield 'Hello,'
    yield 'TechAE Blogs!'

def index(request):
    # NOTE: No Content-Length header!
    return StreamingHttpResponse(hello())

A question arises when you can get your response in one GO, then why else would I need to use StreamingHttpResponse? Here my goal is achieved because sending huge files, such as a large CSV file, is one of the finest uses for streaming responses. Normally, you would transmit the full file—whether it was generated dynamically or not—to the client using a HttpResponse. This uses up memory on the server and "time to first byte" (TTFB) communicated to the client for a large file.

You can load portions of the file into memory or generate portions of the file dynamically with a StreamingHttpResponse and start transmitting these portions to the client right away. Interestingly, the complete file does not have to be loaded into memory.

Now, let's start with what requirements we need to start our project.

PREREQUISITES:

Django 4.0.5 (To check the Django version python -m django --version)
Pandas Library Installed
CSV Files present in the folder

Creating forms.py
Editing models.py
Editing urls.py
Creating index.html
Editing views.py
Executing Application

I would prefer that you have finished properly setting up the Django project and are prepared to begin.

Step 1: Creating forms.py

Let's first create a forms.py file in our app (home) directory.

from django import forms

class DocumentForm(forms.Form):
    docfile = forms.FileField(
        label='Select a file'
    )

Step 2: Editing models.py

Now, you can create the form which will be shown in your Django admin.

from django.db import models

class Document(models.Model):
    title = models.CharField(max_length=255)
    docfile = models.FileField(upload_to="static/")

    def __str__(self):
        return "File_"+self.title

Make sure, that your Document model is registered in admin.py.

from django.contrib import admin

from .models import Document

# Register your models here.
admin.site.register(Document)

Step 3: Editing urls.py

Now, add some new URLs to your urls.py file in your app (home) directory.

from django.urls import path
from home import views

urlpatterns = [
    path('', views.index, name='index'),
    path('data', views.data, name='data')
]

Step 4: Creating index.html

Now, in the frontend, we have to create a form so that you can input the file name and upload files. I have downloaded bootstrap for extra styling, you can download it too.

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  {% load static %}
  <link href="{% static 'css/bootstrap.min.css' %}" rel="stylesheet" />
  <script src="{% static 'js/bootstrap.min.js' %}"></script>
  <title>File Retrieval Django</title>
</head>

<body>
  <div class="container d-flex justify-content-center align-items-center flex-column">
    {% if documents %}
    <ul>
      {% for document in documents %}
      <li>
        <a href="{{ document.docfile.url }}">{{ document.docfile.name }}</a>
      </li>
      {% endfor %}
    </ul>
    {% else %}
    <p>No documents.</p>
    {% endif %}
    <form action="{% url 'index' %}" method="post" enctype="multipart/form-data">
      {% csrf_token %}
      <label for="title" class="form-label">Title:</label>
      <input type="text" class="form-control" id="title" name="title" required />
      <p>{{ form.non_field_errors }}</p>
      <p class="form-text">
        {{ form.docfile.label_tag }} {{ form.docfile.help_text }}
      </p>
      <p>{{ form.docfile.errors }} {{ form.docfile }}</p>
      <button type="submit" class="btn btn-dark">Upload</button>
      <a href="{% url 'data' %}" class="btn btn-primary">Data Tables</a>
    </form>
  </div>
</body>

</html>

Step 5: Editing views.py

Lastly, to add functionality to the form, we have to create functions. In the "index" function, if the form is correctly filled then the details will be saved and you will be redirected to data.html. Now, the second function "data" write a data.html file where your file is streamingly rendered into an HTML table using pandas.to_html function.

from django.http import StreamingHttpResponse
from django.shortcuts import render
from django.http import HttpResponseRedirect
from django.urls import reverse
from django.template import loader
from django.contrib import messages

from .models import Document
from .forms import DocumentForm

def data(request):
    html = """
    <!DOCTYPE html>
    <html>
    <meta name="viewport" content="width=device-width, initial-scale=1" />
<head>
  {% load static %}
    <link href="{% static 'css/bootstrap.min.css' %}" rel="stylesheet" />
    <script src="{% static 'js/bootstrap.min.js' %}"></script>
    <title>Data from table</title>
</head></html>"""
    import pandas as pd
    fn = 'static/file_name.csv'
    result = pd.concat((chunk for chunk in pd.read_csv(
        f'{fn}', encoding="utf8",iterator=True, chunksize=5000)))
    data = result.to_html(
        classes='table table-light table-striped table-hover text-center table-bordered', justify='center')
    # write html to file
    text_file = open("templates/data.html", "w", encoding="utf-8")
    text_file.write(html+data)
    text_file.close()
    template = loader.get_template('data.html')
    context = {}
    messages.success(request, "Message sent.")
    return StreamingHttpResponse(template.render(context, request))


def index(request):
    # Handle file upload
    if request.method == 'POST':
        form = DocumentForm(request.POST, request.FILES)
        if form.is_valid():
            newdoc = Document(
                docfile=(request.FILES['docfile']), title=(request.POST["title"]))
            newdoc.save()
            return HttpResponseRedirect(reverse('data'))

    else:
        form = DocumentForm()  # A empty, unbound form

    # Load documents for the list page
    documents = Document.objects.all()
    print(documents)

    return render(
        request,
        'index.html',
        {'documents': documents, 'form': form}
    )

Step 6: Executing Application

The form looks like this:

After clicking the Upload button, the details will be saved in Django admin, where it looks like this:

Conclusion:

That's all there is to it. We started off thinking about when to use StreamingHttpResponse over HttpResponse to an example of using StreamingHttpResponse in our Django project, uploading the file, and successfully streaming table response. And we were able to stream a response — bravo!