Using GCP storage bucket from a Django app running in GKE

If you have a django app running in a GKE cluster, and you want to store your static files in Google Storage bucket, then this is guide is for you. Let’s get started!

1. Creating the storage bucket

On the GCP dashboard, create a new bucket, and let it be not public. In order to be able to access this bucket from our django app, we need to create a service account (you can create it from the google cloud dashboard), and gives Storage Admin permission so that we can access it. We also need to create a key that we use to authenticate and access the bucket (from the google cloud console, you can go to service accounts, you select the service account for which you want to create a key, and then from the dotted menu, click on manage keys), download the key and save it somewhere on your computer.

2. Storing the credentials in Kubernetes secret

Create a Kubernetes secret to store the credentials file in, and then mount it in the Kubernetes deployment file. Here’s how to create the secret and store the file

kubectl create secret generic gcp-service-account-key --from-file=credentials.json=<path/to/gcp-credentials.json>

You point to the json credentials file key path in this environment variable: GOOGLE_APPLICATION_CREDENTIALS, and Google Storage will automatically use this variable to get the credentials and interact with GCP storage API. In your django deployment file, add this env variable:

 env:
 - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /path/to/your/key/in/kuberntes/secret/gcp-credentials.json

Mount the secret as a volume in your django app deployment file:

volumes:
- name: gcp-service-account-key
  secret:
    secretName: gcp-service-account-key
volumeMounts:
- name: gcp-service-account-key
  mountPath: /path/to/your/key/in/kuberntes/secret

3. Django App

Once done with setting up the bucket storage and the credentials file in Kubernetes, let’s move now to code level at the level of the Django app. First, add the following dependencies:

django-storages[google]
google-api-python-client==2.50.0

Create a python file cusotm_storage.py you can put it anywhere, but as a best bractice, create a folder, put the file in it, create another file __init.py. Now, in the file cusotm_storage.py, create a class that extends the class GoogleCloudStorage, and override these two methods: url (to return a signed url for the files stored in the bucket), and save method so that the uploaded files are saved automatically in the google cloud bucket:

from google.cloud import storage
from storages.backends.gcloud import GoogleCloudStorage
from django.conf import settings
from datetime import timedelta

class CustomGoogleCloudStorage(GoogleCloudStorage):
    def url(self, name):
        """
        Generates a signed URL for accessing a file.
        """
        if settings.DEBUG:
            # In development, serve files directly from the local media folder
            return super().url(name)
        else:
            # In production, generate a signed URL
            storage_client = storage.Client()
            bucket = storage_client.bucket(self.bucket_name)
            blob = bucket.blob(f'someOtherFolder/{name}')
            return blob.generate_signed_url(
                expiration=timedelta(minutes=10),
                method='GET'
            )
    def _save(self, name, content):
        """
        Saves new content to the file specified by name.
        """
        # Prepend 'someOtherFolder/' that is under your bucket (if you have any) to the name to match your bucket structure
        name = f'someOtherFolder/{name}'
        return super()._save(name, content)

Then, in your settings.py, add the following:

DEFAULT_FILE_STORAGE = 'custom_storages.custom_storage.CustomGoogleCloudStorage'
GS_BUCKET_NAME = 'your-gcp-bucket-name'
MEDIA_URL = f'https://storage.googleapis.com/{GS_BUCKET_NAME}/'

This way, you made the default storage Google Cloud Storage for your django app while in production, a signed url will be used to retrieve resources from the storage bucket. Happy reading! 😊