python 3.x - How to mount a Google Bucket in Kubeflow Pipeline?

Question

I have a KubeFlow Pipeline up and running on a VM in GCP with KF. I create the pipeline using a Jupyter Notebook server with image jupyter-kale and python. The first part of the pipeline is doing the dataprep, it downloads images and saves them to a PVC. This all works just fine but I run out of storage space so I decided to save the images downloaded directly to a google Bucket instead using the PVC. I modified my pipeline as shown in the code below:

import kfp
import kfp.dsl as dsl
import kfp.onprem as onprem
import kfp.compiler as compiler
import os

@dsl.pipeline(
  name='try_mount',
  description='...'
)

def one_d_pipe(pvc_name = "gs://xxx-images/my_folder/"):
    
    trymount = dsl.ContainerOp(
        name="trymount",
        #image = "sprintname3:0.2.0",
        image = "eu.gcr.io/xxx-admin/kubeflow/trymount_1:0.1"
    )
    
    steps = [trymount]
    for step in steps:
        step.apply(onprem.mount_pvc(pvc_name, "gs://xxx-images/my_folder/", '/home/jovyan/data'))

But this code resulting in an error message right after start saying that the volume has invalid value and could not be found:

This step is in Error state with this message: Pod "try-mount-75vrt-3151677017" is invalid: [spec.volumes[2].name: Invalid value: "gs://xxx-images/my_folder/": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is 'a-z0-9?'), spec.containers[0].volumeMounts[3].name: Not found: "gs://xxx-images/my_folder/", spec.containers[1].volumeMounts[0].name: Not found: "gs://xxx-images/my_folder/"]

So, my question:

• How to mount a google bucket in Kubeflow Pipelines?

Answer 1

You can't mount a bucket as a volume. It's not a file system. However, I'm sure that you can cheat by use gcsfuse on your VM.

• On your VM, mount the GCS bucket with fuse

gcsfuse xxx-images /path/to/mount-gcs

• Then in your code, use this directory. No mount required, the GCS is already mount with GCSFuse.