python - Pyspark: Select part of the string(file path) column values

Question

Pyspark: Split and select part of the string column values

How can I select the characters or file path after the 4th(from left) backslash from the column in a spark DF?

Sample rows of the pyspark column:

\DDevjohnnyDesktopTEST
\DDevmattDesktopTESTNEW
\DDevmattDesktopTESTOLDTEST
\EdevpeterDesktopRUNSUBFOLDERNew
\K924produmsDesktopRUNSUBFOLDERNew
\LE345jskx
fkDesktopRUNSUBFOLDERNew
.
.
.
\ls53f7sn3vsohskmwqsdsfkse

Expected Output

johnnyDesktopTEST
mattDesktopTESTNEW
mattDesktopTESTOLDTEST
peterDesktopRUNSUBFOLDERNew
umsDesktopRUNSUBFOLDERNew
rfkDesktopRUNSUBFOLDERNew
.
.
.
vsohskmwqsdsfkse

My previous question led to this new question. Appreciate any help.

Answer 1

You may use a regular expression in regexp_replace eg.

from pyspark.sql import functions as F

df = df.withColumn('sub_path',F.regexp_replace("path","^\\\\[a-zA-Z0-9]+\\[a-zA-Z0-9]+\\",""))

you may also be more flexible with this solution eg.

from pyspark.sql import functions as F
no_of_slashes=4 # number of slashes to consider here

# we build the regular expression by repeating `"[a-zA-Z0-9]+\\"`
# NB. We subtract 2 since we start with the frst 2 slashes
df = df.withColumn('sub_path',F.regexp_replace("path","^\\\\"+("[a-zA-Z0-9]+\\"*(no_of_slashes-2)),""))

Let me know if this works for you.