regex - Rename Blob folders with Copy Foreach loops?

Question

I have a hierarchy of parquet files in my blob stored as

yyyy/MM/dd/yyyy_MM_dd_hh_mm_ss_[source].parquet

The dd folders contain multiple files throughout a 24hr period at least one an hour.

examples across folders:

2018/04/01/2018_04_01_06_37_14_ABCXYZ01.Parquet

2019/10/03/2019_10_03_21_52_34_ABCXYZ01.parquet

2020/08/23/2020_08_23_03_00_13_ABCXYZ01.parquet

I would like the folder structure to look like this after the copy.

Year=2019/Month=04/Day=01/2019_04_01_00_00_00_ABCXYZ01.Parquet

I'm probably going to need to copy all 8GB of these files and iterate through but the dynamic content code is what's throwing me off.

Any help on approach would be great. if I do not need to iterate can I use a copy activity and the copy behavior?

Thanks

Answer 1

you need to dynamically generate your folder path in your sink dataset. Please check below implementation for better Idea.

Step1: GetMetaData activity to get file names from source folder dynamically. enter image description here

Step2: ForEach activity to iterate each file and inside ForEach activity I am dynamically generating destination path format and storing it in to newPath variable. expression used in Set variable: Expression ForEachSetVariable

Step3: Inside ForEach Activity, Copy activity Sink dataset with dynamic parameters to take new Path. ForEachCopy