Linux command to keep the latest data and remove the other duplicates in a csv file

Question

I have a csv file which is sorted based on the timestamp. like below:

DCR_Path,Direction for Translation,Date & Time

data1,Send for Translation To CTM,Sep 30 2014 03:22

data2,Send for Translation To CTM,Sep 30 2014 03:21

data1,Send for Translation To CTM,Sep 30 2014 03:12

data3,Send for Translation To CTM,Sep 30 2014 03:10

data2,Send for Translation To CTM,Sep 30 2014 02:22

data1,Send for Translation To CTM,Sep 30 2014 02:20

---

I need to take the latest and remove the other duplicates and output should be:

DCR_Path,Direction for Translation,Date & Time

data1,Send for Translation To CTM,Sep 30 2014 03:22

data2,Send for Translation To CTM,Sep 30 2014 03:21

data3,Send for Translation To CTM,Sep 30 2014 03:10

---

I tried some sort commands, but unable to get the result. How Can I acheive the desired output?

Answer 1

Try:

sort -u -t, -k1,2 test.txt

• -u is to get unique record
• -t, to use comma as delimiter to separate field
• -k1,2 to select field 1 and 2 for comparison and select uniq