awk - generating a output file by comparing the 2 files in shell script

Question

I have 2 .txt files

  INPUT FILE 1:    contents of file 1(REQUEST DETAILS: RegRequest.txt):
    2020-12-21 18:28:32,0000000001,abc@gmail.com,919876543210
    2020-12-21 18:28:32,0000000002,abc@yahoo.com,919876543211
    2020-12-21 18:28:32,0000000003,abc@gmail.com,919876543212


 INPUT FILE 2:  contents of file 2(RESPONSE DETAILS: RegReponse.txt):
    0000000001
    0000000003

Output file:
    2020-12-21 18:28:32,0000000001,abc@gmail.com,919876543210,true
    2020-12-21 18:28:32,0000000002,abc@yahoo.com,919876543211,false
    2020-12-21 18:28:32,0000000003,abc@gmail.com,919876543212,true

The content of the input file2 tells us that which all request are success, in the above case 0000000001 and 0000000003 are success, so I have to create a output file containing the request file data and new column gets added saying whether its success or failure.
like below: 2020-12-21 18:28:32,0000000001,abc@gmail.com,919876543210,true.

I will be getting 1 single file these 2 files are generated by executing my script but I got stuck at this point.

Here is my script I wrote to generate 2 different files as per the requirement

#!/bin/bash

#fetching only one file
`cat mainfile.cdr | grep -v Orig |awk '
BEGIN{ FS="|"; OFS="," }
{
split($4,RegReqArr,":");
if (RegReqArr[4] == 10) { 
        split($1,dateTime,",");
        split($2,ReqidArr,":");
        gsub(/^[^<]*|;.*/,"",$8)
        split($17,MsisdnArr,":");
        print dateTime[1],ReqidArr[2],substr($8,2,length($8)-2),MsisdnArr[2] }  }' > RegRequest.txt`

`cat mainfile.cdr | grep -v Orig |awk '
BEGIN{ FS="|"; OFS="," } 
{         
split($4,RegReqArr,":");
if (RegReqArr[4] == 11) { 
        split($2,ReqidArr,":"); 
        split($7,Response,":"); 
        if (Response[2]==200) {
                print ReqidArr[2]
        }     }  }' > RegResponse.txt`

After generating these 2 files RegRequest.txt and RegReponse.txt I want to generate final file like below

2020-12-21 18:28:32,0000000001,abc@gmail.com,919876543210,true
        2020-12-21 18:28:32,0000000002,abc@yahoo.com,919876543211,false
        2020-12-21 18:28:32,0000000003,abc@gmail.com,919876543212,true

question from:https://stackoverflow.com/questions/65558259/generating-a-output-file-by-comparing-the-2-files-in-shell-script

Answer 1

With your shown samples, could you please try following. Written and tested with GNU awk.

awk '
BEGIN{
  FS=OFS=","
}
FNR==NR{
  arr[$0]
  next
}
{
  print $0,($2 in arr)?"true":"false"
}
'  Input_file2  Input_file1

Also you are setting field separator as | which is not valid here because your shown samples shows you have comma separated lines, so changed it to , in above.

Explanation: Adding detailed explanation for above.

awk '                      ##Starting awk program from here.
BEGIN{                     ##Starting BEGIN section of this program from here.
  FS=OFS=","               ##Setting field separator and output field separator as , here.
}
FNR==NR{                   ##Checking condition which will be TRUE when file2 is being read.
  arr[$0]                  ##Creating array arr with index of current line.
  next                     ##next will skip all further statements from here.
}
{
  print $0,($2 in arr)?"true":"false"  ##Printing current line and then printing either true OR false based on 2nd field is present in arr or not.
}
' file2 file1              ##Mentioning Input_file name here.