parsing the json kind of file to find out particular values using unix shell script/Python

Question

I would like to extract few lines of given file using shell script. The problem is I have set of files which need to parsed and extract data. The below examples explains my scenario: I am looking to extract only "source" and "destination" values from the given files.

lists = [ 
{
name = "Account - UU ",
source = "1-account",
destination = "account-hhh",
other = "111111"
},
{
name = "Account - PP,
source = "2-account",
destination = "account-hhh12345",
other = "1212"
},
{
name = "Account - GG ",
source = "3-account",
destination = "account-gg567",
other = "44444"
},
{
name = "Account - QQ,
source = "4-account",
destination = "account-manager123456",
other = "23232323"
}
]

My expected output would be

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

Answer 1

If you can modify the list to a valid json format (as below) and python is your option, please try:

#!/usr/bin/python

lists = [
{
"name" : "Account - UU",
"source" : "1-account",
"destination" : "account-hhh",
"other" : 111111
},
{
"name" : "Account - PP",
"source" : "2-account",
"destination" : "account-hhh12345",
"other" : 1212
},
{
"name" : "Account - GG",
"source" : "3-account",
"destination" : "account-gg567",
"other" : 44444
},
{
"name" : "Account - QQ",
"source" : "4-account",
"destination" : "account-manager123456",
"other" : 23232323
}
]

for i in lists:
    print('source = "%s" | destination = "%s"' % (i["source"], i["destination"]))

Output:

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

EDIT

Assuming you have a HOCON file file.conf which looks like:

lists = [
{
name = "Account - UU",
source = "1-account",
destination = "account-hhh",
other = 111111
},
{
name = "Account - PP",
source = "2-account",
destination = "account-hhh12345",
other = 1212
},
{
name = "Account - GG",
source = "3-account",
destination = "account-gg567",
other = 44444
},
{
name = "Account - QQ",
source = "4-account",
destination = "account-manager123456",
other = 23232323
}
]

Then try to execute the python script:

#!/usr/bin/python

from pyhocon import ConfigFactory

conf = ConfigFactory.parse_file('./file.conf')
for i in conf['lists']:
    print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))

Output:

source = "1-account" | destination = "account-hhh"
source = "2-account" | destination = "account-hhh12345"
source = "3-account" | destination = "account-gg567"
source = "4-account" | destination = "account-manager123456"

EDIT2

If you want to parse multiple files in multiple directories recursively, save the following script as something like myscript.py:

#!/usr/bin/python

from pyhocon import ConfigFactory
import sys

conf = ConfigFactory.parse_file(sys.argv[1])
try:
    if type(conf['lists']) is list:
        for i in conf['lists']:
            print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))
except Exception:
    pass

Then run:

find "$dir" -name "*.conf" -type f -exec python myscript.py {} ;

The command line above assumes the HOCON files have .conf extension and the variable $dir holds the top directory path of the conf files. Please modify them according to your environment.

EDIT3
The python script above may be altered by:

#!/usr/bin/python

from pyhocon import ConfigFactory
import sys

conf = ConfigFactory.parse_file(sys.argv[1])
if 'lists' in conf.keys():
    for i in conf['lists']:
        print('source = "%s" | destination = "%s"' % (i['source'], i['destination']))

which will be a bit smarter.