Converting large CSV to JSON in AWS Lambda using python

Question

I have a lambda function which takes an incoming csv file and converts it to JSON. But this is only working on a small file. When I tested with a large file (1GB) it failed with an out of disk space error. I've been searching for ways around this and have read about the boto streaming option. But I'm struggling with how to code for this, since I'm very new to python. Can anyone give me some pointers with this?

Are there any better ways to accomplish my task...converting a large csv to json in lambda using python. Thanks!!

        s3_object = s3.get_object(Bucket=bucket, Key=key)
        data = s3_object['Body'].read()
        contents = data.decode('utf-8')
        
        print(type(filename_csv))
        print(type(key))
        
        with open(filename_csv, 'a') as csv_data:
           csv_data.write(contents)
        
        with open(filename_csv) as csv_data:
            csv_reader = c1.DictReader(csv_data)
            
            with open(filename_json, 'a') as output_file:
                for csv_row in csv_reader:
                    json.dump(csv_row, output_file)
                    output_file.write('
')
            
        with open(filename_json, 'r') as json_file_contents:
            response = s3.put_object(Bucket=bucket, Key=keyname_s3, Body=json_file_contents.read())

question from:https://stackoverflow.com/questions/65894967/converting-large-csv-to-json-in-aws-lambda-using-python

Answer 1

Waitting for answers