apache spark - Unable to set configuration variable in SparkConf

Question

Background:

Iam currently using Spark Lineage information about all the operations happening around. I have a transformation which has more than 35 fields and I need to log the same. However In Spark the default you can log 25 fields as per Spark Code. This could be overwritten by setting

spark.debug.maxToStringFields

So here is how I do the same

Code

val sparkConf = new SparkConf().set("spark.debug.maxToStringFields","100")
                               .setMaster("local[*]").setAppName("My App")
val sparkSession = SparkSession.builder().conf(sparkConf).getOrCreate()

However the property doesnt seem to be setting in the spark session.

DEBUG

val allConfs = sparkSession.sparkContext.getConf
allConfs.foreach(conf =>println(conf._1+" value "+conf._2))

Here iam unable to see the property that I have set. Also I still get the error/message that spark gives when the default length is 25

What am i missing here?

question from:https://stackoverflow.com/questions/65626012/unable-to-set-configuration-variable-in-sparkconf

Answer 1

Waitting for answers