I'm trying to use the DateTimeFormatter from java.time.format in Spark but it appears to be not serializable. This is the relevant chunk of code:
val pattern = "<some pattern>".r
val dtFormatter = DateTimeFormatter.ofPattern("<some non-ISO pattern>")
val logs = sc.wholeTextFiles(path)
val entries = logs.flatMap(fileContent => {
val file = fileContent._1
val content = fileContent._2
content.split("\r?\n").map(line => line match {
case pattern(dt, ev, seq) => Some(LogEntry(LocalDateTime.parse(dt, dtFormatter), ev, seq.toInt))
case _ => logger.error(s"Cannot parse $file: $line"); None
})
})
How can I avoid the java.io.NotSerializableException: java.time.format.DateTimeFormatter
exception? Is there a better library to parse timestamps? I've read that Joda is also not serializable and has been incorporated in Java 8's time library.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…