jodatime - Convert DStream of case class with joda.DateTime to Spark DataFrame -
i want save dstream hdfs using parquet format. problem case class use joda.datetime while spark sql doesn't support this. example: case class log (timestamp: datetime, ...dozen of other fields here...) but got error: java.lang.unsupportedoperationexception: schema type org.joda.time.datetime not supported when trying convert rdd df: def output(logdstream: dstream[log]) { logdstream.foreachrdd(elem => { val df = elem.todf() df.saveasparquet(...) }); } my models complex , have lot of fields, don't want write different case classes rid of joda.datetime. option save directly json parquet it's not ideal. there easy way automatic conversion joda.datetime sql.timestamp used spark (convert spark's dataframe). thanks. it's little bit verbose, try mapping log spark sql row: logdstream.foreachrdd(rdd => { rdd.map(log => row( log.timestamp.todate, log.field2, ... )).todf().saveas