jodatime - Convert DStream of case class with joda.DateTime to Spark DataFrame -


i want save dstream hdfs using parquet format. problem case class use joda.datetime while spark sql doesn't support this. example:

    case class log (timestamp: datetime, ...dozen of other fields here...)

but got error: java.lang.unsupportedoperationexception: schema type org.joda.time.datetime not supported when trying convert rdd df: 

    def output(logdstream: dstream[log]) {         logdstream.foreachrdd(elem => {            val df = elem.todf()            df.saveasparquet(...)         });     }

my models complex , have lot of fields, don't want write different case classes rid of joda.datetime. option save directly json parquet it's not ideal. there easy way automatic conversion joda.datetime sql.timestamp used spark (convert spark's dataframe).

thanks.

it's little bit verbose, try mapping log spark sql row:

logdstream.foreachrdd(rdd => {   rdd.map(log => row(     log.timestamp.todate,     log.field2,     ...   )).todf().saveasparquest(...) })

-

Comments

Post a Comment

Popular posts from this blog

unity3d - Rotate an object to face an opposite direction -

i have object , b. when click on object b, rotates object b (a faces b). b doesn't face a. need following : when faces b, need face opposite direction. have code rotating @ b. how rotate face opposite direction after that? vector3 targetdirection = target - transform.position; float step = speed * time.deltatime; vector3 newdirection = vector3.rotatetowards (turretdome.transform.forward, targetdirection, step, 0.0f); turretdome.transform.rotation = quaternion.lookrotation (newdirection); you object facing object b, want inverse direction of object after that? objecta.transform.rotation = quaternion.inverse(objecta.transform.rotation) but lets assume example turretdome object a, (negate direction): turretdome.transform.rotation = quaternion.lookrotation (-newdirection); naturally, both of these snippets not show how smoothen rotation, seem know how use time.deltatime. just incase unsure, quaternion.lerp this
Read more

angular - Is it possible to get native element for formControl? -

i've got angular2 reactive form . created formcontrol s , assigned input fields by [formcontrol]=... . understand creates nativeelement <-> formcontrol link. my question: possible nativeelement formcontrol ? wanna myformcontrol.nativeelement.focus() i can share 1 terrible solution works me. in reactive forms can use either 1) formcontroldirective ts mycontrol = new formcontrol('') template <input type="text" [formcontrol]="mycontrol"> or 2) formcontrolname ts myform: formgroup; constructor(private fb: formbuilder) {} ngoninit() { this.myform = this.fb.group({ foo: '' }); } template <form [formgroup]="myform"> <input type="text" formcontrolname="foo"> </form> so these directives write patch like 1) formcontroldirective const originformcontrolngonchanges = formcontroldirective.prototype.ngonchanges; formcontroldirective.proto
Read more

javascript - Why jQuery Select box change event is now working? -

i working on product filters user select select box according his/her preference not getting selected value user in jquery. html code:- <select name="product_filter" id="product_filter"> <option value="price_low_first">price : low high</option> <option value="price_high_first">price : high low</option> <option value="latest">latest</option> <option value="popular">most popular</option> </select> jquery code: $(function () { $("#product_filter").change(function () { alert("hi") var selectedtext = $(this).find("option:selected").text(); var selectedvalue = $(this).val(); alert("selected text: " + selectedtext + " value: " + selectedvalue); }); }); hi believe code perfect. once please check jquery.min.js
Read more