apache spark - DecimalType Issue - scala.MatchError of class java.lang.String -


i using spark 1.6.1 scala 2.10.5 built in. examining weather data have decimal values. here code:

 val sqlcontext = new org.apache.spark.sql.sqlcontext(sc)  import sqlcontext.implicits._   import org.apache.spark.sql.row  import org.apache.spark.rdd.rdd  import org.apache.spark.sql._  import org.apache.spark.sql.types._  import org.apache.spark.sql.sqlcontext  val rawdata=sc.textfile("example_weather.csv").map(_.split(","))  val header=rawdata.first  val rawdatanoheader=rawdata.filter(_(0)!= header(0))  rawdatanoheader.first  object schema { val weatherdata= structtype(seq( structfield("date", stringtype, true), structfield("region", stringtype, true), structfield("temperature", decimaltype(32,16), true), structfield("solar", integertype, true), structfield("rainfall", decimaltype(32,16), true),  structfield("windspeed", decimaltype(32,16), true)) ) }  val datadf=sqlcontext.createdataframe(rawdatanoheader.map(p=>row(p(0),p(1),p(2),p(3),p(4),p(5))), schema.weatherdata)  datadf.registertemptable("weatherdatasql")  val datasql = sqlcontext.sql("select * weatherdatasql")  datasql.collect().foreach(println)

when running code, expected schema , sqlcontext:

scala> object schema {  | val weatherdata= structtype(seq(  | structfield("date", stringtype, true),  | structfield("region", stringtype, true),  | structfield("temperature", decimaltype(32,16), true),  | structfield("solar", integertype, true),  | structfield("rainfall", decimaltype(32,16), true),  | structfield("windspeed", decimaltype(32,16), true))  | )  | } 16/09/24 09:40:58 info blockmanagerinfo: removed broadcast_2_piece0 on localhost:56288 in memory (size: 4.6 kb, free: 511.1 mb) 16/09/24 09:40:58 info blockmanagerinfo: removed broadcast_2_piece0 on localhost:39349 in memory (size: 4.6 kb, free: 2.7 gb) 16/09/24 09:40:58 info contextcleaner: cleaned accumulator 2 16/09/24 09:40:58 info blockmanagerinfo: removed broadcast_1_piece0 on localhost in memory (size: 1964.0 b, free: 511.1 mb) 16/09/24 09:40:58 info blockmanagerinfo: removed broadcast_1_piece0 on localhost:41412 in memory (size: 1964.0 b, free: 2.7 gb) 16/09/24 09:40:58 info contextcleaner: cleaned accumulator 1 defined module schema  scala> val datadf=sqlcontext.createdataframe(rawdatanoheader.map(p=>row(p(0),p(1),p(2),p(3),p(4),p(5))), schema.weatherdata) datadf: org.apache.spark.sql.dataframe = [date: string, region: string, temperature: decimal(32,16), solar: int, rainfall: decimal(32,16), windspeed: decimal(32,16)]

however, last line of code gives me following:

16/09/24 09:41:03 warn tasksetmanager: lost task 0.0 in stage 2.0 (tid 2, localhost): scala.matcherror: 20.21666667 (of class java.lang.string)

the number 20.21666667 indeed first temperature observed specific geographical region. thought had specified temperature decimaltype(32,16). there problem code or sqlcontext calling on?

as recommended changed datadf follows:

val datadf= sqlcontext.createdataframe(rawdatanoheader.map(p=>row(p(0),p(1),bigdecimal(p(2)),p(3),bigdecimal(p(4)),bigdecimal(p(5)))), schema.weatherdata)

unfortunately, casting problem

16/09/24 10:31:35 warn tasksetmanager: lost task 0.0 in stage 2.0 (tid 2, localhost): java.lang.classcastexception: java.lang.string cannot cast java.lang.integer

the code in first edit correct - p(3) had converted toint

i created sample csv file without headers:

2016,a,201.222,12,12.1,5.0 2016,b,200.222,13,12.3,6.0 2014,b,200.111,14,12.3,7.0

the results:

val datadf= sqlcontext.createdataframe(rawdata.map(p=>row(p(0),p(1),bigdecimal(p(2)),p(3).toint,bigdecimal(p(4)),bigdecimal(p(5)))), schema.weatherdata)  datadf.show +----+------+--------------------+-----+-------------------+------------------+ |date|region|         temperature|solar|           rainfall|         windspeed| +----+------+--------------------+-----+-------------------+------------------+ |2016|     a|201.2220000000000000|   12|12.1000000000000000|5.0000000000000000| |2016|     b|200.2220000000000000|   13|12.3000000000000000|6.0000000000000000| |2014|     b|200.1110000000000000|   14|12.3000000000000000|7.0000000000000000| +----+------+--------------------+-----+-------------------+------------------+

-

Comments

Post a Comment

Popular posts from this blog

elasticsearch python client - work with many nodes - how to work with sniffer -

i have 1 cluster 2 nodes. i trying understand best practise connect nodes, , check failover when there downtime on 1 node. from documentation : es = elasticsearch( ['esnode1', 'esnode2'], # sniff before doing sniff_on_start=true, # refresh nodes after node fails respond sniff_on_connection_fail=true, # , every 60 seconds sniffer_timeout=60 ) so tried connect nodes this: client = elasticsearch([ip1, ip2],sniff_on_start=true, sniffer_timeout=10,sniff_on_connection_fail=true) where ip1/ip2 machine ip's (for example 10.0.0.1, 10.0.0.2) in order test it, terminated ip2 (or put non existent if) now, when trying connect, get: transporterror: transporterror(n/a, 'unable sniff hosts - no viable hosts found.') even ip1 exist , up. if trying connect this: es = elasticsearch([ip1, ip2]) then can see in log if client not getting response ip2, move ip1, , return valid response. so missing here something? thought s...
Read more

unity3d - Rotate an object to face an opposite direction -

i have object , b. when click on object b, rotates object b (a faces b). b doesn't face a. need following : when faces b, need face opposite direction. have code rotating @ b. how rotate face opposite direction after that? vector3 targetdirection = target - transform.position; float step = speed * time.deltatime; vector3 newdirection = vector3.rotatetowards (turretdome.transform.forward, targetdirection, step, 0.0f); turretdome.transform.rotation = quaternion.lookrotation (newdirection); you object facing object b, want inverse direction of object after that? objecta.transform.rotation = quaternion.inverse(objecta.transform.rotation) but lets assume example turretdome object a, (negate direction): turretdome.transform.rotation = quaternion.lookrotation (-newdirection); naturally, both of these snippets not show how smoothen rotation, seem know how use time.deltatime. just incase unsure, quaternion.lerp this
Read more

angular - Is it possible to get native element for formControl? -

i've got angular2 reactive form . created formcontrol s , assigned input fields by [formcontrol]=... . understand creates nativeelement <-> formcontrol link. my question: possible nativeelement formcontrol ? wanna myformcontrol.nativeelement.focus() i can share 1 terrible solution works me. in reactive forms can use either 1) formcontroldirective ts mycontrol = new formcontrol('') template <input type="text" [formcontrol]="mycontrol"> or 2) formcontrolname ts myform: formgroup; constructor(private fb: formbuilder) {} ngoninit() { this.myform = this.fb.group({ foo: '' }); } template <form [formgroup]="myform"> <input type="text" formcontrolname="foo"> </form> so these directives write patch like 1) formcontroldirective const originformcontrolngonchanges = formcontroldirective.prototype.ngonchanges; formcontroldirective.proto...
Read more