python - Extremely slow writing speed when inserting rows into Hive table using impyla -


i'm experiencing extremely slow writing speed when trying insert rows partitioned hive table using impyla.

this example of code wrote in python:

from impala.dbapi import connect  targets = ... # targets dictionary of objects of specific class yesterday = datetime.date.today() - datetime.timedelta(days=1) log_datetime = datetime.datetime.now() query = """         insert my_database.mytable         partition (year={year}, month={month}, day={day})         values ('{yesterday}', '{log_ts}', %s, %s, %s, 1, 1)         """.format(yesterday=yesterday, log_ts=log_datetime,                    year=yesterday.year, month=yesterday.month,                    day=yesterday.day) print(query) rows = tuple([tuple([i.campaign, i.adgroup, i.adwordsid])               in targets.values()])  connection = connect(host=os.environ["hadoop_ip"],                      port=10000,                      user=os.environ["hadoop_user"],                      password=os.environ["hadoop_passwd"],                      auth_mechanism="plain") cursor = connection.cursor() cursor.execute("set hive.exec.dynamic.partition.mode=nonstrict") cursor.executemany(query, rows)

interestingly, though i'm launching executemany command impyla still resolve multiple mapreduce jobs. in fact can see many mapreduce jobs launched many tuples included in tuple of tuples object i'm passing impyla.executemany method.

do see wrong? give idea after more hour wrote 350 rows.


-

Comments

Post a Comment

Popular posts from this blog

angular - Is it possible to get native element for formControl? -

i've got angular2 reactive form . created formcontrol s , assigned input fields by [formcontrol]=... . understand creates nativeelement <-> formcontrol link. my question: possible nativeelement formcontrol ? wanna myformcontrol.nativeelement.focus() i can share 1 terrible solution works me. in reactive forms can use either 1) formcontroldirective ts mycontrol = new formcontrol('') template <input type="text" [formcontrol]="mycontrol"> or 2) formcontrolname ts myform: formgroup; constructor(private fb: formbuilder) {} ngoninit() { this.myform = this.fb.group({ foo: '' }); } template <form [formgroup]="myform"> <input type="text" formcontrolname="foo"> </form> so these directives write patch like 1) formcontroldirective const originformcontrolngonchanges = formcontroldirective.prototype.ngonchanges; formcontroldirective.proto
Read more

unity3d - Rotate an object to face an opposite direction -

i have object , b. when click on object b, rotates object b (a faces b). b doesn't face a. need following : when faces b, need face opposite direction. have code rotating @ b. how rotate face opposite direction after that? vector3 targetdirection = target - transform.position; float step = speed * time.deltatime; vector3 newdirection = vector3.rotatetowards (turretdome.transform.forward, targetdirection, step, 0.0f); turretdome.transform.rotation = quaternion.lookrotation (newdirection); you object facing object b, want inverse direction of object after that? objecta.transform.rotation = quaternion.inverse(objecta.transform.rotation) but lets assume example turretdome object a, (negate direction): turretdome.transform.rotation = quaternion.lookrotation (-newdirection); naturally, both of these snippets not show how smoothen rotation, seem know how use time.deltatime. just incase unsure, quaternion.lerp this
Read more

javascript - Why jQuery Select box change event is now working? -

i working on product filters user select select box according his/her preference not getting selected value user in jquery. html code:- <select name="product_filter" id="product_filter"> <option value="price_low_first">price : low high</option> <option value="price_high_first">price : high low</option> <option value="latest">latest</option> <option value="popular">most popular</option> </select> jquery code: $(function () { $("#product_filter").change(function () { alert("hi") var selectedtext = $(this).find("option:selected").text(); var selectedvalue = $(this).val(); alert("selected text: " + selectedtext + " value: " + selectedvalue); }); }); hi believe code perfect. once please check jquery.min.js
Read more