beeswax - Hive - Out of Memory Exception -

-

which results in mr job. mr job runs successfully, when beeswax tries render result oom exception.

i wondering if there configuration setting me passed issue.

java.lang.outofmemoryerror: java heap space     @ java.util.arrays.copyofrange(arrays.java:2694)     @ java.lang.string.<init>(string.java:203)     @ java.nio.heapcharbuffer.tostring(heapcharbuffer.java:561)     @ java.nio.charbuffer.tostring(charbuffer.java:1201)     @ org.apache.hadoop.io.text.decode(text.java:394)     @ org.apache.hadoop.io.text.decode(text.java:371)     @ org.apache.hadoop.io.text.tostring(text.java:273)     @ org.apache.hadoop.hive.serde2.serdeutils.buildjsonstring(serdeutils.java:280)     @ org.apache.hadoop.hive.serde2.serdeutils.getjsonstring(serdeutils.java:220)     @ org.apache.hadoop.hive.serde2.delimitedjsonserde.serializefield(delimitedjsonserde.java:59)     @ org.apache.hadoop.hive.serde2.lazy.lazysimpleserde.serialize(lazysimpleserde.java:427)     @ org.apache.hadoop.hive.ql.exec.listsinkoperator.processop(listsinkoperator.java:91)     @ org.apache.hadoop.hive.ql.exec.operator.process(operator.java:474)     @ org.apache.hadoop.hive.ql.exec.fetchoperator.pushrow(fetchoperator.java:498)     @ org.apache.hadoop.hive.ql.exec.fetchtask.fetch(fetchtask.java:137)     @ org.apache.hadoop.hive.ql.driver.getresults(driver.java:1474)     @ com.cloudera.beeswax.beeswaxserviceimpl$runningquerystate.materializeresults(beeswaxserviceimpl.java:434)     @ com.cloudera.beeswax.beeswaxserviceimpl$runningquerystate.fetch(beeswaxserviceimpl.java:543)     @ com.cloudera.beeswax.beeswaxserviceimpl$5.run(beeswaxserviceimpl.java:986)     @ com.cloudera.beeswax.beeswaxserviceimpl$5.run(beeswaxserviceimpl.java:981)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:415)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1438)     @ com.cloudera.beeswax.beeswaxserviceimpl.dowithstate(beeswaxserviceimpl.java:772)     @ com.cloudera.beeswax.beeswaxserviceimpl.fetch(beeswaxserviceimpl.java:980)     @ com.cloudera.beeswax.api.beeswaxservice$processor$fetch.getresult(beeswaxservice.java:987)     @ com.cloudera.beeswax.api.beeswaxservice$processor$fetch.getresult(beeswaxservice.java:971)     @ org.apache.thrift.processfunction.process(processfunction.java:39)     @ org.apache.thrift.tbaseprocessor.process(tbaseprocessor.java:39)     @ org.apache.thrift.server.tthreadpoolserver$workerprocess.run(tthreadpoolserver.java:244)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615)

update increased memory settings in cloudera manager no cigar. after restarting service first time run query works. second time run fails:

  • hue - beeswax server (default) / resource management - java heap size of beeswax server in bytes [1 gib]
  • hive - gateway (default) / resource management - client java heap size in bytes [1 gib]
  • hive - hiveserver2 (default) / resource management - java heap size of hiveserver2 in bytes [1 gib]

there 3 -xmx can play (increase) - client java, hive serever 2 , hive meta store server. guess you're hitting 1 of these limtits.


Comments

Post a Comment

Popular posts from this blog

PHP DOM loadHTML() method unusual warning -

-
i had following code works fine on localhost: $document = new domdocument(); $document->loadhtml($p_result); $form = $document->getelementsbytagname('form')->item(0); // code continues using $form variable after same code updated on outside server, loadhtml() failed , issued warning. warning: domdocument::loadhtml() expects parameter 1 valid path, string given in path/name/to/script.php returned null instead of object code pretty gets fatal error. note contents of $p_result absolutely same on outside server , on localhost. but why displays kind of warning , why not work? doesn't loadhtml() expects argument 1 string in first place? why method expects parameter 1 valid path ? just make clear i'm not calling loadhtmlfile() , i'm calling loadhtml() . thanks. you're affected one of php bugs . issue present in php 5.6.8 , 5.6.9. have affected php version on server, , bug-free version on localhost. the bug forbids null
Read more

python - How to create jsonb index using GIN on SQLAlchemy? -

-
here's current code creating index jsonb. index("mytable_data_idx_id_key", mytable.data['id'].astext, postgresql_using='gin') but got error. sqlalchemy.exc.programmingerror: (psycopg2.programmingerror) data type text has no default operator class access method "gin" hint: must specify operator class index or define default operator class data type. [sql: "create index event_data_idx_id_key on event using gin ((data ->> 'id'))"] is there way create index on sqlalchemy? the postgresql specific sqlalchemy docs @ http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#operator-classes mention postgresql_ops dictionary provide "operator class" used postgresql, , provide example illustrating use: index('my_index', my_table.c.id, my_table.c.data, postgresql_ops={ 'data': 'text_pattern_ops',
Read more

c# - TransactionScope not rolling back although no complete() is called -

-
i'm using transactionscope rollback transaction fail bool errorreported = false; action<importerrorlog> newerrorcallback = e => { errorreported = true; errorcallback(e); }; using (var transaction = new transactionscope()) { foreach (importtaskdefinition task in taskdefinition) { loader.load(streamfile, newerrorcallback, task.destinationtable, processingtaskid); } if (!errorreported) transaction.complete(); } i'm sure there no transactionscope started ahead or after code. i'm using entity framework insert in db. regardless state of errorreported transaction never rolled in case of error. what missing ? transactionscope sets transaction.current . that's does. wants transacted must @ property. i believe ef each time connection opened reason. probably, connection open when scope installed. open connection inside of scope or enlist manually. ef has nasty "design decision": default opens
Read more