我试图通过我在 Hue 中创建的 Oozie 工作流程运行 Hive 操作,但该操作永远“心跳”并且不会执行 Hive SQL。
我读过其他关于心跳永远的文章,但这篇文章似乎发生在不同的时刻,即在解析 SQL 语句之后。我检查了集群中每个节点上的内存,并验证了任务计数参数是否合理。
这是 hive-config.xml 文件:
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:hive://10.1.10.250:10000/testdb</value>
<description>JDBC connect string</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.apache.hadoop.hive.jdbc.HiveDriver</value>
<description>JDBC driver</description>
</property>
</configuration>
我知道 Hive 连接正在工作,因为如果提供了错误的 SQL 语句、错误的 URL 或错误的驱动程序名称,操作就会失败。
这是操作标准输出日志:
[...truncated]
=================================================================
>>> Invoking Hive command line now >>>
4283 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
4284 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
4284 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
4339 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
4354 [main] INFO hive.ql.parse.ParseDriver - Parsing command: create table testdb.temp99 (col1 int)
4665 [main] INFO hive.ql.parse.ParseDriver - Parse Completed
4667 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=parse start=1418968298270 end=1418968298598 duration=328 from=org.apache.hadoop.hive.ql.Driver>
4667 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
4733 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Starting Semantic Analysis
4735 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Creating table testdb.temp99 position=13
4760 [main] INFO org.apache.hadoop.hive.ql.Driver - Semantic Analysis Completed
4775 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=semanticAnalyze start=1418968298598 end=1418968298706 duration=108 from=org.apache.hadoop.hive.ql.Driver>
4784 [main] INFO org.apache.hadoop.hive.ql.Driver - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
4784 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=compile start=1418968298215 end=1418968298715 duration=500 from=org.apache.hadoop.hive.ql.Driver>
4785 [main] INFO org.apache.hadoop.hive.ql.Driver - Concurrency mode is disabled, not creating a lock manager
4785 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
4785 [main] INFO org.apache.hadoop.hive.ql.Driver - Starting command: create table testdb.temp99 (col1 int)
4792 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=TimeToSubmit start=1418968298215 end=1418968298723 duration=508 from=org.apache.hadoop.hive.ql.Driver>
4792 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
4792 [main] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
4815 [main] INFO hive.ql.exec.DDLTask - Default to LazySimpleSerDe for table testdb.temp99
4935 [main] INFO org.apache.hadoop.hive.metastore.HiveMetaStore - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
4959 [main] INFO org.apache.hadoop.hive.metastore.ObjectStore - ObjectStore, initialize called
5261 [main] INFO DataNucleus.Persistence - Property datanucleus.cache.level2 unknown - will be ignored
Heart beat
Heart beat
[...forever...]
为什么工作流心跳在日志中的此时跳动而不是继续?
附录:
与此 Hive 操作关联的 Oozie 工作流程是:
<workflow-app name="Hive-copy" xmlns="uri:oozie:workflow:0.4">
<start to="Hive"/>
<action name="Hive">
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/test/hive-config.xml</job-xml>
<script>/user/test/test.sql</script>
<file>hive-config.xml#hive-config.xml</file>
</hive>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>