创建iceberg表成功
CREATE TABLE iceberg_test1 (i int)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler';
insert数据失败
INSERT INTO iceberg_test1 values(1);
hive的hive.execution.engine为mr
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-2: Map: 1 Cumulative CPU: 10.77 sec HDFS Read: 170682 HDFS Write: 2654 FAIL
Total MapReduce CPU Time Spent: 10 seconds 770 msec
查看hive.log无异常
查看hadoop log，
resourcemanager
的日志：
org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore\n
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:84)\n
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:34)\n
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125)\n
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)\n
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)\n
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122)\n
at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:158)\n
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97)\n
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80)\n
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:47)\n at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:124)\n
at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:111)\n
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitTable(HiveIcebergOutputCommitter.java:320)\n
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.lambda$commitJob$2(HiveIcebergOutputCommitter.java:214)\n
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)\n
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)\n
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)\n
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)\n
at org.apache.iceberg.mr.hive.HiveIcebergOutputCommitter.commitJob(HiveIcebergOutputCommitter.java:207)\n
at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)\n
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:286)\n
at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:238)\n
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n
at java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface\n at java.lang.ClassLoader.defineClass1(Native Method)\n at java.lang.ClassLoader.defineClass(ClassLoader.java:756)\n
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)\n
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)\n
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)\n
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)\n
at java.security.AccessController.doPrivileged(Native Method)\n
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)\n
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n
at java.lang.Class.forName0(Native Method)\n at java.lang.Class.forName(Class.java:348)\n at org.apache.hadoop.hive.metastore.utils.JavaUtils.getClass(JavaUtils.java:52)\n
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:146)\n at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:119)\n at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:112)\n at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n
at java.lang.reflect.Method.invoke(Method.java:498)\n
at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60)\n
at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:72)\n
at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:185)\n
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:63)\n ... 24 more\nCaused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface\n
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)\n
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n ... 50 more\n\n,totalAllocatedContainers=2
其中
at java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface\n
格外刺眼
想起来在部署 hive 的时候在lib中加了 libfb303-0.9.3.jar ，那么 是不是在hadoop中也要加上
随便找了个hadoop的lib文件夹 ～/hadoop-3.2.3/share/hadoop/common/lib ，把 libfb303-0.9.3.jar 加进去
ps.前提是在 mapred-site.xml 的 mapreduce.application.classpath 配置过相关路径。
再次执行，成功
