利用hbase api在本地访问并操作服务器的hbase数据库

2023-11-19

最近因为实验室项目需要,开始研究了hbase。然后想一次性往集群服务器上写入大量的数据,并存到hbase中,考虑到在hbase shell下只能单个数据put,这样对于批量插入数据的要求完全不合适。于是就研究起hbase的java api,然后去大量填充数据到hbase以测试查询的性能。于是,故事开始了。

首先在eclipse下写好了,如下的Java代码:

<span style="font-size:12px;">import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;

public class testCreate {
    public static void main(String[] args){
    	Configuration conf = HBaseConfiguration.create();
    	conf.set("hbase.rootdir", "hdfs://xxx-a.test.com:8020/apps/hbase/data");//使用eclipse时必须添加这个,否则无法定位
        conf.set("hbase.zookeeper.quorum", "xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com");
        conf.set("hbase.zookeeper.property.clientPort", "2181");
        conf.set("zookeeper.znode.parent","/hbase-unsecure");//这行代码也是后来加上的
        try {           
            Connection conn = ConnectionFactory.createConnection(conf);
            System.out.println("StartConnect...");
            Admin hAdmin = conn.getAdmin();
            HTableDescriptor hTableDesc = new HTableDescriptor(TableName.valueOf("Customer2"));
            hTableDesc.addFamily(new HColumnDescriptor("name"));
            hTableDesc.addFamily(new HColumnDescriptor("contactinfo"));
            hTableDesc.addFamily(new HColumnDescriptor("address"));
            hAdmin.createTable(hTableDesc);
            System.out.println("Table created Successfully...");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}</span>
<span style="font-size:12px;"><span style="white-space: pre;">	</span>代码写好,一堆错误。是因为缺少依赖的jar包。以下是添加的jar截图。</span>


这样jar包都是程序运行报错一点点加进去的,分享给大家可以少走冤枉路。(debug也是一种乐趣,虽然有时候想死的心都有,但是坚持找,肯定会解决的大笑

此时还得配置电脑的hosts,在C:\Windows\System32\drivers\etc\目录,有一个hosts文件,在其中加上xxx-a.test.com对应的实际ip地址,这样程序运行才会访问到正确的地方,格式类似于这样xxx-a.test.com      X.X.X.X。基于这样,程序是可以运行的,Java的运行日志会有如下显示:


表示连接hbase数据库成功,创建表成功。皆大欢喜。

下面说一下几个需要注意的地方:

1、conf.set("hbase.rootdir", "hdfs://xxx-a.test.com:8020/apps/hbase/data");//使用eclipse时必须添加这个,否则无法定位
这句话必须要加上,通过这句配置,才能准确的定位hbase实际存储的hdfs位置,所以这里面的<span style="font-family: Arial, Helvetica, sans-serif;">xxx-a.test.com换成详细的ip地址也是可以的。</span>
2、conf.set("hbase.zookeeper.quorum", "xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com");
<span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;">Zookeeper集群的地址列表,用逗号分割。例如:</span><span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;">"xxx-a1.test.com,xxx-a2.test.com,xxx-a3.test.com</span><span style="font-size: 14px; line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">".默认是localhost,是给伪分布式用的。要修改才能在完全分布式的情况下使用。如果在hbase-env.sh设置了HBASE_MANAGES_ZK,这些ZooKeeper节点就会和HBase一起启动。</span><span style="font-family: Helvetica, Tahoma, Arial, sans-serif; font-size: 14px; line-height: 25.2px;">
</span>3、conf.set("hbase.zookeeper.property.clientPort", "2181");
这句话是设置zookeeper监控连接的客户端的端口号是2181,这是默认的配置。
4、conf.set("zookeeper.znode.parent","/hbase-unsecure");//这行代码也是后来加上的
<span style="font-family: Helvetica, Tahoma, Arial, sans-serif; line-height: 25.2px;">ZooKeeper中的HBase的根ZNode。所有的HBase的ZooKeeper会用这个目录配置相对路径。默认情况下,所有的HBase的ZooKeeper文件路径是用相对路径,所以他们会都去这个目录下面。</span>
<span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">5.没加上面那句代码之前我的报错信息一直是:</span></span>
<span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Wed Aug 10 21:29:19 CST 2016, RpcRetryingCaller{globalStartTime=1470835740311, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
</span></span>
<span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4212)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:748)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:669)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:602)
<span style="white-space:pre">	</span>at hbase.testCreate.main(testCreate.java:24)
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1535)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1555)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1706)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
<span style="white-space:pre">	</span>... 5 more
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:907)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:546)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1485)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1526)
<span style="white-space:pre">	</span>... 9 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
<span style="white-space:pre">	</span>at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
<span style="white-space:pre">	</span>at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
<span style="white-space:pre">	</span>at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541)
<span style="white-space:pre">	</span>at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:896)
<span style="white-space:pre">	</span>... 12 more
</span></span>
<span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">加上</span></span><span style="line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">conf.set("zookeeper.znode.parent","/hbase-unsecure");</span><span style="line-height: 25.2px; font-family: Helvetica, Tahoma, Arial, sans-serif;">就可以成功访问并操作hbase数据库。</span>
<span style="font-family:Helvetica, Tahoma, Arial, sans-serif;"><span style="line-height: 25.2px;">以上是个人实际学习过程的一些经验,在此记录一下,欢迎大家一起学习交流hbase相关技术!</span></span>


本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)

利用hbase api在本地访问并操作服务器的hbase数据库 的相关文章

随机推荐