问题描述:

I use hbase to store my data which comes from a crawler. These data are indexed from hbase to Solr. But I have a problem with hBase. As I have larger DB now, everytime I try to index data from hbase the error shown below occures. This is the part of log file hadoop.log

There is something like Caused by: This could be a sign that the server has too many connections (30 is the default). and I found here on SO that this could be beacues of maxClientConnections property so I set up this value to 0 but this solution did not help.

2013-11-11 15:30:22,638 ERROR zookeeper.ClientCnxn - Error while calling watcher

java.lang.NullPointerException

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:315)

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1340)

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)

at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:488)

2013-11-11 15:30:27,925 WARN mapred.FileOutputCommitter - Output path is null in cleanup

2013-11-11 15:30:27,925 WARN mapred.LocalJobRunner - job_local1497313370_0001

org.apache.gora.util.GoraException: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)

at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)

at org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)

at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)

Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:127)

at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)

at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)

... 6 more

Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)

at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)

at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)

at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:109)

... 8 more

Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)

at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)

at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)

at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)

... 14 more

Do you know where is the problem please? Or do you know how can I make sure I have not opened multiple HTable instances simultaneously?

网友答案:

If somebody is interested how I figured out this problem, I created a thread in hBase mailing-list and solved it there http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-td4052728.html#a4052873

I don't know what exactly solved the problem but I tried to combine these things:

1) property maxClientConnections set up to:

<property>
   <name>hbase.zookeeper.property.maxClientCnxns</name>
   <value>0</value>
</property>

2) increased time out

<property>
  <name>zookeeper.session.timeout</name>
  <value>1200000</value>
</property>

<property>
  <name>hbase.zookeeper.property.tickTime</name>
  <value>6000</value>
</property>

3) https://wiki.apache.org/nutch/ErrorMessagesInNutch2#Nutch_2.1_.2B-_HBase_0.90.4_cluster_settings_-WARN_zookeeper.ClientCnxn_-_Session_0x0_for_server_node1.xxxxxx.com.2Fxxx.xxx.xxx.xxx:2181.2C_unexpected_error.2C_closing_socket_connection_and_attempting_reconnect_java.io.IOException:_Connection_reset_by_peer

Then I killed all java processes, start and stop hBase and then run rolling-restart.sh and everything works fine now.

相关阅读:
Top