Error when connect hbase trough spark

Hadoop

Error when connect hbase trough spark

Hi! I have error while when i am trying to connect hbase trough spark. In CHD enabled Kerberos auth. How i can fix this error?

 

Error listing:

 

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:

Wed May 31 13:22:21 MSK 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68214: row 'TestTable,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hw2288-12.example.com,60020,1496066186344, seqNum=0

 

        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)

        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)

        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)

        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)

        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)

        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)

        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)

        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)

        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:867)

        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)

        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)

        at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)

        at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:88)

        at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)

        at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)

        at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)

        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:526)

        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:453)

        at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

        at scala.Option.getOrElse(Option.scala:120)

        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

        at scala.Option.getOrElse(Option.scala:120)

        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

        at scala.Option.getOrElse(Option.scala:120)

        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)

        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)

        at scala.Option.getOrElse(Option.scala:120)

        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)

        at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:190)

        at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:165)

        at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:174)

        at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499)

        at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1499)

        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53)

        at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2086)

        at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$execute$1(DataFrame.scala:1498)

        at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$collect(DataFrame.scala:1505)

        at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1375)

        at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1374)

        at org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:2099)

        at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1374)

        at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1456)

        at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:170)

        at org.apache.spark.sql.DataFrame.show(DataFrame.scala:350)

        at org.apache.spark.sql.DataFrame.show(DataFrame.scala:311)

       at org.apache.spark.sql.DataFrame.show(DataFrame.scala:319)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)

        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)

        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)

        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:37)

        at $iwC$$iwC$$iwC.<init>(<console>:39)

        at $iwC$$iwC.<init>(<console>:41)

        at $iwC.<init>(<console>:43)

        at <init>(<console>:45)

        at .<init>(<console>:49)

        at .<clinit>(<console>)

        at .<init>(<console>:7)

        at .<clinit>(<console>)

        at $print(<console>)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)

        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)

        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)

        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)

        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)

        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)

        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)

        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)

        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)

        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)

        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)

        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)

        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)

        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)

        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)

        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)

        at org.apache.spark.repl.Main$.main(Main.scala:31)

        at org.apache.spark.repl.Main.main(Main.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68214: row 'TestTable,,00000000000000' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hw2288-12.example.com,60020,1496066186344, seqNum=0

        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)

        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to hw2288-12.example.com/10.116.172.75:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hw2288-12.example.com/10.116.172.75:60020 is closing. Call id=9, waitTime=6

        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289)

        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272)

        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)

        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)

        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)

        at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:394)

        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)

        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)

        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)

        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)

        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)

        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)

        ... 4 more

Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to hw2288-12.example.com/10.116.172.75:60020 is closing. Call id=9, waitTime=6

        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084)

        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863)

        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580)

 

Product versions:

Version: Cloudera Express 5.8.3

SPARK version 1.6.0

HBase 1.2.0-cdh5.8.3

Hive 1.1.0-cdh5.8.3

 

 

1 REPLY

Re: Error when connect hbase trough spark

Some debug logs:

Spark debug:

 

17/05/31 10:24:33 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hw2288-01.example.com:2181,hw2288-02.example.com:2181,hw2288-03.example.com:2181 sessionTimeout=90000 watcher=hconnection-0x467f47f80x0, quorum=hw2288-01.example.com:2181,hw2288-02.example.com:2181,hw2288-03.example.com:2181, baseZNode=/hbase

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false

17/05/31 10:24:33 INFO zookeeper.ClientCnxn: Opening socket connection to server hw2288-03.example.com/10.116.172.66:2181. Will not attempt to authenticate using SASL (unknown error)

17/05/31 10:24:33 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.116.172.65:59049, server: hw2288-03.example.com/10.116.172.66:2181

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Session establishment request sent on hw2288-03.example.com/10.116.172.66:2181

17/05/31 10:24:33 INFO zookeeper.ClientCnxn: Session establishment complete on server hw2288-03.example.com/10.116.172.66:2181, sessionid = 0x35c401212065057, negotiated timeout = 60000

17/05/31 10:24:33 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x467f47f80x0, quorum=hw2288-01.example.com:2181,hw2288-02.example.com:2181,hw2288-03.example.com:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null

17/05/31 10:24:33 DEBUG zookeeper.ZooKeeperWatcher: hconnection-0x467f47f8-0x35c401212065057 connected

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 1,3  replyHeader:: 1,403727126903,0  request:: '/hbase/hbaseid,F  response:: s{373662155744,403727047347,1493887655895,1496066189315,28,0,22,0,67,0,373662155744}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 2,4  replyHeader:: 2,403727126903,0  request:: '/hbase/hbaseid,F  response:: #ffffffff000146d61737465723a3630303030ffffffd5ffffff8878222628ffffffdb4550425546a2464383666613638622d343531372d343332652d393565352d663532666130643663313163,s{373662155744,403727047347,1493887655895,1496066189315,28,0,22,0,67,0,373662155744}

17/05/31 10:24:33 DEBUG ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6cfa6a04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null

17/05/31 10:24:33 INFO util.RegionSizeCalculator: Calculating region sizes for table "TestTable".

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 3,8  replyHeader:: 3,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 4,4  replyHeader:: 4,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 5,8  replyHeader:: 5,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 6,4  replyHeader:: 6,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 7,8  replyHeader:: 7,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:33 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 8,4  replyHeader:: 8,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:33 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:33 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 9,8  replyHeader:: 9,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 10,4  replyHeader:: 10,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 11,8  replyHeader:: 11,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 12,4  replyHeader:: 12,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:34 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:34 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:34 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #345

17/05/31 10:24:34 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #345

17/05/31 10:24:34 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 13,8  replyHeader:: 13,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 14,4  replyHeader:: 14,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 15,8  replyHeader:: 15,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:34 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 16,4  replyHeader:: 16,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:34 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:34 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:35 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #346

17/05/31 10:24:35 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #346

17/05/31 10:24:35 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:35 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 17,8  replyHeader:: 17,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:35 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 18,4  replyHeader:: 18,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:35 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 19,8  replyHeader:: 19,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:35 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 20,4  replyHeader:: 20,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:35 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:35 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:36 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #347

17/05/31 10:24:36 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #347

17/05/31 10:24:36 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:37 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #348

17/05/31 10:24:37 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #348

17/05/31 10:24:37 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:37 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 21,8  replyHeader:: 21,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:37 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 22,4  replyHeader:: 22,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:37 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 23,8  replyHeader:: 23,403727126903,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:37 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 24,4  replyHeader:: 24,403727126903,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:37 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:37 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:38 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #349

17/05/31 10:24:38 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #349

17/05/31 10:24:38 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:39 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #350

17/05/31 10:24:39 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #350

17/05/31 10:24:39 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:40 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #351

17/05/31 10:24:40 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #351

17/05/31 10:24:40 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:41 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #352

17/05/31 10:24:41 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #352

17/05/31 10:24:41 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:41 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 25,8  replyHeader:: 25,403727126919,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:41 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 26,4  replyHeader:: 26,403727126919,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:41 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 27,8  replyHeader:: 27,403727126919,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:41 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 28,4  replyHeader:: 28,403727126919,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:41 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:41 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:42 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-01.example.com/10.116.172.85:8020 from user@example.com: closed

17/05/31 10:24:42 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-01.example.com/10.116.172.85:8020 from user@example.com: stopped, remaining connections 1

17/05/31 10:24:42 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #353

17/05/31 10:24:42 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #353

17/05/31 10:24:42 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:43 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #354

17/05/31 10:24:43 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #354

17/05/31 10:24:43 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:44 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #355

17/05/31 10:24:44 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #355

17/05/31 10:24:44 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:45 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #356

17/05/31 10:24:45 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #356

17/05/31 10:24:45 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:46 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #357

17/05/31 10:24:46 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #357

17/05/31 10:24:46 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:47 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #358

17/05/31 10:24:47 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #358

17/05/31 10:24:47 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:48 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #359

17/05/31 10:24:48 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #359

17/05/31 10:24:48 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:49 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #360

17/05/31 10:24:49 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #360

17/05/31 10:24:49 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:50 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #361

17/05/31 10:24:50 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #361

17/05/31 10:24:50 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:51 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #362

17/05/31 10:24:51 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #362

17/05/31 10:24:51 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:51 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 29,8  replyHeader:: 29,403727126919,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:51 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x35c401212065057 after 0ms

17/05/31 10:24:51 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 30,4  replyHeader:: 30,403727126919,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:51 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 31,8  replyHeader:: 31,403727126919,0  request:: '/hbase,F  response:: v{'replication,'meta-region-server,'rs,'splitWAL,'backup-masters,'table-lock,'flush-table-proc,'region-in-transition,'online-snapshot,'acl,'master,'running,'tokenauth,'recovering-regions,'draining,'namespace,'hbaseid,'table}

17/05/31 10:24:51 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x35c401212065057, packet:: clientPath:null serverPath:null finished:false header:: 32,4  replyHeader:: 32,403727126919,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff0001a726567696f6e7365727665723a3630303230ffffffd476ffffffe0434bffffffb9ffffff941950425546a2ba1e6877323238382d31322e6f642d6465762e6f6d6567612e736272662e727510fffffff4ffffffd4318ffffffe8ffffffb8fffffff9ffffffa3ffffffc52b100183,s{403727047413,403727047413,1496066195072,1496066195072,0,0,1,0,84,0,403727047413}

17/05/31 10:24:51 DEBUG ipc.RpcClientImpl: Use SIMPLE authentication for service ClientService, sasl=false

17/05/31 10:24:51 DEBUG ipc.RpcClientImpl: Connecting to hw2288-12.example.com/10.116.172.75:60020

17/05/31 10:24:52 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #363

17/05/31 10:24:52 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #363

17/05/31 10:24:52 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:53 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #364

17/05/31 10:24:53 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #364

17/05/31 10:24:53 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:54 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #365

17/05/31 10:24:54 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #365

17/05/31 10:24:54 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:55 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #366

17/05/31 10:24:55 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #366

17/05/31 10:24:55 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:56 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #367

17/05/31 10:24:56 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #367

17/05/31 10:24:56 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:57 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #368

17/05/31 10:24:57 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #368

17/05/31 10:24:57 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms

17/05/31 10:24:58 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com sending #369

17/05/31 10:24:58 DEBUG ipc.Client: IPC Client (1648869516) connection to hw2288-02.example.com/10.116.172.65:8032 from user@example.com got value #369

17/05/31 10:24:58 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 1ms

17/05/31 10:24:58 DEBUG ipc.Client: The ping interval is 60000 ms.

17/05/31 10:24:58 DEBUG ipc.Client: Connecting to hw2288-01.example.com/10.116.172.85:8020

17/05/31 10:24:58 DEBUG security.UserGroupInformation: PrivilegedAction as:user@example.com (auth:KERBEROS) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726)

 

HBase debug log:

2017-05-31 12:00:22,708 TRACE org.apache.hadoop.ipc.ProtobufRpcEngine: 138: Response <- hw2288-01.example.conf/10.116.172.85:8020: renewLease {}

2017-05-31 12:00:32,708 DEBUG org.apache.hadoop.ipc.Client: IPC Client (815336475) connection to hw2288-01.example.conf/10.116.172.85:8020 from hbase/hw2288-12.example.conf@example.conf: closed

2017-05-31 12:00:32,708 DEBUG org.apache.hadoop.ipc.Client: IPC Client (815336475) connection to hw2288-01.example.conf/10.116.172.85:8020 from hbase/hw2288-12.example.conf@example.conf: stopped, remaining connections 0

 

ZooKeeper debug log:

2017-05-31 10:24:33,674 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x35c401212065057 with negotiated timeout 60000 for client /10.116.172.65:59049

2017-05-31 10:25:22,005 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x35c401212065057

2017-05-31 10:25:22,006 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.116.172.65:59049 which had sessionid 0x35c401212065057