The MapR on VM stops working after running for a day or so

Question

When the VM starts up, I can initially run HBase with no problem. But then the next day that I come back to work, it fails to startup with this error message:

/opt/mapr/hbase/hbase-0.90.4/bin/hbase thrift start
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2--1, built on 08/30/2011 17:53 GMT
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:host.name=mapr-desktop
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_26
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/mapr/hbase/hbase-0.90.4/bin/../conf:/usr/lib/jvm/java-6-sun-1.6.0.26/lib/tools.jar:/opt/mapr/hbase/hbase-0.90.4/bin/..:/opt/mapr/hbase/hbase-0.90.4/bin/../hbase-0.90.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../hbase-0.90.4-tests.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/activation-1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/asm-3.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/avro-1.3.3.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-cli-1.2.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-codec-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-el-1.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-httpclient-3.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-lang-2.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-logging-1.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-net-1.4.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/core-3.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/guava-r06.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-core-asl-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-jaxrs-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-mapper-asl-1.4.2.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-xc-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jasper-compiler-5.5.23.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jasper-runtime-5.5.23.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jaxb-api-2.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jaxb-impl-2.1.12.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-core-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-json-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-server-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jettison-1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jetty-6.1.26.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jetty-util-6.1.26.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jruby-complete-1.6.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsp-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsr311-api-1.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/log4j-1.2.16.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/protobuf-java-2.3.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/slf4j-api-1.5.8.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/slf4j-log4j12-1.5.8.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/stax-api-1.0.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/thrift-0.2.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/xmlenc-0.52.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/zookeeper-3.3.2.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-capacity-scheduler.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-fairscheduler.jar::/opt/mapr/hadoop/hadoop-0.20.2/lib/logging-0.1.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.1.jar
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/mapr/hbase/hbase-0.90.4/bin/../lib/native/Linux-amd64-64:/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-33-generic
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:user.name=mapr
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/mapr
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/mapr
12/02/01 10:12:21 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:5181 sessionTimeout=180000 watcher=hconnection
12/02/01 10:12:21 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:5181
12/02/01 10:12:21 INFO zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:5181, initiating session
12/02/01 10:12:21 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:5181, sessionid = 0x135324e7741009d, negotiated timeout = 180000
12/02/01 10:12:21 INFO client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found
12/02/01 10:12:21 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 10 failed; retrying after sleep of 1000
org.apache.hadoop.hbase.MasterNotRunningException
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:357)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:94)
    at org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:194)
    at org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler.<init>(ThriftServer.java:188)
    at org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:870)
    at org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:927)
12/02/01 10:12:22 INFO client.HConnectionManager$HConnectionImplementation: ZooKeeper available but no active master location found
12/02/01 10:12:22 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 1 of 10 failed; retrying after sleep of 1000

Any ideas why is this happening, how can it be fixed without restarting the VM, and if there's a way to prevent it from happening again in the future?

EDIT: The log file

2012-02-01 08:44:12,380 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 08:49:12,405 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 08:52:52,064 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 33037ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 08:52:52,065 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 38778ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 10:06:03,322 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 4371181ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 10:06:03,326 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 4371197ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 10:06:03,331 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 4610926ms instead of 300000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 10:06:03,331 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 4610926ms instead of 300000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9
2012-02-01 10:06:03,348 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x135324e77410047, likely server has closed socket, closing socket connection and attempting reconnect
2012-02-01 10:06:03,354 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x135324e77410046, likely server has closed socket, closing socket connection and attempting reconnect
2012-02-01 10:06:03,386 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:06:03,570 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:5181
2012-02-01 10:06:03,571 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:5181, initiating session
2012-02-01 10:06:03,576 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, trying to reconnect.
2012-02-01 10:06:03,576 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Trying to reconnect to zookeeper
2012-02-01 10:06:03,577 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:5181 sessionTimeout=180000 watcher=hconnection
2012-02-01 10:06:03,578 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x135324e77410047 has expired, closing socket connection
2012-02-01 10:06:03,615 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:5181
2012-02-01 10:06:03,669 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:5181, initiating session
2012-02-01 10:06:03,690 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:5181
2012-02-01 10:06:03,690 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:5181, initiating session
2012-02-01 10:06:03,692 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x135324e77410046 has expired, closing socket connection
2012-02-01 10:06:03,695 FATAL org.apache.hadoop.hbase.master.HMaster: master:60000-0x135324e77410046 master:60000-0x135324e77410046 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
2012-02-01 10:06:03,696 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-02-01 10:06:03,696 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-02-01 10:06:03,700 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:5181, sessionid = 0x135324e7741008c, negotiated timeout = 180000
2012-02-01 10:06:03,702 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause, either way it's recommended that you verify your environment.
2012-02-01 10:06:03,702 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-02-01 10:06:04,323 INFO org.apache.hadoop.hbase.master.HMaster$1: mapr-desktop:60000-BalancerChore exiting
2012-02-01 10:06:04,323 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-02-01 10:06:04,324 INFO org.apache.hadoop.hbase.master.CatalogJanitor: mapr-desktop:60000-CatalogJanitor exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-02-01 10:06:04,324 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000: exiting
2012-02-01 10:06:04,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 22 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 25 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 26 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 on 60000: exiting
2012-02-01 10:06:04,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 28 on 60000: exiting
2012-02-01 10:06:04,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 29 on 60000: exiting
2012-02-01 10:06:04,327 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-02-01 10:06:04,329 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer
2012-02-01 10:06:04,330 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-02-01 10:06:04,337 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010
2012-02-01 10:06:04,344 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x135324e77410046 Unable to get data of znode /hbase/master
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:549)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:620)
    at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:189)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297)
2012-02-01 10:06:04,344 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x135324e77410046 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:549)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:620)
    at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:189)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297)
2012-02-01 10:06:04,344 ERROR org.apache.hadoop.hbase.master.ActiveMasterManager: master:60000-0x135324e77410046 Error deleting our own master address node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:549)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:620)
    at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:189)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297)
2012-02-01 10:06:04,344 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: mapr-desktop:60000.timeoutMonitor exiting
2012-02-01 10:06:04,346 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x135324e7741008c
2012-02-01 10:06:04,356 INFO org.apache.zookeeper.ZooKeeper: Session: 0x135324e7741008c closed
2012-02-01 10:06:04,356 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2012-02-01 10:06:04,356 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
Wed Feb  1 10:06:44 EST 2012 Starting master on mapr-desktop
ulimit -n 1024
Wed Feb  1 10:06:44 EST 2012 Waiting for filesystem to come up
Wed Feb  1 10:12:00 EST 2012 Waiting for filesystem to come up
Wed Feb  1 10:14:17 EST 2012 Waiting for filesystem to come up
Wed Feb  1 10:18:58 EST 2012 Waiting for filesystem to come up
Wed Feb  1 10:21:36 EST 2012 Starting master on mapr-desktop
ulimit -n 1024
Wed Feb  1 10:21:36 EST 2012 Waiting for filesystem to come up
ERROR (10003) -  Volume create mapr.hbase failed, File exists
2012-02-01 10:21:42,882 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000
2012-02-01 10:21:43,581 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting
2012-02-01 10:21:43,583 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: starting
2012-02-01 10:21:43,591 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: starting
2012-02-01 10:21:43,591 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: starting
2012-02-01 10:21:43,591 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: starting
2012-02-01 10:21:43,592 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: starting
2012-02-01 10:21:43,638 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: starting
2012-02-01 10:21:43,638 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: starting
2012-02-01 10:21:43,638 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: starting
2012-02-01 10:21:43,638 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: starting
2012-02-01 10:21:43,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: starting
2012-02-01 10:21:43,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: starting
2012-02-01 10:21:43,651 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60000: starting
2012-02-01 10:21:43,657 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 60000: starting
2012-02-01 10:21:43,658 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60000: starting
2012-02-01 10:21:43,658 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 60000: starting
2012-02-01 10:21:43,659 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60000: starting
2012-02-01 10:21:43,661 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60000: starting
2012-02-01 10:21:43,662 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000: starting
2012-02-01 10:21:43,672 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60000: starting
2012-02-01 10:21:43,673 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60000: starting
2012-02-01 10:21:43,673 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60000: starting
2012-02-01 10:21:43,673 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60000: starting
2012-02-01 10:21:43,677 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60000: starting
2012-02-01 10:21:43,678 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 22 on 60000: starting
2012-02-01 10:21:43,691 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000: starting
2012-02-01 10:21:43,692 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60000: starting
2012-02-01 10:21:43,692 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 25 on 60000: starting
2012-02-01 10:21:43,694 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 26 on 60000: starting
2012-02-01 10:21:43,694 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 27 on 60000: starting
2012-02-01 10:21:43,694 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 28 on 60000: starting
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2--1, built on 08/30/2011 17:53 GMT
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=mapr-desktop
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_26
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/opt/mapr/hbase/hbase-0.90.4/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/opt/mapr/hbase/hbase-0.90.4/bin/..:/opt/mapr/hbase/hbase-0.90.4/bin/../hbase-0.90.4-tests.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../hbase-0.90.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/activation-1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/asm-3.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/avro-1.3.3.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-cli-1.2.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-codec-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-el-1.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-httpclient-3.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-lang-2.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-logging-1.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/commons-net-1.4.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/core-3.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/guava-r06.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-core-asl-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-jaxrs-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-mapper-asl-1.4.2.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jackson-xc-1.5.5.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jasper-compiler-5.5.23.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jasper-runtime-5.5.23.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jaxb-api-2.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jaxb-impl-2.1.12.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-core-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-json-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jersey-server-1.4.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jettison-1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jetty-6.1.26.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jetty-util-6.1.26.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jruby-complete-1.6.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsp-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/jsr311-api-1.1.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/log4j-1.2.16.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/protobuf-java-2.3.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/slf4j-api-1.5.8.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/slf4j-log4j12-1.5.8.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/stax-api-1.0.1.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/thrift-0.2.0.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/xmlenc-0.52.jar:/opt/mapr/hbase/hbase-0.90.4/bin/../lib/zookeeper-3.3.2.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-capacity-scheduler.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-fairscheduler.jar::/opt/mapr/hadoop/hadoop-0.20.2/lib/logging-0.1.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.1.jar
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/opt/mapr/hbase/hbase-0.90.4/bin/../lib/native/Linux-amd64-64:/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-33-generic
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=root
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/root
2012-02-01 10:21:43,711 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/opt/mapr/hbase/hbase-0.90.4/bin
2012-02-01 10:21:43,712 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:5181 sessionTimeout=180000 watcher=master:60000
2012-02-01 10:21:43,741 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 29 on 60000: starting
2012-02-01 10:21:43,782 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:5181
2012-02-01 10:21:43,790 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:5181, initiating session
2012-02-01 10:21:43,796 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:5181, sessionid = 0x13536124c440011, negotiated timeout = 180000
2012-02-01 10:21:43,807 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=mapr-desktop:60000
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2012-02-01 10:21:43,843 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-02-01 10:21:43,844 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-02-01 10:21:43,884 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=mapr-desktop:60000
2012-02-01 10:21:44,473 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:5181 sessionTimeout=180000 watcher=hconnection
2012-02-01 10:21:44,503 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:5181
2012-02-01 10:21:44,504 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:5181, initiating session
2012-02-01 10:21:44,507 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:5181, sessionid = 0x13536124c440012, negotiated timeout = 180000
2012-02-01 10:21:44,684 INFO org.apache.hadoop.hbase.master.HMaster: Server active/primary master; mapr-desktop:60000, sessionid=0x13536124c440011, cluster-up flag was=true
2012-02-01 10:21:45,105 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-02-01 10:21:45,510 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-02-01 10:21:45,513 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60010
2012-02-01 10:21:45,528 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 60010 webServer.getConnectors()[0].getLocalPort() returned 60010
2012-02-01 10:21:45,528 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 60010
2012-02-01 10:21:45,528 INFO org.mortbay.log: jetty-6.1.26
2012-02-01 10:21:47,064 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:60010
2012-02-01 10:21:48,564 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:50,065 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:51,565 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:53,066 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:54,566 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:56,066 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:57,567 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:59,067 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2012-02-01 10:21:59,806 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=mapr-desktop,60020,1328052118732, regionCount=0, userLoad=false
2012-02-01 10:22:00,567 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:02,068 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:03,568 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:05,068 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:06,569 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:08,069 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:09,570 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:11,071 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:12,571 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:14,072 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:15,572 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2012-02-01 10:22:17,074 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for regionserver count to settle; count=1, sleptFor=30000
2012-02-01 10:22:17,075 INFO org.apache.hadoop.hbase.master.ServerManager: Exiting wait on regionserver(s) to checkin; count=1, stopped=false, count of regions out on cluster=0
2012-02-01 10:22:17,078 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328000018251 doesn't belong to a known region server, splitting
2012-02-01 10:22:17,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 1 hlog(s) in maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328000018251
2012-02-01 10:22:17,127 INFO org.apache.hadoop.hbase.util.FSMapRUtils: Recovering file maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328000018251/mapr-desktop%3A60020.1328000023942 by changing permission to readonly
2012-02-01 10:22:17,150 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328000018251/mapr-desktop%3A60020.1328000023942 to maprfs://127.0.0.1:7222/hbase/.oldlogs/mapr-desktop%3A60020.1328000023942
2012-02-01 10:22:17,151 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for split writer threads to finish
2012-02-01 10:22:17,165 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2012-02-01 10:22:17,166 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2012-02-01 10:22:17,166 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers finished
2012-02-01 10:22:17,167 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path maprfs://127.0.0.1:7222/hbase/.META./1028785192/recovered.edits/0000000000000000202 (wrote 7 edits in 14ms)
2012-02-01 10:22:17,167 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path maprfs://127.0.0.1:7222/hbase/-ROOT-/70236052/recovered.edits/0000000000000000201 (wrote 1 edits in 13ms)
2012-02-01 10:22:17,167 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file splitting completed in 72 ms for maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328000018251
2012-02-01 10:22:17,167 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder maprfs://127.0.0.1:7222/hbase/.logs/mapr-desktop,60020,1328052118732 belongs to an existing region server
2012-02-01 10:22:17,225 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of -ROOT-,,0 at address=mapr-desktop:60020; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
2012-02-01 10:22:17,226 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKeeper
2012-02-01 10:22:17,405 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT- assigned=1, rit=false, location=mapr-desktop:60020
2012-02-01 10:22:17,455 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=null; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2012-02-01 10:22:17,610 INFO org.apache.hadoop.hbase.zookeeper.MetaNodeTracker: Detected completed assignment of META, notifying catalog tracker
2012-02-01 10:22:17,616 INFO org.apache.hadoop.hbase.master.HMaster: .META. assigned=2, rit=false, location=mapr-desktop:60020
2012-02-01 10:22:17,616 INFO org.apache.hadoop.hbase.master.HMaster: Master startup proceeding: cluster startup
2012-02-01 10:22:17,639 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning 7 region(s) across 1 server(s), retainAssignment=true
2012-02-01 10:22:17,644 INFO org.apache.hadoop.hbase.master.AssignmentManager: Bulk assigning done
2012-02-01 10:22:17,680 INFO org.apache.hadoop.hbase.master.HMaster: Master has completed initialization
2012-02-01 10:22:17,683 INFO org.apache.hadoop.hbase.master.AssignmentManager: mapr-desktop,60020,1328052118732 unassigned znodes=4 of total=7
2012-02-01 10:22:17,716 INFO org.apache.hadoop.hbase.master.AssignmentManager: mapr-desktop,60020,1328052118732 unassigned znodes=7 of total=7
2012-02-01 10:27:17,685 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:32:17,682 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:37:17,682 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:42:17,683 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:47:17,683 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:52:17,682 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 10:57:17,682 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:02:17,683 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:07:17,684 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:12:17,684 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:17:17,683 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:22:17,684 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9
2012-02-01 11:27:17,685 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.  servers=1 regions=9 average=9.0 mostloaded=9 leastloaded=9

Answer 1

0

I assume you are running whole Hbase on VM. It is quite an interesting exercise, as Hbase requires fare amount of memory to run nicely where VM amount of memory is limited. Regarding the question itself - did you by any chance examined Hbase Master logs to see if there are issues there?

link

answered 31 Jan, 16:10

yufeldman ♦
1.2k●2●7
accept rate: 31%

I am using the VM for development purposes and right now less than 10-20 rows are stored across 5 ot 6 tables in the HBase. Is HBase on the VM known to be having problem with this amount of loads as well? I had a look at the logs and I will post in a moment.

(31 Jan, 16:27) Behrang

As you can see from the logs - ZK session got expired because of long GC cycles. 2012-02-01 08:52:52,065 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 38778ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://wiki.apache.org/hadoop/Hbase/Troubleshooting#A9

And HBaseMaster got restarted in 10 minutes after you were trying to reach it. So it might be OK at the moment.

(31 Jan, 17:31) yufeldman ♦

These long GC cycles are largely due to the memory starvation diet necessary in a small VM. In addition, clocks in VMs are challenging. We have a patch under review for Zookeeper that should make clock instability much less of an issue. You can also try adding memory to the VM container and resizing the various processes.

yufeldman can comment more intelligently on how to force the resizing.

(31 Jan, 18:14) TedDunning ♦♦

Answer 2

@ Behrang

What hardware is the VM running on? Are you running this in a laptop or desktop? Did the system go to sleep? Or was it paused? I was able to replicate a similar situation with the "We slept" warning by pausing the VM waiting 30 seconds and un-pausing it.

In most cases restarting warden will recover hbase process in this situation:

sudo /etc/init.d/mapr-warden restart

Are you able to recover or is this still not working for you? Let me know and we can open a support case and work on it.

Wade

Answer 3

0

@Behrang: While we work on making the VM work seamlessly, you could, in the interim, use the suggestion by Wade (to restart the warden). I tried that suggestion and it works. The reported problem was reproduced by suspending the VM for about 30 minutes. Restarting the warden allows one to relaunch HBase thrift server without having to restart the VM.

link

answered 22 Feb, 12:06

Prasad B
16
accept rate: 0%

The MapR on VM stops working after running for a day or so

Follow this question

Related questions