hbase lzo fatal crash

Question

Hi,

We are heavy hbase user and looking the M3 version of MapR for a week before to decide change our current hadoop base cluster to mapr .M5 . We have a few questions to better understand problem we faced here.

We are testing now 3 server cluster M3. All installation went well and we can import our hbse table without problem. However we cant use the LZO in hbase , we installed as explained http://www.mapr.com/doc/display/MapR/HBase#HBase-compression here.

We import a table to M3 hbase.. Than change the Compression field to LZO.. After that we tried to major_compact to copress colums.. Than regionserver gives:

ATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=dts0.net,60020,1323942299576, load=(requests=0, regions=213, usedHeap=1759, maxHeap=3976): Uncaught exception in service thread regionserver60020.compactor java.lang.AbstractMethodError: com.hadoop.compression.lzo.LzoCompressor.reinit(Lorg/apache/hadoop/conf/Configuration;)V at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:105) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112) at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:199) at org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:397) at org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:383) at org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:354) at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:536) at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:501) at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:836) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:941) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:733) at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:774) at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:719) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:81)

error... and all reio server dead restarting.

So we couldnt use LZO in M3 . is there any suggestion ?

Secondly can we use snappy also with M3 hbase ?

This compression is important for us..

One other thing is that if we use the internal compression of M3 (enabling the compression directory) can we get smilar performance for compression rate and the speed.

Best Regards

Answer 1

0

You may want to go through this one (http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/wiki/FAQ) in addition to the link listed above.

And set following property in conf that is read by your Hbase.

<property>
  <name>io.compression.codecs</name>
  <value>org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec, org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

link

answered 16 Dec '11, 12:36

yufeldman ♦♦
1.9k●2●7
accept rate: 25%

edited 24 Dec '11, 12:54

TedDunning ♦♦
2.4k●3●15

This FAQ link no longer works. Is it still available anywhere?

I have similar problems with LZO tables (that were created before LZO was properly enabled). After going through LZO install steps (from OP link) they are still not recognized / working. Do I need to restart / reboot all nodes or just restarting Region Servers should be enough?

(30 Apr '12, 22:07) tc_dev

It works, its just that the url mistakenly has the closing bracket included in it. Click the URL, and remove the trailing close bracket and reload.

(30 Apr '12, 23:56) Nabeel ♦♦

hbase lzo fatal crash

Follow this question

Related questions