Client trying to run PigMix on small cluster and get the follwoing. What could be the reason?

ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:213) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:133) at org.apache.pig.impl.PigContext.connect(PigContext.java:183) at org.apache.pig.PigServer.<init>(PigServer.java:226) at org.apache.pig.PigServer.<init>(PigServer.java:215) at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:55) at org.apache.pig.Main.run(Main.java:452) at org.apache.pig.Main.main(Main.java:107) Caused by: java.io.IOException: No FileSystem for scheme: maprfs at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)

asked 16 Aug '11, 08:22

lfedotov's gravatar image

lfedotov ♦♦
201124
accept rate: 27%


In order to have this working correctly, MapR have modifyed pigmix, pointing to correct configurations and correct class paths. Ask MapR support if you need one.

link

answered 17 Aug '11, 08:33

lfedotov's gravatar image

lfedotov ♦♦
201124
accept rate: 27%

This error also occurs if a pig script is executed inside java. The pig executable in the mapr package is set up with the correct classpaths, but while running pig scripts in java, the classpath needs to be set up manually. Here's how the appropriate java properties and classpath need to be setup for mapr:

/usr/lib/jvm/java-6-sun/bin/java -Djava.library.path=/opt/mapr/lib -cp /opt/mapr/pig/pig-0.9.0/pigperf.jar:/opt/mapr/pig/pig-0.9.0/conf/pig.properties:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/pig/pig-0.9.0/pig-0.9.0-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/conf:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.1.jar:/opt/mapr/zookeeper/zookeeper-3.3.2/zookeeper-3.3.2.jar:/opt/mapr/pig/pig-0.9.0/lib/* org.apache.pig.Main yourpigscript.pig
link

answered 23 Aug '11, 14:34

richa's gravatar image

richa
22615
accept rate: 21%

This sounds like you are using the ordinary hadoop distribution on the client side. You may need to install the MapR client distribution on the machine running the program from.

Can you include more details about which nodes are running what software?

link

answered 16 Aug '11, 11:02

TedDunning's gravatar image

TedDunning ♦♦
3.3k318
accept rate: 22%

Working Pigmix script can be downloaded from http://package.mapr.com/tools/pigmix2.tar.gz It have all class paths defined correctly.

link

answered 23 Aug '11, 14:38

lfedotov's gravatar image

lfedotov ♦♦
201124
accept rate: 27%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or __italic__
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×67
×23

Asked: 16 Aug '11, 08:22

Seen: 2,733 times

Last updated: 23 Aug '11, 14:38

powered by OSQA