Hi we are using the mapr hadoop distribution. I just installed mapr-pig-0.9.0. Pig works fine in mapreduce mode, but does not work in local mode.

when I tried to run pig in local mode, I got the following error.
I believe this is some issue with the configuration. Could you please help?

I double checked. the file 'toy.txt' is there.

*[ltang@dev-trgt00 tmp]$ /opt/mapr/bin/pig -x local*

2013-01-21 16:17:48,821 [main] INFO  org.apache.pig.Main - Logging error messages to: /data1/home/ltang01/tmp/pig_1358813868819.log
2013-01-21 16:17:49,158 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///

grunt> A = load 'toy.txt';

grunt> dump A;

2013-01-21 16:18:25,559 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-01-21 16:18:25,725 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-01-21 16:18:25,761 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-01-21 16:18:25,762 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-01-21 16:18:25,788 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2013-01-21 16:18:25,813 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-01-21 16:18:25,848 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-01-21 16:18:30,876 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-01-21 16:18:30,915 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2013-01-21 16:18:30,922 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-01-21 16:18:30,941 [Thread-3] INFO  org.apache.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
2013-01-21 16:18:31,423 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-01-21 16:18:31,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
2013-01-21 16:18:31,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-01-21 16:18:31,441 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backend error: ENOENT: No such file or directory
2013-01-21 16:18:31,441 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-01-21 16:18:31,441 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2013-01-21 16:18:31,443 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion   PigVersion      UserId  StartedAt       FinishedAt      Features
0.20.2-dev      0.9.1-SNAPSHOT  ltang01 2013-01-21 16:18:25     2013-01-21 16:18:31     UNKNOWN


Failed Jobs:
JobId   Alias   Feature Message Outputs
N/A     A       MAP_ONLY        Message: ENOENT: No such file or directory
        at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
        at java.lang.Thread.run(Thread.java:662)

Failed to read data from "file:///data1/home/ltang01/tmp/toy.txt"

Failed to produce result in "file:/tmp/temp-859096669/tmp-120753216"

Job DAG:

2013-01-21 16:18:31,443 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-01-21 16:18:31,449 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias A
Details at logfile: /data1/home/ltang01/tmp/pig_1358813868819.log

asked 21 Jan '13, 16:25

Lei%20Tang's gravatar image

Lei Tang
accept rate: 100%

edited 21 Jan '13, 22:05

TedDunning's gravatar image

TedDunning ♦♦

Problem solved. The staging dir does not grant write access to normal users. By changing the staging dir into writable resolves the problem.


answered 08 Feb '13, 11:47

Lei%20Tang's gravatar image

Lei Tang
accept rate: 100%

This is what I was suggesting in my answer above. For the benefit of other users, What did you specifically do?

(08 Feb '13, 15:00) gera

This is what I do:

  1. locate the staging dir from mapr configration:

cat /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml | grep staging.

In our specific case, the staging dir is configred as /var/mapr/cluster/mapred/jobTracker/staging

  1. The owner of the dir was set to root and access mode was 755. I simply changed the folder access to 777.

sudo chmod -R 777 /var/mapr/cluster/mapred/jobTracker/staging

Then problem solved.

(08 Feb '13, 15:06) Lei Tang

You hit a bug related to the way local mode pig/MapReduce is executed as non-root user with MapR. I was able to work around it in the following way:

Create the local runner dir and make it writable by everybody:

$ sudo mkdir /tmp/mapr-hadoop/mapred/local/localRunner $ sudo chmod a+w /tmp/mapr-hadoop/mapred/local/localRunner

Then your script should work.


answered 23 Jan '13, 03:14

gera's gravatar image

accept rate: 19%

Hi Gera,

I just tried your solution. But it did not work. I still encounter the same issue. Is it possible due to the different disks mounted?

The mapr is installed under /data1/opt, while /tmp is under /.

Filesystem          Available Use% Mounted on
/dev/xvda3         10% /
tmpfs               1% /dev/shm
/dev/xvda1          8% /boot
/dev/xvdb1         38% /data1
(23 Jan '13, 10:08) Lei Tang

OK, Let us first verify that basic stuff works

bash $ pig -x local grunt> ls toy.txt grunt> cat toy.txt

Note last two statements are executed in Pig. See if you can list and cat the file.

I also noticed that I did my test with pig 0.10 instead of your 0.9. Consider upgrading.

(23 Jan '13, 11:45) gera

Just tried. It is working fine as expected.

I tested on both pig-0.9.0 (come with mapr) and native hadoop pig 0.10.0. Both are working fine to ls and cat, but signals error when I tried to dump or store one table in local mode.

(31 Jan '13, 22:25) Lei Tang
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or __italic__
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported



Asked: 21 Jan '13, 16:25

Seen: 4,115 times

Last updated: 08 Feb '13, 15:06

powered by OSQA